By yigal


2016-10-02 03:30:02 8 Comments

I have a Go program that generates a lot of HTTP requests from multiple goroutines. after running for a while, the program spits out an error: connect: cannot assign requested address.

When checking with netstat, I get a high number (28229) of connections in TIME_WAIT.

The high number of TIME_WAIT sockets happens when I the number of goroutines is 3 and is severe enough to cause a crash when it is 5.

I run Ubuntu 14.4 under docker and go version 1.7

This is the Go program.

package main

import (
        "io/ioutil"
        "log"
        "net/http"
        "sync"
)
var wg sync.WaitGroup
var url="http://172.17.0.9:3000/";
const num_coroutines=5;
const num_request_per_coroutine=100000
func get_page(){
        response, err := http.Get(url)
        if err != nil {
                log.Fatal(err)
        } else {
                defer response.Body.Close()
                _, err =ioutil.ReadAll(response.Body)
                if err != nil {
                        log.Fatal(err)
                }
        }

}
func get_pages(){
        defer wg.Done()
        for i := 0; i < num_request_per_coroutine; i++{
                get_page();
        }
}

func main() {
        for i:=0;i<num_coroutines;i++{
                wg.Add(1)
                go get_pages()
        }
        wg.Wait()
}

This is the server program:

package main

import (
    "fmt"
    "net/http"
    "log"
)
var count int;
func sayhelloName(w http.ResponseWriter, r *http.Request) {
    count++;
    fmt.Fprintf(w,"Hello World, count is %d",count) // send data to client side
}

func main() {
    http.HandleFunc("/", sayhelloName) // set router
    err := http.ListenAndServe(":3000", nil) // set listen port
    if err != nil {
        log.Fatal("ListenAndServe: ", err)
    }
}

1 comments

@JimB 2016-10-03 14:40:19

The default http.Transport is opening and closing connections too quickly. Since all connections are to the same host:port combination, you need to increase MaxIdleConnsPerHost to match your value for num_coroutines. Otherwise, the transport will frequently close the extra connections, only to have them reopened immediately.

You can set this globally on the default transport:

http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = numCoroutines

Or when creating your own transport

t := &http.Transport{
    Proxy: http.ProxyFromEnvironment,
    DialContext: (&net.Dialer{
        Timeout:   30 * time.Second,
        KeepAlive: 30 * time.Second,
    }).DialContext,
    MaxIdleConnsPerHost:   numCoroutines,
    MaxIdleConns:          100,
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   10 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
}

Similar question: Go http.Get, concurrency, and "Connection reset by peer"

@yigal 2016-10-04 05:17:34

JimB, I used the first option above and it greatly improved the behavior of the program. now it does not crash on low numbers of num_conections, but I does break for high numbers (for example 10000). I'll try the more more verbose option and see it helps more.

@JimB 2016-10-04 12:30:45

@yigal: of course it will break if you raise concurrency high enough, what is the point of testing 10000 concurrent connections with a single http client and server over loopback? You only have so many file descriptors and ephemeral ports you can make use of without some system tuning and better configuration.

@yigal 2016-10-04 19:34:27

the idea is to stress test our system using just one client machine. The advantage of single client machine over multiple client machines is that is should be simpler to develop and test the stress test code. I am trying out golang for this purpose as it is a fast language with low overhead for spawning threads/coroutines. I am not completely versed at linux optimizations, but my logic says that 10000 concurrent connections should be achievable with stock Linux. I just need to handle the TIME_WAIT problem more efficiently.

@James Sapam 2016-12-20 00:21:51

Any difference when we do a POST request ?

@Amir Khoshhal 2020-06-22 15:16:16

How can I set different proxies for every request in this way ? Is it possible ?

Related Questions

Sponsored Content

1 Answered Questions

[SOLVED] Can a lot of TIME_WAIT bring down a server?

2 Answered Questions

1 Answered Questions

Lot of connections in TIME_WAIT

1 Answered Questions

[SOLVED] Different results for N>1 goroutines (on N>1 Cpu:s). Why?

2 Answered Questions

Receive data on socket in TIME_WAIT state

1 Answered Questions

1 Answered Questions

Sponsored Content