OS + Networking TogetherAdvanced

Connection errors to a healthy upstream at 400 requests per second

Symptoms

  • A proxy service starts failing calls to its upstream with connect: cannot assign requested address (EADDRNOTAVAIL) once traffic passes ~400 req/s.
  • The upstream reports no errors and low latency; a curl from the proxy host to the upstream works.
  • The failures come in waves lasting about a minute, then clear briefly, then return.
$ ss -s
TCP:   29418 (estab 212, closed 28906, orphaned 3, timewait 28871)

$ ss -tan state time-wait '( dport = :9200 )' | wc -l
28190

$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768	60999

$ sysctl net.ipv4.tcp_tw_reuse
net.ipv4.tcp_tw_reuse = 0

# proxy log
dial tcp 10.0.9.5:9200: connect: cannot assign requested address
dial tcp 10.0.9.5:9200: connect: cannot assign requested address

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Upstream health and capacity
Firewall and security groups
`ss -s` socket summary
How the proxy opens upstream connections
File descriptor limit
DNS resolution of the upstream