Under load, some requests hang for exactly 60 seconds

• checkout calls pricing over plain HTTP on every cart change: POST pricing.internal:8080/quote, behind an internal load balancer with three healthy targets.
• Four checkout pods run per Kubernetes node; at peak the node originates about 1 200 pricing calls per second.
• The pricing client is a thin wrapper around a Python requests.post(...) call with timeout=(60, 5) — a 60 s connect timeout and a 5 s read timeout.
• Nothing in checkout or pricing has been deployed in nine days; peak traffic is about 30% higher than a month ago.
• Available: application logs on both sides, pricing access logs, LB metrics, and shell access to a checkout node.

# checkout   application log, 12:04
12:04:11.020  pricing.quote cart=c_91a  connect pricing.internal:8080
12:05:11.024  pricing.quote cart=c_91a  ETIMEDOUT after 60 001 ms (connect)
12:04:11.031  pricing.quote cart=c_91b  200 in 11 ms
12:04:11.033  pricing.quote cart=c_88f  OSError [Errno 99] Cannot assign requested address (EADDRNOTAVAIL, 0 ms)

# pricing   access log, cart=c_91a, 12:04–12:06
(no entries)

# checkout node, 12:04
$ ss -s
TCP:   31 402 (estab 118, closed 31 180, orphaned 0, timewait 31 020)

$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768	60999            # 28 232 ephemeral ports

$ ss -tan state time-wait | awk '{print $4}' | cut -d: -f1 | sort | uniq -c
  10 391 10.20.4.11        # the three pricing targets
  10 322 10.20.4.12
  10 307 10.20.4.13

$ conntrack -C ; sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_tcp_timeout_time_wait
131072
net.netfilter.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120

$ dmesg | tail -3
nf_conntrack: nf_conntrack: table full, dropping packet
nf_conntrack: nf_conntrack: table full, dropping packet
nf_conntrack: nf_conntrack: table full, dropping packet

# checkout/src/clients/pricing.py
def quote(cart):
    r = requests.post(PRICING_URL + "/quote", json=cart, timeout=(60, 5))
    return r.json()