IP, IPv4, IPv6, Subnets, NAT & PortsAdvanced
Outbound calls fail from the whole cluster whenever traffic peaks
Symptoms
- During peak hours, services across all nodes see sporadic
connect: connection timed outto external APIs. - The same APIs answer instantly from a laptop outside the cluster.
- Failures correlate with request volume, not with any particular destination.
- Some connections are reset mid-request rather than failing to connect.
# on a worker node (Linux)
$ dmesg | tail -3
[812031.117] nf_conntrack: nf_conntrack: table full, dropping packet
[812031.121] nf_conntrack: nf_conntrack: table full, dropping packet
$ sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_max = 65536
net.netfilter.nf_conntrack_count = 65536
$ conntrack -L 2>/dev/null | awk '{print $4}' | sort | uniq -c | sort -rn | head -3
52210 TIME_WAIT
9105 ESTABLISHED
3811 SYN_SENT
$ sysctl net.netfilter.nf_conntrack_tcp_timeout_time_wait
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
# the cloud NAT gateway metrics for the same window
ErrorPortAllocation spikes to 1,400/min at 17:00–17:20
ConnectionAttemptCount ~ 9,800/s across the clusterInvestigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
The kernel log
What fills the table
Why so many new connections
The cloud NAT gateway
The external APIs
DNS
The mid-request resets