OS + Networking TogetherIntermediate
One slow client makes the whole API slow
Symptoms
- Every few minutes the API’s p99 jumps from 40 ms to 8–12 s for all clients, then recovers.
- CPU on the API hosts is under 20% during the incident; no errors are logged.
- The spikes correlate with a mobile client on a poor connection downloading a large report export.
$ ss -tino state established '( sport = :8080 )' | head ESTAB 0 3145728 10.0.4.12:8080 198.51.100.77:51203 cubic wscale:7,7 rto:2280 rtt:1140/210 ato:40 mss:1448 cwnd:4 ssthresh:7 bytes_sent:52428800 bytes_retrans:1245280 unacked:4 rcv_space:14600 notsent:3087500 ESTAB 0 0 10.0.4.12:8080 10.0.7.31:44821 ... rtt:0.3/0.1 cwnd:10 ESTAB 0 0 10.0.4.12:8080 10.0.7.32:44830 ... rtt:0.4/0.1 cwnd:10 $ curl -s localhost:9090/metrics | grep -E 'threads_(busy|max)' http_threads_busy 32 http_threads_max 32 # app log (one line per second) [pool] all 32 worker threads busy; queue depth 417
Investigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
Server CPU and load average
Database latency
The slow client’s connection in `ss -tin`
How the handler writes the response
Thread pool size and queue
Load balancer health checks
Kernel socket buffer sizes