OS + Networking TogetherIntermediate
"Too many open files" — and then the health check passes
Symptoms
- After 6–8 hours of traffic, the service starts returning 500s; logs show
accept: too many open filesandEMFILEfrom an HTTP client. - Restarting the process fixes it for another 6–8 hours.
- The load balancer’s health check keeps passing right up to the crash, so traffic is never drained.
$ ls /proc/$(pgrep -f api-server)/fd | wc -l
65536
$ ulimit -n
65536
$ ss -tanp | awk '{print $1}' | sort | uniq -c
61 ESTAB
64890 CLOSE-WAIT
11 LISTEN
$ ss -tan state close-wait | head -3
CLOSE-WAIT 1 0 10.0.3.4:41822 10.0.5.9:443
CLOSE-WAIT 1 0 10.0.3.4:41830 10.0.5.9:443
# app log, hours earlier
WARN upstream 10.0.5.9:443 returned 503, retrying
WARN upstream 10.0.5.9:443 returned 503, retryingInvestigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
The `ulimit -n` value
What the open descriptors are
The code path that talks to 10.0.5.9
Upstream 503 rate
Memory growth
Why the health check kept passing