Diagnosing EMFILE

EMFILE means the per-process descriptor table is full — the soft RLIMIT_NOFILE, often 1,024 by default and raised by the service manager to 65,536 (label: Linux/Unix; ENFILE is the system-wide table). The first question is not "how high" but "what is in the table": ls /proc/<pid>/fd | wc -l sampled every few minutes tells you whether the count tracks load (bounded, fine) or grows monotonically since start (a leak) (File Descriptors).

Then classify. lsof -p <pid> (or ls -l /proc/<pid>/fd) shows each descriptor’s type: regular files with a path, sockets with peers, pipes, anon_inode entries for epoll/eventfd/timerfd. Hundreds of sockets to the same upstream is one story; hundreds of the same log path is another; thousands of pipes means child processes. For sockets, ss -tanp | grep <pid> gives the state: many `CLOSE_WAIT` means the peer closed and the application never called close() — a leak in the client code, not a kernel problem. TIME_WAIT sockets, by contrast, hold no descriptor and are the kernel’s business (The Socket: A Descriptor With Two Kernel Buffers Behind It).

Common causes: an HTTP client or database connection created per request and never closed or pooled; a response body not consumed/closed (resp.Body.Close() in Go, unconsumed streams in Node.js); a file opened in a code path whose error branch skips close; a keepAlive agent per request whose idle sockets outlive any reference; child processes inheriting descriptors. Fix the ownership — pooling, try/finally, RAII, using — and add a descriptor-count metric so the next leak is a graph, not an outage (The OS Debugging Playbook: Four Symptoms, Fourteen Causes).

Raising the limit is the *second* step, and legitimate: a server holding 20,000 connections needs a table that size, and the soft limit should be set at the hard limit (LimitNOFILE= in systemd, worker_rlimit_nofile in nginx). But raising it for a leak only moves the failure to a bigger, later, more confusing incident.