Files, File Systems & DescriptorsIntermediate

EMFILE after thirty hours of uptime

Symptoms

  • Error: EMFILE: too many open files from accept() and, moments later, from opening the log file.
  • It happens about 30 hours after each deploy; a restart clears it.
  • Concurrent client connections are steady at ~40; request rate is unchanged.
  • The soft descriptor limit is 1,024.
$ ls /proc/4242/fd | wc -l                        # Linux
1021
$ cat /proc/4242/limits | grep 'open files'
Max open files            1024                 65536                files

$ lsof -p 4242 -a -d 0-99999 | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn
    962 IPv4
     30 REG
     14 a_inode
      6 CHR
      5 FIFO

$ ss -tanp | awk '$1=="CLOSE-WAIT"' | grep 'pid=4242' | wc -l
940
$ ss -tanp state close-wait | grep 'pid=4242' | head -3
Recv-Q Send-Q Local Address:Port    Peer Address:Port
   1      0    10.0.1.12:51734      10.0.3.17:443   users:(("node",pid=4242,fd=812))
   1      0    10.0.1.12:51802      10.0.3.17:443   users:(("node",pid=4242,fd=813))
   1      0    10.0.1.12:51944      10.0.3.17:443   users:(("node",pid=4242,fd=814))

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Raise the limit
Legitimate connection count
Descriptor types (`lsof`)
Socket states (`ss`)
The HTTP client code
Log rotation