ProcessesIntermediate

Thousands of `<defunct>` processes, then fork fails

Symptoms

  • ps lists thousands of [worker] <defunct> entries, all with the same parent.
  • The parent starts failing with fork: Resource temporarily unavailable (EAGAIN).
  • Memory and CPU are normal; the defunct entries do not consume RAM.
  • The count grows by exactly one per job processed.
$ ps -eo pid,ppid,stat,rss,cmd | awk '$3 ~ /Z/' | head -4        # Linux
  PID  PPID STAT RSS CMD
 8811  4242 Z      0 [worker] <defunct>
 8814  4242 Z      0 [worker] <defunct>
 8820  4242 Z      0 [worker] <defunct>
 8823  4242 Z      0 [worker] <defunct>
$ ps -eo stat | grep -c Z
4071

$ ps -o pid,ppid,stat,cmd -p 4242
  PID PPID STAT CMD
 4242    0 Ss   node scheduler.js          # PID 1 inside the container

$ cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max
4096
4096
$ kill -9 8811; ps -o stat -p 8811
STAT
Z                                           # still there

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Memory consumption of the zombies
Killing the zombies
The children’s code
The parent’s child-exit handling
The PID limit that turns zombies into failure
Who is PID 1 in the container