I/OIntermediate

The worker is stuck and `kill -9` does nothing

Symptoms

  • A batch worker stopped making progress an hour ago.
  • kill -TERM and then kill -9 were sent; the process is still listed.
  • ps shows the process in state D; CPU is idle; load average rose by exactly one.
  • The stall began at the same time as a brief outage of the storage server.
$ ps -o pid,stat,wchan:28,etime,cmd -p 4242              # Linux
  PID STAT WCHAN                        ELAPSED CMD
 4242 D    rpc_wait_bit_killable       03:12:40 python worker.py

$ cat /proc/4242/stack
[<0>] rpc_wait_bit_killable+0x24/0xb0
[<0>] __rpc_execute+0x1d8/0x4e0
[<0>] nfs_file_read+0x9a/0x120
[<0>] vfs_read+0xa2/0x1b0
[<0>] ksys_read+0x5f/0xe0

$ dmesg -T | tail -3
[Tue 14:02:11] nfs: server storage01 not responding, still trying
[Tue 14:05:41] nfs: server storage01 not responding, still trying
[Tue 14:09:11] nfs: server storage01 not responding, still trying

$ strace -p 4242
strace: Process 4242 attached
                                # hangs: ptrace cannot stop a task in D state
$ mount | grep storage01
storage01:/data on /mnt/data type nfs (rw,hard,vers=4.1,timeo=600,retrans=2)

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Application locks / deadlock
Attaching a debugger or strace
`/proc/<pid>/stack` and `wchan`
`dmesg` and the mount options
CPU and memory
The process’s own timeout logic