Concurrency, Synchronization & DeadlocksAdvanced
The API stopped answering; the process is alive and idle
Symptoms
- All requests time out; the process is alive, at 0.3% CPU, memory normal.
- A TCP connect to the port succeeds but no HTTP response ever arrives.
- It happened right after a burst of concurrent cache refreshes; a restart fixes it for a while.
- The database shows no active queries from this service.
$ ss -ltn 'sport = :8080' # Linux
State Recv-Q Send-Q Local Address:Port
LISTEN 129 128 0.0.0.0:8080 # accept backlog full: nobody is accepting
$ ps -L -o tid,stat,wchan:20,comm -p 4242 | head -6
TID STAT WCHAN COMMAND
4242 Sl futex_wait_queue api
4250 Sl futex_wait_queue worker-1
4256 Sl futex_wait_queue worker-7
4258 Sl futex_wait_queue worker-9
$ jstack 4242 | tail -12
"worker-1": waiting to lock <0x00000007a1c0e2f0> (a CacheLock)
which is held by "worker-7"
at Cache.refresh(Cache.java:88)
- locked <0x00000007a1c0e410> (a StatsLock)
"worker-7": waiting to lock <0x00000007a1c0e410> (a StatsLock)
which is held by "worker-1"
at Stats.record(Stats.java:41)
- locked <0x00000007a1c0e2f0> (a CacheLock)
Found 1 deadlock.
"worker-2" … "worker-32": waiting to lock <0x00000007a1c0e2f0> (a CacheLock)Investigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
Network and load balancer
The database
Blocked I/O in the kernel (`D` state)?
The thread dump
GC pause
Lock acquisition order in the code