Concurrency, Synchronization & DeadlocksAdvanced

Every core pinned at 100%, throughput lower than yesterday

Symptoms

  • All eight cores at 100% since the worker pool was raised from 4 to 32 threads.
  • Throughput is 40% *lower* than with 4 threads; p99 latency has tripled.
  • top shows the process at 780% CPU with most of it as system time, not user time.
  • The application profiler shows a flat profile — no function above 3%.
  • Disk and network are idle; the database reports no slow queries.
$ top -b -n1 | head -5            # Linux
%Cpu(s): 28.3 us, 68.9 sy,  0.0 ni,  2.1 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
  PID USER  PR NI    VIRT    RES  %CPU %MEM  TIME+   COMMAND
 4242 svc   20  0 2841.2m 412.8m 781.0  2.6  92:14.7 metrics-svc

$ perf top -p 4242 --sort symbol            # 10 s sample
  41.2%  [k] native_queued_spin_lock_slowpath
  17.8%  [k] _raw_spin_lock_irqsave
   9.4%  [k] futex_wake
   8.1%  [k] futex_wait_queue
   6.0%  [.] __lll_lock_wait
   2.9%  [.] MetricsRegistry::record

$ strace -c -f -p 4242  (10 s)
% time   seconds   usecs/call   calls     syscall
 94.1   9.412331          4   2317642   futex
  3.2   0.320117         12     26502   epoll_wait

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

The application profiler
User vs system time in `top`
`perf top` and `strace -c`
What the pool-size change did
The code under the lock
GC and memory
Downstream dependencies