Concurrency, Synchronization & DeadlocksAdvanced
Every core pinned at 100%, throughput lower than yesterday
Symptoms
- All eight cores at 100% since the worker pool was raised from 4 to 32 threads.
- Throughput is 40% *lower* than with 4 threads; p99 latency has tripled.
topshows the process at 780% CPU with most of it as system time, not user time.- The application profiler shows a flat profile — no function above 3%.
- Disk and network are idle; the database reports no slow queries.
$ top -b -n1 | head -5 # Linux %Cpu(s): 28.3 us, 68.9 sy, 0.0 ni, 2.1 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st PID USER PR NI VIRT RES %CPU %MEM TIME+ COMMAND 4242 svc 20 0 2841.2m 412.8m 781.0 2.6 92:14.7 metrics-svc $ perf top -p 4242 --sort symbol # 10 s sample 41.2% [k] native_queued_spin_lock_slowpath 17.8% [k] _raw_spin_lock_irqsave 9.4% [k] futex_wake 8.1% [k] futex_wait_queue 6.0% [.] __lll_lock_wait 2.9% [.] MetricsRegistry::record $ strace -c -f -p 4242 (10 s) % time seconds usecs/call calls syscall 94.1 9.412331 4 2317642 futex 3.2 0.320117 12 26502 epoll_wait
Investigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
The application profiler
User vs system time in `top`
`perf top` and `strace -c`
What the pool-size change did
The code under the lock
GC and memory
Downstream dependencies