CPU SchedulingIntermediate
One core at 100% forever, and the service still answers
Symptoms
- Since the last deploy the process sits at exactly 100% CPU (one core) around the clock.
- Requests are served normally; latency is unchanged.
- Load average is one higher than before, on every host running the new version.
ps -o cputimefor the process grows by 60 seconds every minute.
$ top -H -b -n1 -p 4242 | head -8 # Linux, per-thread
PID USER PR NI VIRT RES S %CPU %MEM TIME+ COMMAND
4261 svc 20 0 1.9g 310m R 99.9 1.9 412:07.9 worker-3
4258 svc 20 0 1.9g 310m S 1.3 1.9 3:11.2 worker-0
4259 svc 20 0 1.9g 310m S 1.1 1.9 3:09.8 worker-1
4242 svc 20 0 1.9g 310m S 0.3 1.9 0:41.0 svc
$ strace -p 4261 -f -tt (30 s)
strace: Process 4261 attached
# no output for 30 s
$ gdb -p 4242 -batch -ex 'thread apply 4 bt' 2>/dev/null | head -4 # sampled twice, 10 s apart
#0 Retry::next_delay (this=0x7f21c0012a40) at retry.cc:41
#1 Retry::run<...> (this=0x7f21c0012a40) at retry.cc:58
#2 Worker::refresh_token (this=0x7f21c0010000) at worker.cc:113
(second sample: identical frames)Investigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
GC or runtime activity
Per-thread CPU (`top -H`)
`strace` on the hot thread
Two stack samples ten seconds apart
The retry code in the deploy diff
System time and the kernel