CPU SchedulingIntermediate

One core at 100% forever, and the service still answers

Symptoms

  • Since the last deploy the process sits at exactly 100% CPU (one core) around the clock.
  • Requests are served normally; latency is unchanged.
  • Load average is one higher than before, on every host running the new version.
  • ps -o cputime for the process grows by 60 seconds every minute.
$ top -H -b -n1 -p 4242 | head -8          # Linux, per-thread
  PID USER  PR NI  VIRT  RES  S %CPU %MEM  TIME+    COMMAND
 4261 svc   20  0 1.9g 310m  R 99.9  1.9  412:07.9 worker-3
 4258 svc   20  0 1.9g 310m  S  1.3  1.9   3:11.2  worker-0
 4259 svc   20  0 1.9g 310m  S  1.1  1.9   3:09.8  worker-1
 4242 svc   20  0 1.9g 310m  S  0.3  1.9   0:41.0  svc

$ strace -p 4261 -f -tt        (30 s)
strace: Process 4261 attached
                                # no output for 30 s

$ gdb -p 4242 -batch -ex 'thread apply 4 bt' 2>/dev/null | head -4   # sampled twice, 10 s apart
#0  Retry::next_delay (this=0x7f21c0012a40) at retry.cc:41
#1  Retry::run<...> (this=0x7f21c0012a40) at retry.cc:58
#2  Worker::refresh_token (this=0x7f21c0010000) at worker.cc:113
(second sample: identical frames)

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

GC or runtime activity
Per-thread CPU (`top -H`)
`strace` on the hot thread
Two stack samples ten seconds apart
The retry code in the deploy diff
System time and the kernel