Distributed SystemsdebuggingAdvanced

One cache node is at 95% CPU while the other three idle

Symptoms

  • cache-c runs at 95% CPU and serves ~45% of all cache reads; cache-a, -b and -d sit at 10–25%.
  • Only cache-c evicts keys; its hit rate is 71% against 96% on the others, so the database sees extra load for the keys that live there.
  • p99 for keys on cache-c is 30 ms versus 1 ms elsewhere.
  • Adding a fifth node (cache-e) last week barely helped: cache-c dropped from 45% to 44% of reads.
# client-side consistent hashing: ONE point per node, position = crc32(hostname)
node      point         arc (to next point clockwise)   share of keyspace
cache-a   0x0A1F3C20    → 0x1C08B4E1                    7.0 %
cache-b   0x1C08B4E1    → 0x4D91E002                   19.4 %
cache-d   0x4D91E002    → 0x96000000                   28.3 %
cache-c   0x96000000    → wrap → 0x0A1F3C20            45.4 %      ← owns nearly half the ring

# after adding cache-e (point 0x05A00000, lands inside cache-c's arc, near its end)
cache-e   0x05A00000    → 0x0A1F3C20                    1.8 %
cache-c   0x96000000    → wrap → 0x05A00000            43.6 %      ← barely changed

# per-node reads/s: a 9,800 · b 27,100 · d 39,600 · c 63,400 · e 2,500
# hottest single key: 0.4 % of reads — there is no hot key

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

cache-c hardware
Hot keys
Ring construction
Adding cache-e
Client library versions
Traffic changes
Redis maxmemory on cache-c