Virtual Memory & PagingAdvanced

The box is at 3% CPU and everything takes ten seconds

Symptoms

  • Every process on the host is slow — the API, PostgreSQL, even ssh login takes 20 seconds.
  • CPU is almost idle but load average is 30 on 8 cores.
  • top shows high wa and dozens of processes in state D.
  • The disk activity light is solid; the SSD is doing constant I/O although nobody deployed anything.
  • It started after the JVM heap was raised to 12 GB "because we had spare RAM".
$ vmstat 1 5                                          # Linux
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2 24 7912340 118204   1120 210400 48120 51220 49300 51400 9100 12000  2  4  8 86  0
 1 27 7940112 116980   1104 208112 50200 52880 51100 53000 9300 11800  1  3  9 87  0

$ free -m
              total   used   free  shared  buff/cache  available
Mem:          16000  15650    118       4         232         210
Swap:          8192   7750    442

$ top -b -n1 -o RES | head -9
  PID USER  PR NI  VIRT    RES   SHR S %CPU %MEM  TIME+   COMMAND
 5120 app   20  0 14.1g  11.2g  8m  D  1.2 70.0  91:12   java -Xmx12g
 2210 pg    20  0  2.9g   2.1g  1.8g D  0.4 13.1  40:08   postgres: checkpointer
 4242 app   20  0  1.4g   0.6g  4m  D  0.3  3.9  12:41   node api.js
  871 root  20  0  0.2g   0.0g  1m  D  0.0  0.1   0:03   sshd

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

CPU profile of the slow processes
Disk health
`vmstat` swap columns
Who owns the resident memory
The load average
`vm.swappiness` and overcommit settings