Memory Pressure, Swap and the OOM Killer

Virtual memory lets processes reserve far more than exists (Why Virtual Memory?), and even what they *touch* can exceed RAM, because a page that is present now can be made not-present later and brought back on a fault (Page Faults). The kernel’s job under pressure is to choose which pages stay: the ones being used now — each process’s working set — and to evict the rest. If the combined working sets fit in RAM, the machine runs at full speed while the sum of everything touched is 12 GB. If they do not, every evicted page is needed again soon, and the machine thrashes.

Every page in RAM belongs to one of two classes, and the class decides what eviction costs. File-backed pages — program code, shared libraries, and the page cache holding file contents — have a copy on disk already. A clean one can be dropped for free and re-read on demand; a dirty one (a file write not yet flushed) must be written back first. Anonymous pages — heaps, stacks, anything from malloc or anonymous mmap — have no file behind them. The only way to evict one is to write it to swap (a partition or file), and if there is no swap they cannot be evicted at all.

Linux keeps every page on one of a few lists per memory zone: active and inactive, each split into file and anonymous. New pages enter inactive; a page referenced again while on inactive (seen via the PTE accessed bit, Paging) is promoted to active; pages age from active back to inactive as reclaim scans them. Reclaim evicts from the tail of the inactive lists — an approximation of LRU over the whole machine, in the same sense as the LRU Cache: recency as a stand-in for the working set, with the same failure when a one-pass scan (a backup reading 50 GB) floods the inactive list and pushes hot pages out. Linux mitigates that with a workingset shadow-entry mechanism that detects refaults and, since 5.x, a multi-generational LRU (MGLRU) that tracks age more finely.

Reclaim runs at two urgency levels. When free memory drops below a per-zone low watermark, the kswapd kernel thread wakes and reclaims in the background until the high watermark is restored; the machine stays responsive. If allocations outrun kswapd and free memory hits the min watermark, the allocating process itself enters direct reclaim — it stalls inside its own malloc or page fault, scanning lists and writing pages out — which is where latency spikes come from under pressure. vm.swappiness (0–200, default 60) biases reclaim between file pages and anonymous pages; it does not mean "how eager to swap" in any simple sense.

Swap is not a failure mode; it is where cold anonymous memory belongs. A daemon’s initialisation data, an idle tab’s heap, the part of a long-running process that was used once at startup — paging those out makes room for page cache that is actually hot, and a machine with swap and a small swap usage is doing exactly what it should. The pathology is swap traffic, not swap *usage*: pages going out and coming back in continuously because the working sets do not fit.

That is thrashing. Each process runs a few instructions, faults, sleeps for the ~100 µs to 10 ms it takes to page in, runs a few more, faults again; the CPU sits mostly idle in I/O wait while the disk saturates, and nothing makes progress. Load average climbs (processes waiting on I/O are counted), si/so in vmstat are continuously non-zero, %wa is high, and interactive response goes from milliseconds to seconds. The only cures are less demand (kill or shrink something) or more RAM; tuning does not make working sets smaller.

Reclaim can fail: no file pages left to drop, no swap (or swap full), and an allocation that must be satisfied. Linux then invokes the OOM killer, which picks the process with the highest oom_score — roughly its share of RAM plus swap, adjusted by oom_score_adj (−1000 exempts, +1000 volunteers) — and sends it SIGKILL. The choice is a heuristic, and it is frequently the wrong one from the operator’s point of view: the largest process is often the database you were trying to protect. dmesg records the decision in detail; systemd and Kubernetes record OOMKilled and exit code 137.

The killer exists because of overcommit. With vm.overcommit_memory = 0 (the default heuristic) the kernel lets processes reserve more than RAM + swap on the theory that they will not touch it all; 1 always allows; 2 refuses any reservation beyond swap + overcommit_ratio × RAM, which turns OOM into malloc returning NULL — deterministic, and the choice of systems that would rather fail a request than lose a process. Windows works in mode 2 by default: commit charge is enforced and there is no OOM killer, so a VirtualAlloc(MEM_COMMIT) beyond the limit fails immediately.

Containers make this local. A cgroup with memory.max = 2G gets its own reclaim and its own OOM killer: when the group’s usage hits the limit, reclaim runs inside the group, and if it fails a process *in the group* is killed, regardless of how much free memory the host has. The page cache a containerised process uses counts against the group too, which is the answer to "my container was OOM-killed at 2 GB while top showed 500 MB RSS" — see Containers Are Processes With the Kernel’s View Narrowed.

A healthy Linux box shows almost no free memory, because unused RAM is wasted RAM and the kernel fills it with page cache. free -m prints free (truly unused), buff/cache (page cache, most of it reclaimable) and — since kernel 3.14 — available, the kernel’s estimate of how much can be allocated without swapping, counting reclaimable cache. Available is the number to alert on. Likewise per process, RSS counts shared pages in every process that maps them (ten Chrome renderers "using" 300 MB each share most of it); PSS (proportional set size, in /proc/<pid>/smaps_rollup) divides shared pages by the number of sharers and is what adds up to the real total.

The question to ask is never "how much is free" but "is anything waiting for memory": major faults per second, si/so, direct-reclaim stalls (/proc/pressure/memory on kernels with PSI reports the fraction of time tasks were stalled on memory), and kswapd CPU. A machine at 2% free with zero swap traffic and PSI at 0 is fine; a machine at 20% free with a process in direct reclaim every second is not.