Reading Memory: RSS, Heap, Working Set and the Number on Your Dashboard
The memory number everyone watches is usually the wrong one. Resident, virtual, heap, cache and cgroup working set answer different questions, and allocation rate — the one nobody charts — often matters more than any of them.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Five numbers, five different questions
Virtual size (VSZ) is address space the process has mapped. It includes memory never touched, shared libraries, and large reservations that runtimes make on principle. A JVM or a Go binary can show tens of gigabytes of virtual size on a 2 GB container and be perfectly healthy. Alerting on it produces pure noise.
Resident set size (RSS) is physical memory currently backing those pages. This is closer to real but still slippery: it includes shared pages counted against every process that maps them, and it includes file-backed pages the kernel could drop under pressure. Heap size is what the runtime has allocated from its own arena — and it can be flat while RSS climbs, because the runtime freed objects without returning pages to the OS.
The number that decides whether you get killed is the cgroup working set: anonymous memory plus unreclaimable page cache, measured against memory.max. That is what the OOM killer compares. Everything else is diagnostic colour. And the number that predicts *future* trouble better than any level does is allocation rate — bytes per second churned — because it drives GC frequency, pause time and cache displacement long before any level looks alarming (Allocation Rate Is a Cost Even Without a Leak).
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Virtual size (VSZ) | 14.2 GB | Address space, mostly untouched reservations. Not evidence of anything. Do not alert on it. | normal |
| RSS | 1.6 GB | Physical pages backing the process, including shared and file-backed. A reasonable first look. | normal |
| Runtime heap in use | 840 MB | Live objects the runtime tracks. The gap to RSS is arena fragmentation plus pages not returned to the OS. | suspect |
| cgroup working set / limit | 1.71 GB / 2.0 GB (86%) | The reading the OOM killer uses. This is the one to alert on. | smoking gun |
| Host "memory used" | 94% | Mostly reclaimable page cache. On Linux this is normal and healthy; it is not a shortage. | normal |
| Allocation rate | 410 MB/s | Nothing leaks, but this churn drives GC frequency and cache pressure. Rarely charted, often the real story. | smoking gun |
Why the host chart says 94% and everything is fine
On Linux, unused memory is wasted memory, so the kernel fills it with page cache: file data kept around in case it is read again. A host that reports 94% "used" is usually 60% page cache, all of which the kernel will release the instant anyone needs it. This is the single most common false alarm in infrastructure monitoring, and it trains people to ignore memory alerts — which is how the real one gets missed.
The corollary matters for databases and anything file-backed: that page cache is doing real work. When a database's buffer hit ratio falls and disk reads spike, one plausible cause is that something else on the box displaced its cached pages (The Buffer Pool). "Free" memory that was serving reads is not free capacity you can reclaim without cost; it is a cache you are about to invalidate (Disk and Storage: Latency, Throughput, IOPS and the fsync Tax).
In containers the accounting narrows to your cgroup and the picture gets sharper but not simpler: page cache generated by your process counts against your limit, and dirty pages that cannot be written back are unreclaimable. A service that streams large files can be OOM-killed by its own page cache while its heap is small and stable — a failure mode that looks nothing like a leak and is not fixed by tuning the runtime.
| Decision | Reading to use | Reading that misleads |
|---|---|---|
| Will this container get OOM-killed? | cgroup working set vs memory.max | Host free memory; VSZ |
| Is there a leak? | Working set trend under stable load, over hours (Memory Leaks: Growth That Does Not Come Back) | A single high reading; RSS right after startup |
| Is GC hurting latency? | Allocation rate + GC pause time and frequency (Garbage Collection: Pause, Throughput, Footprint — Pick Two) | Heap size — a large calm heap can be healthier than a small churning one |
| Do we need a bigger instance? | Working set at peak plus Headroom: The Capacity You Deliberately Do Not Use, with allocation rate flat | Host "used" percentage, which is mostly page cache |
| Is the database losing its cache? | Buffer hit ratio and disk read rate (Which Signal Actually Means "The Database Is Slow") | Host memory free, which will look "fine" precisely because cache was evicted |
Allocation rate: the signal nobody charts
Two services with an identical 800 MB heap can behave completely differently. One allocates 5 MB/s and its garbage collector barely runs. The other allocates 400 MB/s, collects constantly, and pays for it in pause time, CPU and cache displacement — every collection walks memory and evicts the CPU caches that the request path depended on, so the cost shows up as latency in code that has nothing to do with allocation.
This is why "memory is fine, we are at 40%" is compatible with a memory-caused latency problem. The level is fine; the *flow* is not. Allocation rate is the flow, and it responds to things a level chart cannot see: a serializer that builds intermediate strings, a per-request logger that formats objects eagerly, a defensive copy in a hot loop.
Practically: chart allocation rate next to heap level, and when latency has an unexplained periodic component, check whether its period matches GC frequency before looking anywhere else. The runtime-specific mechanics — generational collection, pause characteristics, escape analysis — are the domain of Garbage Collection: Pause, Throughput, Footprint — Pick Two and JavaScript Runtime Performance: V8 Where It Costs; what belongs here is the habit of treating rate and level as two different signals.
Service A Service B
heap in use 820 MB heap in use 840 MB
alloc rate 6 MB/s alloc rate 410 MB/s
GC cycles 0.4 /min GC cycles 38 /min
GC CPU 0.3% GC CPU 11%
p99 latency 42 ms p99 latency 210 ms
p99 has a 1.6s period
matching GC frequency
The level chart cannot tell these apart. The rate chart
names the problem in one glance.Key points
- Virtual size is address space and is close to meaningless; alerting on it is pure noise.
- The cgroup working set against its limit is the reading the OOM killer uses — that is the one to alert on.
- Host "memory used" is dominated by reclaimable page cache; 94% is usually healthy, and treating it as an emergency trains people to ignore memory alerts.
- Heap can be flat while RSS grows: runtimes free objects without returning pages, so the two diverge legitimately.
- Allocation rate is a different signal from memory level, and it explains latency that no level chart can.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Request path → allocator: each request allocates intermediate buffers, strings and copies; the volume tracks request rate times per-request allocation.
- 2Allocator → heap: the runtime grows its arena to satisfy demand; heap-in-use rises and the collector schedules more frequently.
- 3Collector → CPU and caches: each cycle costs CPU and evicts CPU-cache lines the request path was using, so unrelated code gets slower.
- 4Heap → RSS: the runtime retains freed pages rather than returning them, so RSS stays high even after the workload subsides.
- 5RSS → cgroup working set → OOM killer: if working set crosses
memory.max, the kernel kills the process regardless of how healthy the host looks.
- • "Memory is at 94%, we are about to fall over" — on a host, that is almost always page cache doing its job.
- • "RSS keeps growing, so we have a leak" — RSS growth with a flat live heap is usually retained pages or fragmentation, not a leak (Leak or Unbounded Cache? The Question That Picks the Fix).
- • "Memory is only 40%, so memory is not our problem" — allocation rate can be causing GC-driven latency at any level.
- • "Virtual size is 14 GB, something is badly wrong" — untouched reservations are free; VSZ is not a resource measurement.
- • "The process restarted, probably a deploy" — check for OOM kills explicitly; silent OOM restarts masquerade as deploys for weeks.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • cgroup `memory.current` / working set against `memory.max`, as a ratio, per container.
- • RSS and runtime heap-in-use as separate series so their divergence is visible.
- • Allocation rate (bytes/s) and GC cycle frequency plus GC CPU fraction, charted next to latency.
- • Page cache versus anonymous memory within the cgroup, so a streaming workload's cache growth is distinguishable from a leak.
- • OOM kill events as a discrete counter — an OOM restart that looks like a deploy in your dashboards is a diagnosis lost.
- • Alert on the cgroup working-set ratio and OOM kill events; delete alerts on VSZ and on host free memory.
- • Chart allocation rate alongside heap level so flow problems stop being invisible.
- • Reduce per-request allocation where the allocation profile points, rather than raising the limit ([[allocation-profiling]]).
- • Right-size the limit using peak working set plus documented [[headroom]] — and treat "raise the limit" as a mitigation, not a fix.
- • For file-heavy workloads, bound your own page-cache growth (direct I/O or explicit hints) if cgroup cache accounting is driving kills.
- • Confirm working-set ratio at peak dropped and stayed down across a full traffic cycle, including any nightly batch.
- • Confirm allocation rate fell and GC frequency fell with it — if GC frequency is unchanged, the allocation fix missed the hot allocation site.
- • Check that p99 improved, since the reason to reduce allocation was latency, not a prettier memory chart.
- • Verify zero OOM kills across a period at least as long as the interval that previously produced them.
- • Reducing allocation often means object pooling or buffer reuse, which trades clarity and safety for throughput and can introduce subtle aliasing bugs.
- • Raising memory limits is instant and costs money forever; it also hides the growth trend that would have identified the real cause.
- • Direct I/O to avoid page-cache accounting gives up the kernel's read caching, which can make disk-bound work substantially slower.
- • An alert on cgroup working set as a fraction of the limit, with a burn-rate style window rather than an instantaneous threshold (Alerts Worth Waking Someone For).
- • A tracked allocation-rate metric with a CI benchmark on the hot path so per-request allocation growth is caught before deploy.
- • A long-running soak test that would expose slow growth that a short load test cannot (Load Test Shapes: The Shape Is the Hypothesis).
- • OOM kills as a first-class alert, never inferred from restart counts.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEAll six readings and the A/B comparison are invented teaching values. Real gaps between heap, RSS and working set depend heavily on the runtime's allocator.
- RUNTIME-SPECIFICWhether freed memory returns to the OS, and how heap-in-use relates to RSS, differs sharply between Go, the JVM, V8, CPython and native allocators.
- ENVIRONMENT-SPECIFICWorking set, page-cache accounting and OOM behavior described here are Linux cgroup semantics. Managed platforms may expose only a subset, under different names.
Misconceptions
memory.max, and it is usually nowhere near the host figure.Apply it
Where the depth lives
Why RSS, VSZ and "free" diverge is a paging-and-reclaim story. This lesson only teaches which number to trust; the mechanism that makes them differ lives in the OS domain.