Memory Leaks: Growth That Does Not Come Back
Stable workload, rising memory, and a sawtooth of OOM restarts. Confirming a leak takes a trend under steady load; finding it takes two heap snapshots and a diff of what is still reachable.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The shape: a ramp, and then a sawtooth
A leak has an unmistakable signature once you look at the right window. Over minutes it is invisible. Over hours you see a ramp: memory rising roughly linearly while the request rate is flat. Over days you see a sawtooth — ramp, OOM kill, restart at baseline, ramp again — with a period that shortens as traffic grows, because a busier process reaches the ceiling faster.
The controlling comparison is memory against *load*, not memory against time. Memory that rises with traffic and falls when traffic falls is a working set, not a leak. Memory that rises during a flat overnight period, or does not return to baseline after a spike subsides, is retention. This is the distinction that decides everything downstream, and it is worked through properly in Leak or Unbounded Cache? The Question That Picks the Fix.
Before the kill, users already feel it. As the live set grows, the collector runs more often and walks more objects each time, so GC CPU and pause time rise together (Garbage Collection: Pause, Throughput, Footprint — Pick Two). The latency damage is concentrated in the tail, which means a service can be visibly degrading for hours while p50 and the memory alert both stay quiet.
mem OOM
2.0G ┤ ╭─╮ ╭─╮ ╭─╮ ← kill
│ ╭──╯ │ ╭──╯ │╭──╯ │
1.5G ┤ ╭──╯ │ ╭──╯ ╰╯ │
│ ╭──╯ │ ╭──╯ │
1.0G ┤ ╭──╯ │ ╭──╯ │
│ ╭──╯ ╰──╯ ╰─
0.5G ┤ ╭──╯ ← restart baseline
└──┴────┴────┴────┴────┴────┴────┴────┴────┴───
Mon Tue Wed Thu Fri
rps ────────────────────────────────────────────── flat
Rising memory + flat load = retention.
Shortening sawtooth period = the leak rate is traffic-proportional.Finding it: two snapshots and a diff
A single heap snapshot tells you what is in memory, which is rarely enough — plenty of large things are supposed to be there. What identifies a leak is the *difference* between two snapshots taken far enough apart, under the same workload: the object classes whose retained count grew, and the reference path keeping them alive.
The workflow is mechanical. Take a snapshot after warm-up, run steady load for long enough that the ramp is clearly visible (an hour is often enough, and the ramp slope tells you how long), take a second snapshot, and diff by retained size and instance count. Then follow the dominator path from the grown objects to a GC root. The answer is almost always one of a handful of shapes: an unbounded collection used as a cache, a registry or listener list that is added to but never removed, closures capturing a large context, or a native resource with a finalizer that never runs.
Two practical notes. First, snapshots are expensive — they often pause the process and can be gigabytes — so take them off a canary instance, not the one serving your most sensitive traffic. Second, if you cannot take heap dumps in your environment, allocation profiling over the same window is a weaker but usable substitute: it names the sites that allocate the most, and the leaking allocation site is very often among them (Allocation Rate Is a Cost Even Without a Leak).
1# 1. CONFIRM it is retention, not workload2plot(working_set) vs plot(request_rate) over >= 6h3 rising memory + flat load -> retention4 memory tracks load, returns down -> working set, stop here5 6# 2. CAPTURE on a canary instance, not a hot one7snapshot_A = heap_dump() # after warm-up8run_steady_load(60.min)9snapshot_B = heap_dump()10 11# 3. DIFF by retained size, not shallow size12diff = compare(snapshot_A, snapshot_B)13diff.sort_by(retained_size_delta).top(20)14 15# 4. FOLLOW the reference path to a GC root16# "who is still pointing at these?"17 SessionContext[] +1.2 GB +214,000 instances18 <- HashMap "activeSessions"19 <- static SessionRegistry.INSTANCE <- GC root20 21# 5. ASK the two questions that name the bug22# Is anything ever removed from activeSessions?23# Is there a bound or an eviction policy? -> no, and noWhat it costs before the kill
Teams often treat a leak as a scheduled-restart annoyance: memory grows, the pod recycles, traffic drains, nobody notices. That framing understates it in three ways, and the signals show all three.
The tail degrades continuously as the live set grows, so users get slower for hours before each restart, with the damage concentrated in p99. The restart itself is not free: in-flight requests are lost, caches are cold, and the runtime pays warm-up cost again (JIT and Warm-Up: The First Thousand Requests Are a Different Program) — so the minutes after a restart are also degraded. And an OOM kill is an *uncontrolled* stop: no graceful drain, no flush, and whatever the process was holding is gone.
There is also a diagnostic cost that compounds. A process that restarts regularly resets its own evidence, so the trend that would identify the leak keeps getting erased, and the restart looks enough like a deploy that it is routinely misattributed for weeks. Recording OOM kills as an explicit event, separate from restarts, is a small change that repeatedly saves large investigations — as the memory-leak incident in the incident library plays out.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Working set / limit | 1.62 GB / 2.0 GB (81%) | Below a typical 85–90% alert threshold, and rising steadily. The trend is the finding, not the level. | suspect |
| Request rate | 1.2k rps (flat 8 h) | Workload has not changed. Growth cannot be attributed to load. | smoking gun |
| GC cycles | 31/min, up from 9/min | A larger live set means more frequent, longer collections. | smoking gun |
| GC CPU fraction | 14%, up from 3% | One core in seven is now collecting garbage instead of serving requests. | smoking gun |
| p50 latency | 52 ms (up from 49 ms) | Essentially unchanged. This is why nobody has noticed. | normal |
| p99 latency | 840 ms (up from 210 ms) | Requests that land in a collection pause pay for it. The harm is already happening. | smoking gun |
Key points
- A leak is memory rising while load is flat — the comparison against load is what makes it a leak rather than a working set.
- The sawtooth of ramp-kill-restart is the signature at day scale; at minute scale a leak is invisible.
- Two heap snapshots diffed by retained size, then the reference path to a GC root, names the bug. One snapshot rarely does.
- The cost arrives before the OOM: rising GC frequency degrades p99 for hours while p50 and memory alerts stay quiet.
- Record OOM kills as an explicit event — restarts that reset the evidence are why leaks survive for weeks undiagnosed.
Progressive depth
Overview
A leak is memory the program will never use again but can still reach, so the collector cannot reclaim it. You see it as memory rising while traffic stays flat, ending in an out-of-memory kill and a restart.
Practical
Confirm by plotting working set against request rate over six or more hours — flat load with rising memory is retention. Then take two heap snapshots an hour apart on a canary, diff by retained size, and follow the reference path from the grown objects to a GC root. The bug is usually an unbounded collection, a registry that is added to but never cleaned, or a closure capturing request state.
Advanced
The harm precedes the kill. As the live set grows, collection frequency rises and each cycle scans more, so GC CPU climbs and pause-sensitive requests move into the tail. That means a leak is a latency incident hours before it is an availability incident, and the alert that catches it early is on working-set *slope* plus GC frequency, not on a level threshold. Where growth is legitimate but unbounded, the distinction in Leak or Unbounded Cache? The Question That Picks the Fix decides whether the fix is deletion or a bound.
Internals
Reachability is the whole definition: tracing collectors free what is not reachable from a root, so a "leak" in a managed runtime is always a live reference someone forgot. That is why the dominator path — the object whose removal would make the whole subgraph unreachable — is the useful view rather than a flat histogram. Below the runtime, growth becomes an OS story: anonymous pages accumulate, reclaim cannot help because none of it is file-backed, and the cgroup OOM killer selects by usage against memory.max. Fragmentation adds a further wrinkle: a runtime can hold a large RSS with a small live set because free objects are scattered across arenas whose pages cannot be returned. See Memory Pressure, Swap and the OOM Killer and Paging for the kernel half.
Leak, Cache or Working Set?
Change an input and watch which number moves — and which one does not.
Memory rises with no relationship to the traffic cycle and never levels off. Under stable load this is retention: something keeps a reference it should have dropped. The tell is that a full daily cycle passes and the floor is higher than yesterday.
You cannot tell these apart from a one-hour window, which is exactly why memory alerts based on a threshold produce so many false pages. The diagnostic question is not "how high" but "does it come back down".
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Request → registry: each request registers an object (session, listener, cache entry, context) in a long-lived structure.
- 2Registry → GC root: the structure is reachable from a static or global root, so nothing it holds is ever collectable.
- 3Live set → collector: the collector must walk an ever-larger live set, so cycles get more frequent and each one longer (Garbage Collection: Pause, Throughput, Footprint — Pick Two).
- 4GC pauses → tail latency: requests unlucky enough to land in a pause pay it, so p99 rises while p50 barely moves.
- 5Working set → limit → OOM killer: the process crosses
memory.max, is killed without draining, restarts cold, and the ramp begins again.
- • "Memory is high, that is the leak" — a high level is normal; a rising trend under flat load is the leak.
- • "It restarts every few days, we will just schedule a restart" — that hides the leak and keeps paying the tail-latency and cold-start cost.
- • "The heap dump shows a huge byte array, found it" — large is not the same as leaking; only the growth between snapshots is evidence.
- • "The restarts line up with deploys" — check OOM kill events explicitly; the correlation is often coincidence, and this misattribution is the most common time sink here.
- • "Memory grew after the traffic spike and stayed up, so it leaked" — some runtimes retain freed pages by design; check live heap, not RSS (Reading Memory: RSS, Heap, Working Set and the Number on Your Dashboard).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Working set plotted against request rate over at least 6 hours, ideally spanning an overnight low-traffic period.
- • The same trend across restarts, so the sawtooth period is visible and its slope can be compared to traffic.
- • GC cycle frequency, GC CPU fraction and pause duration — the leading indicators of harm before the kill.
- • Two heap snapshots on a canary instance, diffed by retained size and instance count.
- • OOM kill events as a discrete counter, never inferred from restart counts.
- • Bound the retaining structure: a size or TTL limit with eviction converts unbounded retention into a working set with a ceiling ([[leak-vs-cache-growth]]).
- • Remove the registration on completion — deregister listeners, close and release contexts, clear per-request state in a `finally`.
- • Break the reference from the long-lived root, or hold it weakly if the runtime supports weak references.
- • If the leak is in a dependency you do not control, isolate it: restart on a schedule *as an explicit mitigation with a ticket*, not as the answer.
- • Raise the limit only to buy investigation time, and say so out loud — it moves the OOM later, it does not remove it.
- • Run steady load for at least as long as the previous ramp took to become visible, and show the working-set trend is now flat.
- • Confirm GC cycle frequency and GC CPU are stable rather than rising over that window.
- • Take a third heap snapshot after the fix and confirm the previously growing class no longer grows between snapshots.
- • Confirm p99 stays flat across the whole window — that was the user-visible harm, and it is the actual proof.
- • Bounding a cache introduces evictions, and evictions mean recomputation — you trade memory safety for a lower hit rate ([[cache-performance-signals]]).
- • Weak references make lifetime implicit and can produce surprising recomputation or subtle behavior differences under memory pressure.
- • Heap snapshots pause the process and can be large; capturing them on a serving instance is itself a small outage.
- • Scheduled restarts are cheap and effective as mitigation, but they mask the trend and add cold-start cost on every cycle.
- • An alert on working-set *slope* under stable load, not only on the level — a level alert fires hours after the trend was diagnosable.
- • A soak test in CI or nightly: the same load for hours, failing if memory grows beyond a small tolerance (Load Test Shapes: The Shape Is the Hypothesis).
- • OOM kills alerting as a distinct, never-silenced event.
- • A code-review habit for the shapes that leak: registries, listener lists, caches without eviction, closures capturing request-scoped state.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe four-day chart, the signal readings and the snapshot diff are invented to show the shape. Ramp slope and sawtooth period depend entirely on leak rate and limit.
- RUNTIME-SPECIFICHeap snapshot tooling, dominator analysis and weak-reference semantics differ substantially between the JVM, V8, Go, CPython and .NET. The workflow generalizes; the tools and the terminology do not.
- ENVIRONMENT-SPECIFICOOM kill behavior, whether you get a graceful signal first, and whether snapshots are permitted at all depend on the platform and the cgroup configuration.