Leak or Unbounded Cache? The Question That Picks the Fix
Both grow, both end in an OOM kill, and they need opposite fixes. Three questions separate them: does the growth correspond to data you would use again, is there an eviction policy, and does usage stabilize?
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Three questions, and what each answer commits you to
The distinction is not academic, because the fixes point in opposite directions. For a leak, the correct action is to stop retaining — remove the reference, deregister the listener, clear the state. Applying that to a cache destroys a deliberate performance feature. For an unbounded cache, the correct action is to bound it and evict. Applying *that* to a leak just changes how fast you reach the ceiling: an LRU over garbage still evicts, and still fills, and still kills the process on a schedule.
Three questions settle it. Does the retained data correspond to something you would genuinely use again? Session objects for sessions that ended two days ago do not. Is there an eviction policy — any bound at all, by size, count or age? And most decisively: under sustained steady load, does usage plateau? A cache over a finite hot set plateaus at the size of that set. A leak keeps a straight line.
The plateau test needs patience and a stable workload, which is exactly what makes it reliable. It is also why the reading matters more than the argument: run steady traffic for several hours and look. If it flattens, you have a working set and the conversation moves to whether that plateau fits in the limit. If it does not flatten, the eviction policy you were about to add is a delay tactic.
| Question | Leak | Unbounded cache | Healthy cache |
|---|---|---|---|
| Would you use the retained data again? | No — it is unreachable in practice, only reachable to the GC | Yes, but the tail of it is cold and never touched again | Yes, and the hot set is what is kept |
| Is there an eviction policy? | None, and eviction would not help | None — that is the defect | Size, count or TTL bound with an eviction strategy |
| Does usage plateau under steady load? | No — straight line to the limit | No, but the *growth rate* decays as the key space is exhausted | Yes, at roughly the hot-set size |
| Correct fix | Stop retaining: break the reference | Add a bound and an eviction policy | Right-size the bound; nothing else |
| What eviction alone would do | Delay the OOM, hide the trend | Actually fix it | n/a |
| What "just restart it" does | Resets the ramp; buys hours | Resets and re-warms; costs hit rate | Costs hit rate for nothing |
The same code, one word apart
The most common unbounded-cache bug is a plain map used as a cache — the fastest possible thing to write, and correct in every functional test. It is not that the author chose an unbounded cache; it is that nobody chose anything at all, and a map is what was at hand.
The bounded version differs by having answered three questions the unbounded one silently deferred: what is the maximum size, what gets evicted when it is reached, and how long is an entry allowed to remain valid. Those are cache-policy decisions (Caching Patterns) and they belong in the code, not in the OOM killer's hands. An LRU with a size bound implements the first two; a TTL handles the third and doubles as a staleness limit (Cache Invalidation, Stampedes and Hot Keys).
Watch the key space, because it decides whether this is a cache at all. Keyed by product id in a catalog of 50,000, the map plateaus and is a genuine cache. Keyed by request id, session id or a rendered query string, the key space is unbounded and the "cache" is a leak with a lookup method — it will never see the same key twice, so it has a 0% hit rate while consuming all your memory. That case is worth checking first, because it is both common and instantly decisive (A 95% Hit Rate Tells You Almost Nothing).
1const priceCache = new Map<string, Price>()2 3function getPrice(key: string): Price {4 let p = priceCache.get(key)5 if (!p) {6 p = computePrice(key)7 priceCache.set(key, p) // grows forever8 }9 return p10}11 12// If key = productId (50k possible) -> plateaus. A cache.13// If key = `${productId}:${userId}:${ts}` -> unbounded key space.14// Every lookup misses, every lookup inserts.15// 0% hit rate, 100% of your memory. A leak with a get() method.1const priceCache = new LruCache<string, Price>({2 maxEntries: 50_000, // what is the ceiling?3 ttlMs: 5 * 60_000, // how long may an entry be stale?4 onEvict: () => metrics.evictions.inc(), // is the bound biting?5})6 7function getPrice(productId: string): Price { // bounded key space8 return priceCache.getOrCompute(productId, computePrice)9}10 11// Now observable: hit rate, eviction rate, entry count vs maxEntries.12// Eviction rate near zero -> bound is generous, memory is the ceiling.13// Eviction rate very high -> bound too small, you are paying for churn.The bounded version is not merely safer — it is *observable*. Entry count against the maximum, hit rate and eviction rate together tell you whether the bound is right, and none of those numbers exist for a plain map. The unbounded version cannot be tuned because it cannot be measured; its only feedback signal is an OOM kill.
Both end the same way, so bound either way
The pragmatic conclusion is that the diagnosis matters for the *fix* but not for the *urgency*. An unbounded cache and a leak have the same failure mode: growth until the cgroup limit, then an uncontrolled kill, with rising GC cost and a degrading tail on the way (Memory Leaks: Growth That Does Not Come Back). Neither is acceptable in production, and both are correctly treated as defects rather than tuning opportunities.
So the operational rule is simple: every in-memory collection whose size is influenced by traffic or user input needs a bound, and the bound needs a metric. That covers caches, registries, buffers, dedupe sets and retry queues. Where a bound is genuinely impossible, the data does not belong in process memory — it belongs in a store designed for eviction, which is one of the reasons Redis exists (Redis: Data Structures, Not a Cache).
And instrument the bound, not just the memory. Entry count against maximum, hit rate and eviction rate turn "how big should this cache be?" from an argument into a measurement, and they make the next incident a three-panel diagnosis instead of a heap dump.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Working set trend (6 h steady load) | rising, no plateau | Consistent with both a leak and an unbounded cache. Not decisive on its own. | suspect |
| Cache entry count vs max | no maximum configured | There is no bound, so there is no back pressure of any kind. Defect regardless of cause. | smoking gun |
| Cache hit rate | 0.4% | Nearly every lookup misses: the key space is effectively unbounded. This is not functioning as a cache. | smoking gun |
| Eviction rate | 0/s | Nothing has ever been evicted — nothing ever can be. | smoking gun |
| Distinct keys seen / hour | 1.1 M, tracking request count | Key space grows with traffic. Keyed by something request-unique. | smoking gun |
| GC CPU fraction | 12% and climbing | The harm is already being paid in latency, well before the kill. | suspect |
Key points
- The discriminator is whether usage plateaus under sustained steady load: a working set flattens at the hot-set size, a leak does not.
- The fixes are opposite — stop retaining (leak) versus bound and evict (cache) — and applying the wrong one wastes the incident.
- Adding eviction to a leak only delays the OOM; an LRU over garbage still fills and still kills.
- A "cache" keyed by something request-unique is a leak with a lookup method: near-zero hit rate, unbounded key space.
- Every traffic-influenced in-memory collection needs a bound and a metric — a bound you cannot observe cannot be tuned.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Request → cache key: the key includes a request-scoped or user-scoped component, so the key space grows with traffic rather than with the data.
- 2Cache key → map insert: every lookup misses and inserts, so entry count rises monotonically with request count.
- 3Map → live set: the map is reachable from a long-lived root, so nothing it holds can be collected (Memory Leaks: Growth That Does Not Come Back).
- 4Live set → collector: GC frequency and pause time rise, degrading p99 hours before any memory alert fires.
- 5Working set → limit: the process crosses
memory.maxand is killed; the restart resets both the memory and the evidence.
- • "Memory rises then flattens — that is a leak" — a plateau is the signature of a healthy working set, not a leak.
- • "We added an LRU, so the memory problem is fixed" — if the underlying growth is retention, eviction only changes the schedule.
- • "The cache is 4 GB but our hit rate is 96%, so it is earning its keep" — check the hit rate of the *evicted tail*; a bound half the size may hold nearly the same hit rate.
- • "It is just the cache warming up" — warm-up completes. Six hours in, "warming up" is a synonym for unbounded.
- • "Memory did not return after the traffic spike, so it leaked" — some runtimes retain freed pages by design; compare live heap, not RSS (Reading Memory: RSS, Heap, Working Set and the Number on Your Dashboard).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Working set over 6+ hours of steady load, checked specifically for a plateau rather than for a level.
- • Cache entry count against its configured maximum — and whether a maximum exists at all.
- • Hit rate and eviction rate; a near-zero hit rate with a growing entry count identifies an unbounded key space immediately.
- • Distinct keys observed per hour compared to request count — if they track each other, the key space is request-unique.
- • GC CPU fraction and cycle frequency, since both shapes cause latency harm before they cause a kill.
- • Run the plateau test first — several hours of steady load — because it decides which of the two remaining fixes is correct.
- • For an unbounded cache: add a size or entry bound plus TTL, with eviction metrics, and fix the key to a bounded key space.
- • For a leak: break the reference from the long-lived root and remove entries on completion; do not "solve" it with eviction.
- • Move genuinely large caches out of process memory into a store built for eviction ([[redis-data-structures]]).
- • Instrument entry count, hit rate and eviction rate so the bound can be tuned with evidence rather than argued about.
- • Repeat the sustained-load test and show a clear plateau, at a level with headroom against the limit.
- • Confirm hit rate is meaningful (a bounded key space) and eviction rate is non-zero but not churning — eviction rate near the insert rate means the bound is too small.
- • Confirm GC CPU and p99 both stabilized; the memory chart flattening is not by itself proof that users are better off.
- • For a leak fix, verify with a heap snapshot diff that the previously growing class no longer grows.
- • A tighter bound means more evictions, more recomputation and a lower hit rate — you are trading memory safety for backend load.
- • TTLs bound staleness and memory together but cause synchronized expiry, which can produce a thundering herd ([[cache-stampede]]).
- • Moving the cache out of process adds a network hop to every lookup, which can cost more latency than the cache saves for cheap computations.
- • Eviction metrics add series per cache instance — small, but not free ([[cardinality]]).
- • An alert on cache entry count approaching its maximum, and a separate one on working-set slope under stable load.
- • A nightly soak test that fails if memory has not plateaued within a defined window (Load Test Shapes: The Shape Is the Hypothesis).
- • A review rule: any map, set or list whose size is influenced by request content must declare a bound.
- • Hit rate and eviction rate on the service dashboard, so an unbounded key space is visible the day it ships (A 95% Hit Rate Tells You Almost Nothing).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe signal values, hit rates and cache sizes are teaching examples. Real plateau levels depend entirely on hot-set size and key distribution.
- WORKLOAD-SPECIFICWhether a cache plateaus is a property of the key distribution in your traffic, not of the code. The same map is a cache for one access pattern and a leak for another.
- RUNTIME-SPECIFICWhether freed entries return memory to the OS, and how quickly, depends on the allocator; a bounded cache can still show a high RSS after the live set shrinks.