ProductionAdvanced

Is it a leak? RSS growth diagnosed

“The RSS of a long-running service grows for days until it is OOM-killed. How do you decide whether it is a leak, where the memory is, and what to do about it?”

What this tests

  • RSS vs virtual size vs the managed heap, and what each includes
  • Leak vs unbounded-but-reachable growth vs allocator fragmentation vs page cache accounting
  • Tools per layer: heap snapshots, smaps, tracemalloc, native memory tracking, dmesg
  • Reading the OOM killer’s message and cgroup memory.stat

Answers by level

Read the beginner answer first and notice what is missing.

Start with what is growing. RSS is resident pages: the managed heap *plus* native allocations, thread stacks, mapped files, allocator arenas. Virtual size is reservations and means almost nothing. The managed heap (V8 heapUsed, JVM old gen, Python objects) is what a language profiler sees. If RSS grows and the heap does not, the problem is native; if both grow, it is in the language; if RSS is flat and the container is killed anyway, look at what the cgroup counts (Memory Pressure, Swap and the OOM Killer).

Then decide leak vs bounded. Growth that plateaus at cache size is a cache; growth that is linear in time or in requests and never plateaus is either a leak (unreachable memory the allocator never sees freed — only really possible with manual memory or native code) or, far more often in GC languages, reachable-but-unbounded data: an in-process queue without a cap, a cache without eviction, listeners added per request, closures capturing large objects, a Map keyed by session id. Two heap snapshots an hour apart, diffed by retained size per constructor, usually point at it directly (Chrome DevTools for Node.js, jmap -histo:live, tracemalloc for CPython).

Native growth has its own suspects: glibc malloc arenas in a multi-threaded process (RSS well above live data; MALLOC_ARENA_MAX=2 or jemalloc is the standard remedy for Java, Ruby and Python services), ArrayBuffer/direct ByteBuffer pools, thread counts (each stack’s touched pages), and mmaped files. /proc/<pid>/smaps_rollup, pmap -x and the JVM’s -XX:NativeMemoryTracking=summary attribute it (label: Linux). CPython adds a false positive: small-object arenas that are rarely returned, so RSS stays at its high-water mark after a large temporary structure is freed — fragmentation that looks like a leak (What Happens When I Allocate Memory?).

What to do depends on the class: bound the queue or cache and add backpressure; fix the reference that keeps objects alive; switch allocator or cap arenas; or, if the growth is page cache under a cgroup limit, understand that the kernel will reclaim it and the kill came from something else. In every case: expose RSS, heap and queue-depth metrics and alert on slope, so the next one is caught in days, not at the OOM kill.

Green flags · Red flags

Strong green flag · Asks "heap or native?" first and knows the glibc arena / CPython high-water-mark false positives.
Green flags
  • Separates RSS, virtual size and the managed heap
  • Distinguishes leak, reachable-unbounded growth, fragmentation and page-cache accounting
  • Names snapshot diffing and native attribution tools
  • Reads the OOM message / memory.stat
  • Proposes bounds and backpressure, not just "find the leak"
Red flags
  • Assumes GC languages cannot leak, or that all growth is a leak
  • Equates VSZ with memory use
  • First action is adding RAM or a nightly restart

Follow-up questions

F1
RSS grows 2 GB, V8 heapUsed is flat at 300 MB. Where do you look?
F2
Container killed at 4 GB; anon-rss in dmesg says 1.2 GB. What happened?
F3
How do you tell a cache from a leak in a heap snapshot diff?

Scenario

A Python ingestion worker in Kubernetes with a 3 GB limit is OOM-killed every ~30 hours. tracemalloc shows Python objects at 400 MB and stable. Where is the memory, and what three things would you check?

Learn this topic