Stack & HeapIntermediate
RSS climbs 200 MB an hour until the OOM killer visits
Symptoms
- The ingest service’s RSS grows linearly and it is restarted every ~18 hours with exit code 137.
dmesgshows the OOM killer choosing the process; heap-used in the runtime’s metrics grows at the same rate as RSS.- Growth rate tracks the incoming message rate, not wall-clock time.
- It began after the downstream database migration two weeks ago; the service code has not changed.
$ ps -o pid,rss,vsz,nlwp,cmd -p 4242 # Linux, sampled hourly PID RSS VSZ NLWP CMD 4242 1180432 2913220 11 node ingest.js # 09:00 4242 1391860 3124512 11 node ingest.js # 10:00 4242 1602004 3335808 11 node ingest.js # 11:00 $ dmesg -T | grep -i oom | tail -1 [Tue 03:12:41] Out of memory: Killed process 4242 (node) total-vm:4210344kB, anon-rss:3902116kB, file-rss:2048kB, shmem-rss:0kB # heap snapshot diff, 10:00 vs 11:00 (Chrome DevTools, sorted by retained size delta) Constructor #New #Deleted Size delta Retained by Array 1 0 +198 MB pendingWrites (closure in BatchWriter) Object 1,410,338 1,102,400 +197 MB (elements of the above) Buffer 12 10 +0.3 MB (closure) 4 4 +0 B
Investigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
Native memory / allocator fragmentation
Page cache accounting in the cgroup
Heap snapshot diff
Producer and consumer rates
GC configuration
Event listener leak