The LLM is unavailable.Investigate with fundamentals.
Operating SystemsAdvanced
Memory usage keeps increasing
Page: catalog-api pods are OOM-killed every 5–7 hours. Working-set memory climbs in a straight line from 380 MiB after start to the 2 GiB limit, then the pod restarts and the line starts again. Traffic has been flat for a month.
What you know
- Node.js 20, 6 replicas, container limit
memory: 2Gi, started with--max-old-space-size=1536. Restarting a pod resets it completely. - Three weeks ago a "response memoization" middleware was added to cut CPU on
GET /products/search, keyed by the request URL. - The mobile client’s HTTP layer appends a cache-busting
_=<timestamp ms>parameter to every request it makes. - Available: container memory and OOM events,
--trace-gcoutput in the pod logs, a heap snapshot taken at t+4h from one pod, socket and file-descriptor counts.
# kubectl describe pod catalog-api-6d9f-x7k2
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Restart Count: 4
Limits: memory: 2Gi
# node dmesg
Memory cgroup out of memory: Killed process 31877 (node) total-vm:2412304kB, anon-rss:2084112kB
# container_memory_working_set_bytes, one pod (MiB)
t+0h 380 t+1h 640 t+2h 910 t+3h 1 190 t+4h 1 470 t+5h 1 750 t+5h40 OOM
# --trace-gc, t+4h (heap before -> after, in MB)
[Mark-Compact (reduce) 1462.1 (1498.3) -> 1458.9 (1497.6) MB, 812 ms]
[Mark-Compact (reduce) 1466.4 (1502.0) -> 1463.0 (1501.5) MB, 831 ms]
# heap snapshot, t+4h, sorted by retained size
Constructor Count Shallow Retained
Map 1 48 B 1 291 MB ← middleware/memo.js const cache
(string) 1 482 913 … 612 MB
Object 1 482 902 … 540 MB
# sample of keys in cache
"GET /products/search?q=lamp&_=1724570311204"
"GET /products/search?q=lamp&_=1724570311391"
"GET /products/search?q=lamp&_=1724570312007"
# inside the pod, t+4h
$ ss -s | head -2
Total: 118
TCP: 61 (estab 42, closed 12, orphaned 0, timewait 7)
$ ls /proc/1/fd | wc -l
88Investigate
For each area: first say why you would check it (reveal the reasoning), then look. Commit to a root cause when you are confident.
Container memory limit and OOM events
GC logs
Heap snapshot and allocation profile
Connection and file-descriptor leak
The caching middleware code
Log buffering
Heap vs RSS