intermediate · investigation

The Export That Took Down the API

API p99 latency spikes to 30s and pods restart with out-of-memory kills every morning around 06:00. The pattern started two weeks after signing the largest customer to date.

Evidence

06:00:11 GET /orders/export           tenant=bigco   -> (in flight 94s, then pod OOMKilled)
06:01:45 GET /orders/export           tenant=bigco   -> retry, same result
06:03:20 GET /orders/export           tenant=bigco   -> retry...

handler: SELECT * FROM orders WHERE tenant_id=$1     (no LIMIT, no pagination params accepted)
         -> serialize full result set to one JSON array in memory
bigco order count: 2.4M rows (~3.1GB serialized)   next-largest tenant: 40K rows
client behavior: cron job, retries on failure  -> retry storm every morning

Investigate

Inspect Endpoint contract

`GET /orders/export` accepts no limit, page, or cursor — it promises the entire collection in one response, whatever its size. That promise was fine at 40K rows and is impossible at 2.4M.

Inspect Handler implementation

The handler materializes the full result set and serializes it in memory before writing — memory scales linearly with tenant size until the OOM kill.

Inspect Client retry loop

The customer's cron retries every failure, so each morning the killed request is immediately re-issued — the outage self-sustains until the cron window ends.

Inspect Limits and isolation

No request timeout budget, no response size cap, no per-tenant concurrency limit: one tenant's request class can consume the shared pods.