The LLM is unavailable.Investigate with fundamentals.
DatabaseIntermediate
Database CPU is at 100%
Page: Postgres primary CPU pinned at 100% since 09:40. p95 for GET /accounts/:id went from 8 ms to 1.9 s; the connection pool is reporting "waiting for connection" on every service that touches accounts. No deploy is on today’s release calendar.
What you know
- Postgres 15, one primary and one streaming replica;
accountshas 3.1 M rows,id bigint PRIMARY KEY. The table fits comfortably inshared_buffers. - The account service uses an ORM; a handful of hot paths use hand-written SQL. Both hit
accountsby primary key constantly — roughly 1 000 lookups per second at this hour. - The release calendar is for feature deploys. Dependency-only PRs go out with the nightly rollout and are not on it.
- Available:
pg_stat_statements, a slow-query archive with yesterday’s plans,pg_stat_activity,pg_stat_user_tables, replica lag, and disk I/O dashboards.
-- pg_stat_statements, last 30 min, ordered by total_exec_time desc calls mean_ms total_s rows/call query 1 842 310 412.6 760 200 1.0 SELECT … FROM accounts WHERE (accounts.id)::text = $1 1 840 977 0.4 736 4.2 SELECT … FROM sessions WHERE account_id = $1 61 233 3.1 190 1.0 UPDATE accounts SET last_seen_at = $1 WHERE id = $2 -- EXPLAIN (ANALYZE, BUFFERS) of the top query, run by hand Seq Scan on accounts (cost=0.00..91442.00 rows=1 width=212) (actual time=0.031..401.887 rows=1 loops=1) Filter: ((id)::text = '48213'::text) Rows Removed by Filter: 3118442 Buffers: shared hit=52461 Planning Time: 0.09 ms Execution Time: 401.912 ms -- the same lookup yesterday, from the slow-query archive Index Scan using accounts_pkey on accounts (cost=0.43..8.45 rows=1 width=212) (actual time=0.019..0.020 rows=1 loops=1) Index Cond: (id = '48213'::bigint) Execution Time: 0.038 ms -- pg_stat_activity, summary state count active 371 idle 19 max_connections = 400
Investigate
For each area: first say why you would check it (reveal the reasoning), then look. Commit to a root cause when you are confident.
Top queries by total time
EXPLAIN of the top query
Recent migrations and ORM upgrade
Connection count
Vacuum and bloat
Replica lag
Disk I/O