The LLM is unavailable.Investigate with fundamentals.
DatabaseIntermediate

Database CPU is at 100%

Page: Postgres primary CPU pinned at 100% since 09:40. p95 for GET /accounts/:id went from 8 ms to 1.9 s; the connection pool is reporting "waiting for connection" on every service that touches accounts. No deploy is on today’s release calendar.

What you know

  • Postgres 15, one primary and one streaming replica; accounts has 3.1 M rows, id bigint PRIMARY KEY. The table fits comfortably in shared_buffers.
  • The account service uses an ORM; a handful of hot paths use hand-written SQL. Both hit accounts by primary key constantly — roughly 1 000 lookups per second at this hour.
  • The release calendar is for feature deploys. Dependency-only PRs go out with the nightly rollout and are not on it.
  • Available: pg_stat_statements, a slow-query archive with yesterday’s plans, pg_stat_activity, pg_stat_user_tables, replica lag, and disk I/O dashboards.
-- pg_stat_statements, last 30 min, ordered by total_exec_time desc
calls       mean_ms   total_s   rows/call   query
1 842 310    412.6    760 200   1.0         SELECT … FROM accounts WHERE (accounts.id)::text = $1
1 840 977      0.4        736   4.2         SELECT … FROM sessions WHERE account_id = $1
   61 233      3.1        190   1.0         UPDATE accounts SET last_seen_at = $1 WHERE id = $2

-- EXPLAIN (ANALYZE, BUFFERS) of the top query, run by hand
Seq Scan on accounts  (cost=0.00..91442.00 rows=1 width=212) (actual time=0.031..401.887 rows=1 loops=1)
  Filter: ((id)::text = '48213'::text)
  Rows Removed by Filter: 3118442
  Buffers: shared hit=52461
Planning Time: 0.09 ms
Execution Time: 401.912 ms

-- the same lookup yesterday, from the slow-query archive
Index Scan using accounts_pkey on accounts  (cost=0.43..8.45 rows=1 width=212) (actual time=0.019..0.020 rows=1 loops=1)
  Index Cond: (id = '48213'::bigint)
Execution Time: 0.038 ms

-- pg_stat_activity, summary
state    count
active     371
idle        19
max_connections = 400

Investigate

For each area: first say why you would check it (reveal the reasoning), then look. Commit to a root cause when you are confident.

Top queries by total time
EXPLAIN of the top query
Recent migrations and ORM upgrade
Connection count
Vacuum and bloat
Replica lag
Disk I/O