Low CPU, High Latency: Lock Contention
The database is 20% busy and every request takes four seconds. Nothing is overloaded — transactions are standing in line for the same rows. This is the shape that defeats capacity-based reasoning, because adding hardware makes the queue longer, not shorter.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The shape: a queue nobody provisioned
Contention converts concurrency into a queue. Twenty transactions want to update the same row; one holds the lock and nineteen wait. The CPU has nothing to do while they wait, so utilization drops, while latency for every one of those transactions grows with its position in line. Throughput does not scale with added capacity because the constraint is not capacity — it is a serialization point, and every additional concurrent request lengthens the line it must join.
This is why the usual reflex fails so completely. Doubling the instance size gives the waiting transactions faster hardware to wait on. Adding application replicas increases arrival rate at the same contended row, which makes latency worse in direct proportion. The only fixes that work reduce either how long the lock is held or how many transactions want it — and both are application-shaped changes, not infrastructure ones.
The diagram is the whole story. A single hot row, an update that holds its lock for the duration of a transaction that also makes a network call, and a queue that grows with traffic. The moment the transaction boundary includes something slow and external, the lock hold time is set by that external service, and the database becomes a queue for it.
Reading the signals that stay green
The panel below is what makes this hard. Seven readings look fine or ambiguous, and the diagnosis rests on two: lock wait time and the count of sessions blocked. If those two are not on a dashboard, this incident is diagnosed by someone remembering to look for it, which is not a strategy.
Note the throughput reading in particular. Throughput *fell* while arrival rate rose, which is the signature that separates contention from ordinary load. Under ordinary load, throughput rises until it plateaus at capacity. Under contention it plateaus early and can invert, because longer queues mean more transactions holding resources while waiting, and in the worst case deadlock detection and retry work starts consuming real capacity.
The second diagnostic move, once lock waits are confirmed, is to identify the blocking statement rather than the blocked ones. Every engine exposes this — a blocking-tree view, pg_locks joined against pg_stat_activity, sys.innodb_lock_waits. The blocked statements are the symptom and they are numerous and identical. The blocking statement is the cause and there is usually exactly one shape of it.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Application CPU | 18% | Idle. Threads are blocked, not computing. | normal |
| Database CPU | 22% | Idle. Rules out "we need a bigger instance". | normal |
| Disk read latency | 0.3 ms | Storage is fine. | normal |
| Buffer hit ratio | 99.4% | Working set is in memory. | normal |
| Request p99 | 4.1 s (was 240 ms) | The symptom. | suspect |
| Throughput | 38 req/s (was 61) | Fell while arrival rate rose — capacity limits plateau, contention inverts. | suspect |
| Lock wait time (avg per txn) | 3.6 s | Almost the entire request duration is spent waiting for a lock. | smoking gun |
| Sessions blocked | 19 of 20 | One transaction holds; the rest queue behind it. | smoking gun |
| Deadlocks detected | 4 / min | Secondary effect — lock ordering under contention. Rollbacks add retry load. | suspect |
Shorten the hold, or stop competing
Every real fix does one of two things: reduce how long the lock is held, or reduce how many transactions want the same lock. The first is usually a transaction-boundary problem, and the single highest-value change is removing external calls from inside transactions. A payment call inside a transaction sets your lock hold time to the payment provider's p99, which means their bad afternoon becomes your outage. Do the external work first, then open a short transaction to record the result — and make that safe against retries with an idempotency key (Idempotency on the architecture side, Idempotency vs Deduplication on the contract side).
The second family reduces competition for the same row. Sharding a hot counter into N sub-counters summed on read turns one lock into N. Moving from an in-place counter to an append-only log with periodic aggregation removes the contended update entirely. Reordering statements so all transactions take locks in the same order does not reduce waiting but does eliminate deadlocks, which removes the retry amplification on top.
Two changes that look like fixes and are not, both worth naming because both get shipped: raising the connection pool size increases the number of transactions that can queue simultaneously, making latency worse while the dashboard shows more "active" connections. Lowering the isolation level can genuinely reduce lock scope for some workloads, and it also changes correctness guarantees — that is a data-integrity decision that belongs with the people who own the invariant, not a performance knob (Isolation Levels).
| Mechanism | Change | Effect on the queue | What it costs |
|---|---|---|---|
| Shorter hold | Move external calls (payments, email, third-party APIs) outside the transaction | Hold time drops from ~420 ms to ~4 ms; queue drains | Needs idempotency and a compensation path for partial failure |
| Shorter hold | Narrow the transaction to the statements that need atomicity | Proportional reduction in hold time | Careful review of what the invariant actually requires |
| Less competition | Shard a hot counter into N rows, sum on read | One lock becomes N; contention divides | Reads get more expensive; N must be chosen and maintained |
| Less competition | Append-only writes with periodic aggregation | The contended update disappears | Aggregation job, storage growth, read-time freshness gap |
| Less competition | Queue the mutation and serialize it deliberately in one worker | Contention becomes a managed queue with visible depth | Asynchrony: the caller no longer gets a synchronous result (The Backlog Arithmetic: Four Levers and a Drain Time) |
| No deadlocks | Acquire locks in a consistent order across all code paths | Waiting unchanged; deadlock rollbacks and retries eliminated | Discipline that must survive every future code change |
| Not a fix | Increase pool size | More transactions queue at once; latency rises | Looks like capacity work, is contention amplification |
| Not a fix | Bigger database instance | Faster hardware to wait on | Money, and a false conclusion recorded in the postmortem |
Key points
- Contention converts concurrency into a queue: latency grows with queue position while CPU falls, because waiting consumes no compute.
- Falling throughput under rising arrival rate distinguishes contention from ordinary capacity limits, which plateau instead of inverting.
- Lock wait time and blocked-session count are the only two readings that identify it; every utilization dashboard stays green.
- External calls inside transactions set lock hold time to the external service's latency — the highest-leverage fix is removing them.
- Raising the pool size or the instance size makes contention worse, and both are the reflexive response to the symptom.
Progressive depth
Overview
When transactions want the same rows, they queue. Waiting uses no CPU, so utilization falls while latency rises — the one shape where every capacity dashboard says "healthy" during a severe outage. Look at lock wait time and blocked sessions, not CPU.
Practical
Confirm with two readings: lock wait time per transaction, and blocked-session count. Then find the *blocking* statement, not the blocked ones. The usual root cause is a transaction boundary that encloses something slow — an external API call, a large batch, or an ORM session left open across a request. Shorten the hold or reduce competition; never respond by adding connections or hardware.
Advanced
Throughput inverting under rising load is the diagnostic signature, and it comes from the same queueing behavior as Queueing: Why Systems Get Slow Before They Get Broken: as utilization of a serialization point approaches one, waiting time grows without bound. Deadlock rate is a second-order effect — more concurrent lock holders means more chances for inconsistent acquisition order, and each rollback adds retry work that increases arrival rate at the contended resource. That feedback loop is the same shape as Retry Storms: The Load You Generated Yourself.
Internals
The lock manager maintains a hash table of lock objects with wait queues and a waits-for graph for deadlock detection — see The Lock Manager and Deadlock Detection: The Waits-For Graph. Under MVCC, readers usually take no locks at all because they read a snapshot (MVCC Internals: Version Chains and Snapshots), which is why read-heavy workloads can be contention-free while a single hot update path serializes everything. Row locks in PostgreSQL are stored on the tuple itself with a multixact fallback when several transactions lock the same row, and the escalation behavior differs sharply from engines that lock at page or index-range granularity (Isolation Levels: The Mechanism Behind Each).
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1User → checkout: p99 rises from 240 ms to 4.1 s during a flash sale; error rate is flat, so nothing is failing, only waiting.
- 2Dashboards → dead end: application CPU 18%, database CPU 22%, disk and buffer ratios normal — no resource is saturated.
- 3Lock view → evidence: average lock wait per transaction is 3.6 s of a 4.1 s request, and 19 of 20 sessions are blocked.
- 4Blocking tree → cause: one statement holds a row lock on the hot SKU while its transaction awaits a 420 ms payment provider call.
- 5Cause → root cause: the transaction boundary encloses an external network call, so lock hold time equals the provider's latency and the queue grows with every arriving request.
- • "CPU is 20%, the database is healthy." Low CPU is the expected reading during contention, because blocked work consumes none.
- • "Latency is high, we need a bigger instance." Faster hardware does not shorten a queue whose length is set by lock hold time.
- • "Add connections so more requests can be served." More connections means more transactions in the same queue and worse latency.
- • "Throughput dropped, so traffic dropped." Arrival rate rose; served throughput fell. That inversion is the diagnosis.
- • "We are seeing deadlocks, so the problem is deadlocks." Deadlocks are usually a secondary effect of heavy contention plus inconsistent lock ordering.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Average and p99 lock wait time per transaction, and the count of currently blocked sessions, as first-class dashboard series.
- • The blocking statement, from the engine's blocking-tree view (`pg_locks` joined to `pg_stat_activity`, `sys.innodb_lock_waits`) — the cause, not the numerous identical victims.
- • Transaction duration split into lock-wait time and execution time, so "the transaction is slow" separates into two different problems.
- • Deadlock rate and transaction rollback/retry rate, which quantify the amplification sitting on top of the base contention.
- • Throughput against arrival rate on one chart: the inversion is the signature.
- • Move external calls out of transactions: perform the network operation first with an idempotency key, then open a short transaction to record the outcome.
- • Narrow transaction boundaries to exactly the statements that must be atomic, and audit for accidental long transactions (an ORM session held open across a request).
- • Reduce competition on hot rows: shard counters, switch to append-only writes with aggregation, or route the mutation through a deliberately serialized worker.
- • Enforce consistent lock ordering across code paths to eliminate deadlock rollbacks and their retry amplification.
- • Revisit isolation level only with the owners of the invariant — it changes correctness, not just performance ([[isolation-levels]]).
- • Lock wait time per transaction falls and blocked-session count returns to near zero, measured over a window with comparable arrival rate.
- • Throughput now rises with arrival rate instead of inverting — run the load test that reproduced the inversion and confirm the curve changed shape.
- • Request p99 improves at the same concurrency; if p99 improved only because traffic fell, nothing was fixed.
- • Deadlock rate drops to zero after lock-ordering work, tracked separately from the latency improvement.
- • Moving external calls out of transactions requires idempotency and a compensation path for the case where the call succeeds and the write fails.
- • Counter sharding makes reads more expensive and adds a shard count that must be maintained as traffic changes.
- • Serializing mutations through a worker makes contention visible and manageable, and converts a synchronous response into an asynchronous one.
- • Consistent lock ordering is a discipline with no runtime enforcement; it degrades quietly as the codebase grows.
- • An alert on lock wait time as a fraction of transaction duration — it fires while CPU dashboards are still green.
- • A lint or review rule forbidding external network calls inside transaction blocks, backed by a test that fails the build.
- • A load test at production-like concurrency against the hot entity, asserting throughput scales rather than inverts.
- • A monitor on longest-running transaction age, which catches accidentally long transactions before they become an incident.
Accuracy
Performance numbers are conditional. These are the conditions.
- DATABASE-SPECIFICLock granularity, escalation and MVCC behavior differ substantially. PostgreSQL readers do not block writers; some engines and isolation levels behave differently, which changes which workloads contend at all.
- ILLUSTRATIVEAll readings are a constructed incident shape. The 420 ms provider call and 19-of-20 blocked sessions are teaching numbers.
- WORKLOAD-SPECIFICContention depends on key distribution. The same code is contention-free with evenly spread keys and pathological with one hot SKU — see Hot Keys: When Aggregate Metrics Hide a Saturated Node.