The question this answers
Seven of my eight threads are waiting on one lock — what is the cost, and which of my options actually reduces it?
Eight request-handling threads recording per-route metrics into one shared hash map, guarded by one mutex, on an eight-core machine.
A single HashMap<Route, Counter> and the mutex protecting it. Every request from every route touches it, which is the entire problem — the sharing is global while the actual conflicts are per-route and rare.
Each route's counter equals the number of requests recorded for that route. That invariant is per-route; the lock is per-map. The gap between the granularity of the invariant and the granularity of the lock is the definition of unnecessary contention, and closing it is the whole of technique two.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Seven waiting, one working, and a CPU graph that says "idle"
The picture below is what a contended lock looks like from above. At any instant exactly one thread is inside the critical section and up to seven are parked on the mutex's wait queue. The total useful work being done is one core's worth, on an eight-core machine, and — crucially — the machine reports low CPU utilisation, because blocked threads consume none.
That is the first thing to internalise: contention shows up as idleness, not as load. A team looking at CPU graphs concludes it has headroom and adds traffic, which adds threads, which lengthens the queue, which increases latency without increasing throughput at all. Little's Law makes this precise — see littles-law — and the visible symptom is the p99 climbing while p50 and CPU stay flat.
The second thing is the arithmetic. If a critical section takes 2 µs and every request needs it once, one lock can serve at most 500 000 requests per second no matter how many cores you own. That number is a hard ceiling set by the serial section, and it is the concrete form of Amdahl's argument — see Amdahl's Law. Adding the ninth thread does not raise it; it only makes the queue longer.
Three reductions, in order of how much they buy
There are exactly three things you can do about contention, and they are not equivalent. Shrink the critical section, shard the lock, or remove the sharing. They attack different terms — hold time, conflict probability, and the existence of the resource — and they compose, but their leverage is wildly different.
Shrinking is the first move because it is usually free and often enormous. The commonest form of unnecessary contention is work inside the lock that did not need to be there: formatting a log line, allocating, computing a hash, or — catastrophically — performing I/O. Moving a 40 µs serialisation step outside a critical section that guards a 200 ns map insert raises the ceiling by a factor of two hundred with no design change at all.
Sharding attacks conflict probability rather than hold time. Sixteen locks selected by hash(route) % 16 means two threads conflict only when they hit the same shard, so contention drops by roughly the shard count until you hit a hot key that all traffic funnels into — at which point sharding buys nothing and you are back to shrinking. Removing the sharing is the strongest and the least often available: per-thread accumulators reduced periodically, an immutable snapshot, or a design where the state is owned by one task and reached by message.
| Move | Term it attacks | Typical gain | Where it stops working | What it costs |
|---|---|---|---|---|
| Shrink the critical section | Hold time — the serial fraction itself. | Often 10–100× when there is I/O, allocation or formatting inside the lock. | Once the section is just the minimal state mutation, there is nothing left to remove. | Usually nothing. Sometimes a reconciliation step, because state can change between the two regions. See Finding the Critical Section. |
| Shard the lock | Conflict probability — how often two threads want the same lock. | Roughly the shard count, if keys are evenly distributed. | Hot keys. If 80% of traffic is one route, sharding by route gives you nothing. See Hot Keys: When Aggregate Metrics Hide a Saturated Node in Observability & Performance. | Any operation spanning shards (a total, a resize, an atomic snapshot) must take all locks in order — reintroducing Lock Ordering obligations. |
| Remove the sharing | The resource — there is no lock to contend on. | Unbounded; scaling becomes linear in the ideal case. | When the invariant genuinely requires a single consistent view across threads. | Memory (one accumulator per thread), staleness (readers see the last snapshot), and a merge step whose ordering may not be deterministic. See Reduction Ordering: The Sum Changed When the Worker Count Did. |
| Read/write lock | Conflict probability for read-heavy access only. | Large when reads dominate and writes are rare. | Write-heavy loads, and short critical sections where the RW lock's own bookkeeping costs more than a plain mutex. | More complexity, a starvation policy decision, and a slower uncontended path. See Read/Write Locks, Honestly. |
| Add more threads | Nothing. | None — it lengthens the queue. | Always. This is the non-move that teams try first. | Latency, memory, context switches. See More Threads Is Not More Speed. |
What "shrink the critical section" looks like
The example below is the pattern that causes most real contention, and it is almost always written by accident. The lock was placed around a block of code rather than around a piece of state, and over time work accumulated inside the block because that is where the variables were in scope.
Count what the bad version holds the lock for: a timestamp call, a string format, a hash lookup, an increment, and a log write. The log write alone can be tens of microseconds and can block on a pipe. Only the increment needs protection — the counter is the shared state, and the invariant is per-counter.
The good version holds the lock across a single map operation. Everything else moves out, and the log write moves out entirely because it touches nothing shared. This is not a micro-optimisation; on the timeline above it changes the length of every running segment, and therefore the length of every waiting segment, and therefore the throughput ceiling. Measure hold time, not lock count — see Hold Time, Wait Time, and the Ratio Between Them.
1def record(route, status, duration_ms):2 with metrics_lock: # held for ~45 us3 now = time.time() # syscall-ish4 key = f'{route}:{status}' # allocation + format5 counters[key] = counters.get(key, 0) + 1 # <- the only shared write6 durations[key].append(duration_ms)7 if duration_ms > 1000:8 log.warning('slow request %s %.1fms at %s', key, duration_ms, now)9 # a write to a pipe, inside the lock, on the request path10 11# 8 threads x 45 us serial section => ceiling of ~22k records/sec,12# regardless of core count. CPU reads ~12%.1def record(route, status, duration_ms):2 now = time.time() # outside: touches nothing shared3 key = f'{route}:{status}' # outside: local allocation4 slow = duration_ms > 10005 6 shard = shards[hash(key) % 16] # shard: conflicts drop ~16x7 with shard.lock: # held for ~0.3 us8 shard.counters[key] = shard.counters.get(key, 0) + 19 shard.durations[key].append(duration_ms)10 11 if slow: # outside: I/O never under a lock12 log.warning('slow request %s %.1fms at %s', key, duration_ms, now)13 14# Two changes, multiplied: ~150x shorter hold and ~16x fewer conflicts.15# A cross-shard total now needs all 16 locks in a fixed order, or a16# best-effort read that tolerates a slightly inconsistent snapshot.The lock exists to protect the counter, not to protect the function. Every statement inside the critical section that does not touch shared state is pure serialisation tax paid by every other thread. The cost of the fix is real but small: the cross-shard total is no longer atomic, so you must decide whether an approximate sum is acceptable — for metrics it invariably is.
Key points
- Contention presents as low CPU utilisation, not high — blocked threads consume nothing, so the machine looks like it has headroom.
- The throughput ceiling of a contended lock is 1 / critical-section length, independent of core count. Adding threads lengthens the queue, not the ceiling.
- Three real reductions exist: shrink hold time, shard to reduce conflict probability, or remove the sharing entirely.
- Shrinking is usually the biggest win because the commonest cause is work that drifted inside a lock and never needed to be there.
- Sharding is defeated by hot keys, and it creates cross-shard operations that reintroduce ordering obligations.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • A thread requests the mutex; if held, it is placed on the wait queue and taken off the run queue.
- • The holder executes the critical section, whose length sets how often the resource becomes available.
- • On release, one waiter is woken (or barges in — see Fairness); the rest keep waiting, so arrival rate above 1/hold-time grows the queue without bound.
- • Total system throughput for lock-covered work equals 1 / hold time regardless of parallelism available elsewhere.
- • Reducing hold time raises the ceiling proportionally; sharding multiplies the number of independent ceilings; removing the lock eliminates the ceiling for that work.
- • Uncontended: T1 acquires, records, releases before T2 arrives. Lock cost is a few nanoseconds of atomic operations and nothing else — which is why uncontended microbenchmarks say locks are cheap.
- • Contended: T1 holds for 45 µs while T2–T8 park. Each of the seven pays the full remaining hold time plus a wakeup, and the wakeups serialise into a queue.
- • Sharded, distinct keys: T1 takes shard 3 and T2 takes shard 11 at the same instant. Both run in parallel; the interleaving that used to serialise now does not exist.
- • Sharded, hot key: all eight threads hash to shard 3 because 80% of traffic is one route. Identical to the unsharded case, plus the memory of fifteen unused locks.
- • Shrunk: T1 holds for 0.3 µs. T2 arriving during that window usually spins briefly rather than parking, and no queue forms at all — the qualitative change is that waiters stop involving the scheduler.
- • A mutex guarantees exclusive access to whatever the programmer remembered to put inside it. It guarantees nothing about how long that will take.
- • Sharding guarantees per-shard exclusivity. It explicitly does not guarantee a consistent view across shards — a total computed by reading sixteen shards is a smear over time, not a snapshot.
- • Shrinking the critical section preserves the per-item invariant and gives up any invariant that spanned the removed statements. That is a real semantic change and must be checked, not assumed.
- • Low reported CPU guarantees nothing about headroom. Under contention it is a symptom of the ceiling, not evidence of capacity.
- • A read/write lock guarantees concurrent reads. It does not guarantee they are faster than a plain mutex — its bookkeeping is heavier, and for very short sections it frequently loses.
- • The cost per waiter is the remaining hold time plus a park/unpark round trip, which is microseconds even when the critical section is nanoseconds — the wakeup often dominates the work.
- • Queue length grows as arrival rate approaches 1/hold-time, and latency grows with queue length. This is ordinary queueing theory applied to a lock; see
queueing. - • Cache effects compound it: the lock word and the guarded data bounce between cores on every handoff, so a heavily contended lock generates coherence traffic proportional to the handoff rate. See What a Shared Write Costs.
- • Contention is superlinear in thread count in practice — more waiters mean more wakeups, more cache-line transfers and more scheduler work per unit of useful output.
- • Throughput plateau: adding cores or threads produces no additional throughput, and the plateau sits well below hardware capacity.
- • Latency amplification: p99 rises steeply while p50 barely moves, because the tail is the requests that arrived while a long holder was inside.
- • Convoy formation once the queue never drains — see Lock Convoys.
- • Priority inversion when a lock holder is descheduled, which turns a short critical section into a long one for everyone. See Priority Inversion.
- • Capacity mis-planning: low CPU is read as spare capacity, more traffic is routed in, and the system degrades non-linearly.
- • A lock is the right answer when the invariant genuinely spans the data and the critical section is short — the cost is then a few nanoseconds and nothing about this lesson applies.
- • Coarse locking helps early: one lock is easy to reason about, and premature sharding of a lock that is never contended is complexity with no return.
- • Contention itself is a useful signal — it tells you exactly where the serial fraction of your program lives, which is information a profile of CPU time will not give you.
- • When the lock granularity is coarser than the invariant's granularity, as with a per-map lock protecting per-key counters. That mismatch is pure avoidable serialisation.
- • When anything blocking happens inside the critical section, which turns a nanosecond ceiling into a millisecond one.
- • When contention is "solved" by adding threads or machines: the serial section is unchanged, so horizontal scaling multiplies cost without multiplying throughput.
- • Lock wait time as a fraction of request time, and hold-time distribution per lock.
lock-contentionin Observability & Performance covers extracting these; the reason to want them is this lesson. - • Throughput against thread count. If it is flat from four threads upward, you are lock-bound, and the flat value tells you the effective critical-section length: ceiling ≈ 1/hold time.
- • Off-CPU profiling or a wall-clock (not CPU-time) flame graph — a CPU profile of a contended system shows almost nothing, because the interesting time is spent not running. See
flame-graphs. - • Contention counters where the platform provides them:
perf lock, JFR lock events,pthreadmutex statistics, Go's mutex profiler. - • The negative signal that misleads: CPU utilisation. Track it alongside throughput, never alone.
- • Sharding adds a shard count (a tuning parameter), a hash choice, and a rule for every operation that must span shards.
- • Shrinking a critical section frequently requires a reconciliation step for state that can change between the two smaller regions, and that step is genuine design work.
- • Removing sharing adds per-thread state, a merge, and a decision about how stale the merged view may be.
- • Every one of these makes the code less obviously correct than one global lock, which is why the honest starting point is one lock plus a measurement.
- • Per-thread accumulators merged on a timer, which is the standard answer for metrics and removes the lock entirely. See Immutability as a Concurrency Strategy and Parallel Reduce.
- • A concurrent map with internal striping, which is sharding someone else has already implemented and tested. See Concurrent Queues for the same idea applied to queues.
- • Atomic operations for single-word counters, where the invariant fits in one word and no lock is needed at all. See Atomics: What Is Actually Indivisible — and note Atomics Are Not Magic for the multi-word case.
- • Batching: accumulate locally and take the lock once per hundred operations, trading a small amount of staleness for a hundredfold reduction in acquisitions.
Eight threads, one lock
A mutex buys correctness with throughput
How much of the task is inside the lock?
What people believe, and what is true
Low CPU means we have capacity.
Under lock contention, low CPU is the symptom. Blocked threads consume nothing, so a fully saturated lock-bound service can sit at 10% CPU while refusing to go any faster.
Locks are slow.
An uncontended mutex is a handful of nanoseconds. What is slow is *waiting*, which is a property of the critical section's length and the arrival rate, not of the primitive.
Sharding the lock will fix it.
Only if the keys are spread. With a hot key — one route, one tenant, one popular product — every thread hashes to the same shard and you have gained nothing but memory.
Go deeper
Overview
One lock, eight threads: one works and seven wait. Throughput is capped at one over the critical-section length no matter how many cores you buy, and the CPU graph will tell you the machine is idle.
Practical
Measure hold time first. Then move everything out of the critical section that does not touch shared state — especially I/O and logging. Only then consider sharding, and check for hot keys before you do.
Advanced
Contention is the serial fraction in Amdahl's law, made concrete and measurable. That reframing is useful because it tells you the ceiling before you build anything: measure the critical section, invert it, and that is your maximum throughput. Every technique in this lesson either shrinks that fraction or creates more independent instances of it.
Internals
Below the API, a contended lock costs more than the wait: the lock word and the guarded cache lines migrate between cores on every handoff, so a lock handed off a million times a second generates a million coherence transactions. This is why per-thread state with a periodic merge can beat a "cheap" shared atomic counter by a wide margin — the winning move is not a faster lock but fewer shared cache lines. See False Sharing: Different Variables, Same Cache Line.