The question this answers
The slow query finished ten minutes ago — why is the lock still backed up?
Sixteen worker threads taking a session-cache mutex on every request, one of which held it for 400 ms during a garbage-collection pause.
A session cache and its mutex. The lock is normally held for about 500 nanoseconds, which is why nobody expected it to be a bottleneck.
The session cache reflects every completed session write — untouched throughout, in both the healthy and the convoyed state. The invariant this lesson is about is a performance one: the lock's throughput should recover to its uncontended value once the slow holder leaves. A convoy is the failure of that recovery, and the persistence is the whole phenomenon.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Formation, and the part that persists
Convoy formation is the easy half. A holder stalls — a GC pause, a page fault, a descheduled thread, a slow syscall someone left inside the lock — and every arriving thread queues. Sixteen threads arrive during a 400 ms stall, and all sixteen are on the wait queue when the holder finally releases.
The hard half is what happens after. Before the stall, the sixteen threads were spread randomly in time: each did its own work for a few microseconds, took the lock briefly, and moved on, so arrivals were uncorrelated and the lock was almost never contended. After the stall, all sixteen are released from the same wait queue in quick succession — and they are now *in phase*. Each one takes the lock, does the same short amount of work, and comes back at nearly the same offset as its neighbours. The queue re-forms from threads that are no longer arriving randomly.
That is the convoy: a self-sustaining lockstep pattern that outlives its cause. Throughput stays depressed and lock-wait p99 stays elevated for seconds or minutes after the GC pause is a distant memory, which is why the incident timeline never lines up and the investigation goes looking for a second cause that does not exist.
The lockstep cycle, step by step
The trace below shows why the pattern sustains itself. Follow T2: it wakes, spends its 500 ns in the critical section, does 3 µs of unshared work, and requests the lock again — by which time T3, T4 and T5 have all queued in the interval it was away. It never finds the lock free again.
The self-sustaining mechanism has two ingredients. First, the *synchronised release* correlates the threads' phases. Second, each handoff costs a wakeup — a few microseconds — while the critical section costs 500 nanoseconds, so the lock is idle for most of each cycle and the queue drains at a rate set by wakeup latency instead of by work. Effective throughput drops by roughly the ratio of the two, and it stays there as long as the phases remain correlated.
This also explains the counter-intuitive interaction with Fairness. A strictly fair FIFO lock *guarantees* the lockstep — every thread is served in rotation, so the correlation never decays. A barging lock breaks it faster, because a running thread that reuses the lock without a handoff perturbs the rotation. The convoy is one of the few places where unfairness is actively the better behaviour.
| # | T2 | T3 | T4 | T5 | State |
|---|---|---|---|---|---|
| 1 | woken from the queue (3 µs wakeup); enters CS | · | · | · | t=0 µs queue=T3,T4,T5 lock idle since=3 µs |
| 2 | CS done in 0.5 µs; releases; wakes T3 | · | · | · | t=0.5 µs queue=T3,T4,T5 |
| 3 | does 3 µs of unshared work | · | · | · | t=3.5 µs queue=T3,T4,T5 T3 state=still being scheduled |
| 4 | · | finally scheduled at t=3.5 µs; enters CS | · | · | t=3.5 µs lock was idle=3 µs of the last 3.5 |
| 5 | · | · | requests the lock — queues behind T3 | · | t=3.6 µs queue=T4 |
| 6 | finishes its work and requests the lock again — queues behind T4, T5 | · | · | · | t=3.6 µs queue=T4,T5,T2 effective rate=~0.28 M/s vs 2 M/s uncontended ✕ Throughput recovery: the queue is the same length it was one cycle ago, with the same threads in the same relative order. The pattern reproduces itself with no external cause. |
| 7 | · | · | · | requests again — the rotation is now stable | queue=T5,T2,T3 convoy age=seconds to minutes |
Recognising it in the metrics, and what to do
The distinguishing signature is *hysteresis*: throughput does not recover when the cause disappears. If lock-wait p99 spikes when a GC pause occurs and returns to baseline within a second, that is ordinary contention. If it spikes and then stays elevated for a minute with unchanged offered load, that is a convoy, and looking for a second cause during that minute is wasted effort.
The second signature is the ratio: acquisitions per second far below 1 / hold_time, with the lock spending most of its time free. A lock held for 500 ns that is only acquired 280 000 times a second is idle 86% of the time while threads wait for it, which is the arithmetic in the trace above, and no amount of ordinary contention produces that pattern.
The fixes are the ones you would expect, plus one that is specific to convoys. Never let a lock holder do anything that can stall — the GC pause is not preventable, but an I/O call, an allocation that may trigger collection, or a syscall inside a critical section are. Reduce acquisition frequency by batching, so the threads have less opportunity to correlate. And if the platform allows it, prefer a barging lock over a strictly fair one for hot, short critical sections, because fairness is what preserves the lockstep.
time gc_pause offered_rps completed_rps lock_wait_p99 lock_held_pct ctx_switches/s t=00 no 12 000 11 980 0.1 ms 0.6% 4 100 t=10 no 12 000 11 970 0.1 ms 0.6% 4 050 t=11 YES 12 000 0 410 ms 100.0% 900 <- formation t=12 no 12 000 1 800 38 ms 9.1% 212 000 <- cause is OVER t=20 no 12 000 2 100 33 ms 9.4% 238 000 <- still convoyed t=45 no 12 000 2 050 34 ms 9.3% 241 000 <- no recovery t=70 no 12 000 11 940 0.1 ms 0.6% 4 200 <- desynchronised read it as: lock_held_pct 9% + lock_wait_p99 34 ms => the lock is FREE 91% of the time and threads are still waiting 34 ms for it. That combination is only produced by handoff latency, i.e. a convoy — not by a busy lock. ctx_switches/s x58 is the same fact from the scheduler's side.
Key points
- A convoy forms when one holder stalls, and persists because the released threads are now phase-correlated and keep re-forming the queue.
- Throughput during a convoy is set by wakeup latency per handoff, not by the critical section — often an order of magnitude below the uncontended rate.
- The diagnostic signature is hysteresis: elevated lock wait continuing after the cause is gone, with unchanged offered load.
- The metric combination "lock held a small percentage of the time" plus "high lock wait" means handoff latency, which only a convoy produces.
- Strict fairness preserves the lockstep; barging perturbs it. This is one of the few cases where an unfair lock is the better behaviour.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • A holder is descheduled or stalls (GC pause, page fault, preemption, syscall) while inside the critical section.
- • Every thread that needs the lock during the stall parks, so a queue forms whose length is arrival rate times stall duration.
- • On release, waiters are woken in sequence; each pays a wakeup latency before it can enter, so the lock is idle most of each cycle.
- • Because they were released together, the threads' subsequent request times are correlated, and each finds the lock held when it returns.
- • The pattern persists until something breaks the correlation — barging, variability in the unshared work, or a change in arrival rate.
- • Healthy: T2 takes the lock at a random offset, holds 500 ns, releases; T3 arrives 40 µs later and finds it free. No queue, no wakeups, ~2 M acquisitions/s.
- • Formation: T1 stalls for 400 ms holding the lock; T2–T16 arrive and park; on release, fifteen threads are queued and in phase.
- • Sustained: T2 wakes (3 µs), runs the CS (0.5 µs), works (3 µs), re-requests, and finds T4, T5 ahead of it. The lock was idle for 3 of every 3.5 µs.
- • Breaking it by barging: T7, already running on another core, acquires without a wakeup and completes two sections in the time one handoff would take, perturbing the rotation.
- • Breaking it by batching: each thread now takes the lock once per 100 operations, so the correlated arrivals are spread over 100× more time and the queue drains between them.
- • A mutex guarantees exclusion, and that is all. Nothing in its contract says throughput recovers after a stall, and no mainstream implementation attempts to desynchronise waiters.
- • A fair lock guarantees bounded waiting and, in doing so, guarantees the rotation is preserved — it makes convoys more stable, not less.
- • Shortening the critical section guarantees a higher uncontended ceiling. It does not guarantee a convoy will not form, because a convoy is caused by a stall, not by hold time.
- • Batching guarantees fewer acquisitions. It does not guarantee shorter holds — a batched critical section is longer, which raises the stall exposure per acquisition.
- • Nothing guarantees the convoy ends at a predictable time. Its lifetime is a property of how quickly the phase correlation decays, which is not a number you can compute in advance.
- • The queue length is arrival rate times stall duration, so a 400 ms stall on a lock taken 12 000 times a second parks every thread in the pool.
- • Each handoff costs a park/unpark round trip; at a 3 µs wakeup and a 0.5 µs section, roughly 86% of the lock's time is unusable idle.
- • Context-switch rate explodes — often by one to two orders of magnitude — because every acquisition now involves the scheduler. See The Cost of a Context Switch.
- • The convoy interacts with thread-pool sizing: a larger pool means more threads to correlate and a longer queue, so "add threads" makes it strictly worse. See More Threads Is Not More Speed.
- • Throughput collapse that outlives its cause, producing an incident timeline where the trigger and the symptom do not line up.
- • Repeated convoys from a periodic stall (GC every 30 s), where the system never fully recovers between events and looks permanently degraded.
- • Latency amplification into upstream timeouts, retries and a Thundering Herd that re-feeds the convoy.
- • Misdiagnosis as capacity: adding instances or threads increases the number of correlated participants and deepens the convoy.
- • Pool exhaustion when all workers are parked on the lock, so the service stops accepting work entirely. See Pool Saturation.
- • Nothing about a convoy helps — but recognising it saves an investigation, because it explains why the metrics do not correlate with any live cause.
- • The analysis is generally useful: any resource with a queue and a per-handoff cost can convoy, including connection pools, single-threaded executors and semaphore-guarded sections.
- • A convoy is also a useful signal that a lock is acquired far too often; the fix (batching, shorter holds) improves the healthy case as well.
- • A convoy is pure loss. Its practical harm is that it survives the fix for its trigger, so a team believes the GC tuning did not work when it did.
- • It hurts most on locks with very short critical sections, where the handoff-to-work ratio is worst — precisely the locks nobody thought needed attention.
- • It hurts capacity planning, because the degraded throughput is not a function of load and does not respond to scaling.
- • Lock wait p99 plotted against the suspected cause on the same axis. Elevated wait continuing after the cause ends is the definitive convoy signature.
- • Lock held-percentage alongside lock wait. Low held-percentage with high wait means handoff latency, and that combination is diagnostic.
- • Context-switch rate (
vmstat,pidstat -w,perf stat -e context-switches). A 10–100× jump concurrent with a throughput drop is a convoy in the scheduler's language. - • Acquisitions per second against
1 / measured_hold_time. A large gap between actual and theoretical means the lock is idle while threads wait. - • Whether any stall can occur inside the critical section at all — audit for allocation, I/O, syscalls and anything that can trigger collection. That audit is worth more than any metric.
- • Batching to reduce acquisition frequency introduces staleness and a flush policy, and makes each critical section longer — a real trade, not a free win.
- • Choosing an unfair lock to break convoys reintroduces the starvation risk that fairness was there to prevent, so the two decisions must be made together. See Fairness.
- • Removing every possible stall from a critical section can require pre-allocating buffers and avoiding any library call whose implementation you do not know.
- • Diagnosing convoys requires metrics most services do not have — lock held-percentage in particular — so there is instrumentation work before there is any fix.
- • Reduce acquisition frequency by an order of magnitude via batching or per-thread buffers, which reduces both contention and correlation. See What Contention Actually Costs.
- • Replace the lock with a lock-free or atomic structure for very short critical sections, removing the handoff entirely. See Atomics: What Is Actually Indivisible, with Atomics Are Not Magic as the caution.
- • Hand the state to a single owner reached by a queue, so there is no lock to convoy on and the queue depth is an explicit, observable design parameter. See Producer / Consumer.
- • Shard so that sixteen threads correlate on sixteen different locks instead of one; the convoy fragments into sixteen much smaller ones.
What people believe, and what is true
The convoy ends when the slow operation ends.
That is the defining surprise. The released threads are phase-correlated and keep re-forming the queue; recovery takes as long as the correlation takes to decay, which can be orders of magnitude longer than the stall.
A fair lock will help, because everyone gets a turn.
Strict FIFO preserves the rotation exactly, which is what sustains the convoy. Barging perturbs the phases and is usually how a convoy actually breaks.
The lock must be held a lot if threads are waiting this long.
In a convoy the lock is free most of the time — threads are waiting on wakeup latency, not on the holder. That is why held-percentage plus wait-time together identify it and neither does alone.
Go deeper
Overview
One thread stalls while holding a lock and everyone queues. When it recovers, the queue does not disperse: the threads are now in step and keep rebuilding it, so the slowdown outlives its cause.
Practical
Look for lock wait that stays high after the trigger ends, with flat offered load and a huge jump in context switches. Fix by removing anything stallable from the critical section and by acquiring far less often — batch.
Advanced
A convoy is a synchronisation phenomenon in the physical sense: independent oscillators coupled through a shared resource fall into phase. That framing predicts the fixes — anything that decouples the phases (jitter, batching, barging, sharding) works, and anything that enforces order (strict fairness) makes it worse.
Internals
The reason handoff dominates is that a park/unpark round trip involves a futex-style syscall, a scheduler decision, and a cache-cold restart of the woken thread, while the critical section may be a single map insert. Adaptive mutexes exist precisely for this: spinning for a short bounded period before parking keeps the fast path off the scheduler entirely, which both raises uncontended throughput and reduces convoy severity. See Spin Locks and Busy Waiting.