The question this answers
Which concurrency constructs look correct in review and fail only under production load?
A code review of a service that added concurrency last quarter, and an incident report from the same service this quarter.
Varies. Several of these anti-patterns exist precisely because someone reached for a global as the shortest path to sharing something, and never revisited it once the access pattern changed.
The system continues to make progress, with bounded resource use, under every arrival rate and every interleaving — including the ones that only occur at peak.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The catalogue: why it is tempting, what it produces
Every entry here was written by a competent engineer under time pressure, and every one of them was locally reasonable. That is the point: these are not mistakes of ignorance, they are the shortest correct-looking path from a working single-threaded design to a concurrent one. The failure is always deferred to a load level the author had not seen.
Read the middle column first. If you cannot name why an anti-pattern is tempting, you will not recognise it when you are the one being tempted.
| Anti-pattern | Why it is tempting | What it produces |
|---|---|---|
| One global lock for everything | It is provably correct and takes one line. Every race disappears at once. | Every request serializes through one critical section. Adding cores does nothing, throughput pins to 1/hold-time, and the lock becomes untouchable because nobody knows what it protects — What Contention Actually Costs |
| Unbounded thread creation | A thread per request is the simplest possible model and works perfectly at ten requests. | At ten thousand, memory goes to thread stacks, the scheduler thrashes on context switches, and throughput falls as concurrency rises — Oversubscription |
| Unbounded async concurrency | Promise.all over the array is one line and there are no threads to worry about. | Ten thousand sockets, ten thousand in-flight requests at a downstream sized for fifty, and file-descriptor exhaustion — Unbounded Concurrency |
| Holding a lock across I/O | The lock is already held and the call is right there. Releasing and reacquiring feels like premature optimization. | A microsecond critical section becomes a 40 ms one; the queue multiplies by three orders of magnitude and a convoy forms — Lock Convoys |
| Nested locks in inconsistent order | Each function locks what it needs. Nobody wrote down a global order because no single function needed one. | A deadlock cycle that appears only when two specific paths run simultaneously — reproduced once a month in production, never in CI — Lock Ordering |
| Shared mutable globals | It is available everywhere and needs no plumbing. It was single-threaded when it was written. | Every future concurrent path races on it, and the ownership question has no answer because there is no owner — Shared Mutable State |
| Busy waiting on a condition | A while loop is obvious and needs no primitive. It works on the developer's idle machine. | One core pinned per waiter, other work starved of CPU, and worse behaviour on a loaded machine than an unloaded one — Busy Waiting |
| Blocking the event loop | The function is synchronous and fast enough locally. Making it async is a refactor. | Every other request on that loop stalls for the duration — a 200 ms CPU-bound parse becomes 200 ms added to every concurrent request — Blocking the Event Loop |
| Assuming an operation is atomic | count++ is one expression, and the map is documented as thread-safe. | Read-modify-write across three steps, and thread-safe per call does not make a sequence of calls atomic — lost updates with no error — The Atomicity Illusion |
| Swallowing task exceptions | catch and log keeps the worker alive, which seemed like resilience. | A permanently failing task retried silently forever, or a fire-and-forget promise whose rejection is never observed and whose work never happened — Orphaned Tasks |
| Orphan background tasks | Fire-and-forget is one keyword shorter and the caller does not need the result. | Nobody owns it, nobody cancels it, nobody notices it failed, and shutdown does not wait for it — work lost mid-flight — Structured Concurrency |
| Unbounded retry loops | Retrying until success is the obvious way to be reliable. | Under contention or a downstream outage the retries become the load, and recovery is prevented by the retry traffic itself — Livelock |
| Lock-free without a reason | It sounds faster and the CAS loop looks clever. | Code almost nobody on the team can review, subtle ABA and memory-ordering bugs, and usually no measured improvement over a mutex — Lock-Free Is a Progress Guarantee |
The two that produce the worst incidents
Holding a lock across I/O and unbounded concurrency deserve special treatment, because they are the two that turn a mild degradation into a total outage, and because the code change that causes each is small enough to pass review unnoticed.
The lock-across-I/O version is a one-line move. Someone needs the current user inside the critical section and adds await fetchUser(id) between the acquire and the release. The critical section goes from two microseconds to forty milliseconds. At 500 requests per second, the queue at the lock grows by twenty waiters per second and never drains; every waiter holds a worker and a database connection while it waits, so the connection pool empties before the lock queue does — and the incident presents as "database connection errors", which sends everyone to the wrong system.
The unbounded version is even shorter: await Promise.all(items.map(process)). With ten items it is correct and elegant. With ten thousand, it opens ten thousand connections at once. The first failure is usually not memory — it is EMFILE from file-descriptor exhaustion, or a downstream rate limit returning 429 for every request, or a database refusing connections. The fix is a permit count, and it is three lines. See Bounding Concurrency.
Both share a property that makes them hard to catch: the code is correct. It passes tests, it produces right answers, and its defect is invisible at any load an engineer will produce by hand.
1// 1. Lock held across a network call.2await mutex.acquire()3try {4 const user = await fetchUser(id) // 40 ms, holding the lock5 cache.set(id, user)6} finally { mutex.release() }7 8// 2. Concurrency equal to the input size.9await Promise.all(items.map((i) => process(i))) // items.length = 12,4001// 1. Do the I/O outside; lock only the state mutation.2const user = await fetchUser(id) // nothing held3await mutex.acquire()4try { cache.set(id, user) } // microseconds5finally { mutex.release() }6 7// 2. A ceiling that does not depend on the input.8const sem = new Semaphore(32)9await Promise.all(items.map((i) => sem.run(() => process(i))))The first fix shortens the critical section from a network round trip to a map write, which divides the queue at that lock by roughly 20,000. The second replaces a bound derived from the input size — which is not a bound — with one derived from what the downstream can absorb. Neither change alters a single result the code produces.
Inconsistent lock order, drawn
The deadlock anti-pattern is worth seeing as a graph, because that is exactly how a thread dump presents it and recognising the shape is the skill. Two transfer operations, each locking the source account then the destination account — a rule that is locally sensible and globally fatal, because "source then destination" is a different order for A→B than for B→A.
The four conditions all hold: mutual exclusion (account locks are exclusive), hold-and-wait (each holds one and waits for the other), no preemption (neither lock can be taken away), and circular wait (the cycle in the graph). Break any one and the deadlock cannot occur; in practice you break circular wait by imposing a global order — lock by account id ascending, regardless of which is source and which is destination. See The Four Conditions and Lock Ordering.
The reason this survives review is that neither function is wrong. transfer(A, B) is correct. transfer(B, A) is correct. Only their simultaneous execution is wrong, and there is no line of code to point at. That is the general shape of every entry in this catalogue: the defect is a property of the system, not of a statement.
Key points
- Every anti-pattern here is locally reasonable; that is why it survives review. Learn the temptation, not just the rule.
- The two worst are holding a lock across I/O and unbounded concurrency, because both are one-line changes to correct-looking code.
- A bound derived from the input size is not a bound — it is the input size wearing a limit's clothes.
- Inconsistent lock ordering has no wrong line of code: each function is correct, and only their simultaneous execution is not.
- Swallowed exceptions and orphaned tasks fail silently, which makes them the hardest to find and the easiest to write.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Locate every lock and ask what invariant it protects; a lock whose invariant nobody can name is a global lock in progress.
- • Trace the longest path between an acquire and its release, and check whether any I/O, allocation or second lock lies on it.
- • Find every place concurrency is created and ask what number bounds it; if the answer is the input length, it is unbounded.
- • Find every fire-and-forget call and ask who observes its failure and who cancels it at shutdown.
- • Find every retry and ask what bounds the attempts and what jitters the delay.
- • For each of those, name the failure it produces at ten times current load — that is the review question, not "does this look right".
- • Global lock: 32 threads on 32 cores each take the same lock for 50 microseconds; 31 wait at any instant; measured throughput is identical to one core and CPU sits at 3%.
- • Lock across I/O: T1 acquires and calls a 40 ms API; T2..T20 arrive and block; the pool has 20 connections and all are held by waiters; T21 fails with a connection-pool timeout and the alert says "database".
- • Inconsistent order: T1 locks A, T2 locks B, T1 requests B, T2 requests A — neither proceeds, and both hold a request thread until a timeout fires.
- • Assumed atomicity: T1 reads count (7), T2 reads count (7), T1 writes 8, T2 writes 8 — two increments, one counted, no error, and the counter is a metric so nobody notices for a quarter.
- • Thread-safe map, unsafe sequence: T1 calls map.get(k) -> absent; T2 calls map.get(k) -> absent; both call map.put(k, expensive()); one result is discarded and both callers believe theirs is stored. Every individual call was atomic.
- • Swallowed exception: a worker's task throws, the catch logs at debug level, the loop continues, and the queue drains normally while 100% of the items are silently dropped.
- • Unbounded retry: a downstream returns 503 under load; 4,000 clients retry every 50 ms with no jitter; the retry traffic is now larger than the original traffic and the downstream cannot recover while the retries continue.
- • Busy wait: a waiter spins on a flag for 8 ms; on a machine with more runnable threads than cores, the spinner consumes a full quantum that the thread it is waiting for needed to make progress.
- • A global lock guarantees correctness and guarantees away all parallelism on the paths it covers — it is not wrong, it is a throughput ceiling written as a mutex.
- • A thread-safe collection guarantees each individual operation is atomic; it does NOT guarantee a sequence of operations is, which is where check-then-act bugs live.
- • A try/finally release guarantees the lock is released on the exception path; it does NOT guarantee the state protected by the lock is consistent when the exception fires.
- • Promise.all guarantees all promises are awaited; it does NOT bound how many run concurrently, and it does NOT cancel the others when one rejects — they keep running, unobserved.
- • A retry guarantees another attempt; it does NOT guarantee the system can recover while attempts continue, and unbounded retries actively prevent recovery.
- • Catching an exception in a worker guarantees the worker survives; it does NOT guarantee anyone learns the work failed.
- • Lock-free code guarantees system-wide progress; it does NOT guarantee it is faster than a mutex, and frequently is not.
- • A global lock concentrates all contention into one place, which at least makes it measurable — the convoy is visible in lock wait metrics long before it is visible in throughput.
- • A lock held across I/O multiplies queue length by the ratio of I/O latency to compute latency, typically three to five orders of magnitude.
- • Unbounded concurrency moves contention downstream, where it appears as somebody else's rate limit or connection ceiling and is attributed to their service.
- • Busy waiting converts contention into CPU consumption, so the metric that would have shown you a wait now shows you utilization — and the system looks busy while doing nothing.
- • Oversubscribed threads contend for cores and cache, so context-switch count and cache-miss rate rise while useful work falls.
- • Convoy and throughput collapse from a global lock or a long critical section.
- • Deadlock from inconsistent lock ordering, including the self-deadlock of a non-reentrant lock reacquired on the same thread.
- • Livelock from unbounded retries or from timeout-and-retry deadlock avoidance without jitter.
- • Resource exhaustion: file descriptors, sockets, memory, connections, or thread stacks.
- • Lost update from assumed atomicity or from an unsafe sequence of individually safe calls.
- • Silent data loss from swallowed exceptions and from orphaned tasks killed at shutdown.
- • Event-loop stall, where one CPU-bound handler adds its full duration to the latency of every concurrent request.
- • Starvation and priority inversion under unfair locks.
- • ABA and memory-ordering bugs in hand-written lock-free code, reproducible only on some architectures.
- • A single global lock genuinely helps as a first step: make it correct, measure, then split the lock where the contention actually is. The anti-pattern is leaving it there, not starting there.
- • Thread-per-request is the right model at low, bounded concurrency, and its simplicity is worth real money. The anti-pattern is the absence of a ceiling.
- • Retrying helps for transient faults. The anti-pattern is unbounded, un-jittered retries — see Thundering Herd.
- • Lock-free structures help in genuinely hot, well-understood paths with a measured baseline. The anti-pattern is choosing them first.
- • Whenever the load level at which the construct fails is one you have not tested — which is the defining property of every entry here.
- • Whenever the failure surfaces in a different system from the cause, as lock-across-I/O does when it exhausts the connection pool.
- • Whenever the defect is a property of two correct functions running simultaneously, so there is no line to fix in review.
- • Whenever a bound was chosen from the input rather than from what the constrained resource can absorb.
- • Lock hold time p99 and lock wait time p99, per lock. A hold time in milliseconds is I/O inside a critical section until proven otherwise.
- • Concurrent in-flight operations per downstream, compared against that downstream's stated limit — the number that catches unbounded fan-out before the downstream does.
- • File descriptor count and socket count against the process limit, which is where unbounded async fails first.
- • Event-loop lag (or the equivalent scheduler delay), which detects blocking handlers directly rather than by inference.
- • Unhandled rejection and swallowed-error counts, which are usually available and almost never alerted on.
- • Retry rate as a fraction of total requests, with an alert threshold — a retry rate above a few percent during an incident means retries are part of the incident.
- • Context switches per second and run-queue length, which reveal oversubscription that CPU utilization alone hides.
- • Fixing a global lock means naming the invariants it protected, which is archaeology — the lock outlived the knowledge of why it exists.
- • Adding bounds means choosing numbers and a policy for exceeding them, which is real design work that the unbounded version skipped.
- • Lock ordering must become a documented, enforced global property, and nothing in the type system will help you.
- • Making orphan tasks owned means introducing structured lifetimes and cancellation, which reaches into every caller.
- • Every fix here trades a hidden failure for a visible constraint, and someone will experience the visible constraint as a regression.
- • Fewer concurrent paths. The most reliable fix for most of this catalogue is to have less concurrency, not better-managed concurrency. See Concurrency Is Always Bought With Complexity.
- • Confine state to one owner and pass messages, which eliminates the lock entirely rather than tuning it. See The Actor Model.
- • Use the runtime's structured primitives — task groups, bounded channels, scoped cancellation — instead of hand-assembling from raw pieces. See Structured Concurrency.
- • Let infrastructure own the queue and the retry policy, where dead-lettering and backoff are already implemented and observable.
- • Immutability, which removes the shared mutable global that half of this catalogue is downstream of. See Immutability as a Concurrency Strategy.
if (balance >= 100) withdraw(100) — drive it until it overdraws
balance = 100
withdraw(amount): # both tasks run this concurrently
b = read(balance) # 1
if b >= amount: # 2 <- decided on a value that may already be stale
debit(amount) # 3| # | Withdrawal A (100) | Withdrawal B (100) | State |
|---|---|---|---|
| 1 | rA ← read balance | · | balance=100 paidOut=0 |
| 2 | if rA >= 100 | · | balance=100 paidOut=0 |
| 3 | debit 100 | · | balance=0 paidOut=100 |
Build a deadlock yourself
A global lock order is a proof, not a habit
More workers than cores
What people believe, and what is true
These are beginner mistakes.
Every one of them is a shortest-path change to working code under deadline. They appear in mature codebases written by experienced engineers, because the failure only manifests at a load the author never produced.
Adding a lock is the safe default.
Adding a lock is the correct default; leaving one global lock in place is a throughput ceiling, and putting I/O inside it is an outage. The lock is not the risk — its scope and hold time are.
Promise.all is bounded because the array is finite.
Finite is not bounded. A bound is a number you chose based on what the constrained resource can absorb; the input length is a number your data chose.
Catching exceptions in workers makes the system resilient.
It makes the worker survive. If nothing records that the item failed and nothing routes it anywhere, the queue drains perfectly while every item is dropped — the most convincing possible impression of health.
Go deeper
Overview
Thirteen shapes that look right and fail under load: one global lock, unbounded spawning, locks across I/O, inconsistent lock order, shared globals, busy waiting, blocked event loops, assumed atomicity, swallowed errors, orphan tasks, unbounded retries, and gratuitous lock-free code.
Practical
In review, ask four questions: what invariant does this lock protect, what is on the path between acquire and release, what number bounds this concurrency, and who observes this task's failure. Those four catch most of the catalogue.
Advanced
Treat bounds as design artifacts with stated policies, and treat lock ordering as a global invariant with a documented total order. Both are properties of the system rather than of any function, which is why neither is caught by reading a diff.
Internals
Several entries share one root cause: an operation that is atomic at one level is assumed atomic at another. A thread-safe map call is atomic; get-then-put is not. A CAS is atomic; a CAS loop's effect is not. An assignment may be atomic at bytecode granularity in one runtime and three instructions in another.