Backend Races
Two requests, one row: where concurrency bugs actually live in a backend, and why they never appear in development.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Two requests read the same row, both decide it is fine, and both write. Which one is wrong?
A concert has 100 seats. When the hundredth is sold, no more may be sold. It must hold when everyone refreshes at the moment tickets are released.
Read the current count. If it is under 100, insert the booking and increment the count. The check is right there above the write.
Two requests read 99 within microseconds of each other. Both see room, both insert, and 101 seats are sold. Neither request did anything wrong in isolation.
- Two requests read 99 within microseconds of each other. Both see room, both insert, and 101 seats are sold. Neither request did anything wrong in isolation.
- It never reproduces locally. One developer, one browser, one request at a time — the interleaving that breaks it requires two requests overlapping inside the same window.
- It gets worse as the product succeeds: the window is fixed, and the probability of two requests landing inside it rises with the request rate.
- Adding a mutex fixes it on one instance and does nothing once a second instance is deployed, because the lock is inside the process (Stateless Services).
- The
SELECTand theINSERTinside one transaction still overlap at read-committed isolation: two transactions can both read 99 and both commit, because neither wrote a row the other read (Isolation Levels).
What is actually happening
- A backend process serves many requests at once. Every request handler is therefore running concurrently with other copies of itself, over shared state — the database rows, the cache entries, the counters.
- The general shape of every bug in this module is read, decide, write, with a gap between the read and the write. Another request can write into that gap, so the decision is made on state that is no longer true when the write lands (Finding the Critical Section).
- The gap does not need to be large. It exists whenever the read and the write are separate operations, even microseconds apart, and the rate of collisions is set by the request rate against that window.
- Concurrency in a backend crosses process boundaries. Two requests may be on different instances, in different containers, in different availability zones. Any coordination has to happen somewhere both can see — which in practice means the database (A Mutex on Server A Does Nothing About Server B).
- Runtime model changes what races look like but not whether they exist. A single-threaded event loop still yields at every
await, so two handlers interleave at exactly the points where they touch the database (Blocking the Event Loop). - There are only four responses, and the rest of this module is those four: make it one atomic statement, detect the conflict with a version, prevent the overlap with a lock, or design the state so overlap is harmless.
Read, decide, write — and the gap in the middle
Almost every backend race has the same three-step shape. Read the current state. Decide something based on it. Write the consequence. The bug is that between step one and step three, another request can complete all three steps, so the decision was made against state that no longer holds.
Recognising the shape is most of the skill. It appears as a balance check before a debit, a capacity check before a booking, an existence check before an insert, a status check before a transition, a rate-limit count before an increment. Different domains, one bug.
Once you see the shape, the four available responses become obvious, and choosing between them is the rest of this module. Collapse the three steps into one statement. Detect that the state changed since you read it. Prevent anyone else reading it while you decide. Or change the data model so the decision does not depend on a value someone else can move.
Four responses, and when each is right
The four mechanisms are not alternatives in the sense that any would do. Each fits a specific combination of conflict frequency, how much logic the decision requires, and how expensive a retry is.
The order to consider them in is roughly the order of cost. A single atomic statement is cheapest and constrains you to what the database can express. Optimistic concurrency handles arbitrary application logic and costs a retry per conflict, so it is right when conflicts are rare. Pessimistic locking costs throughput on the locked row and is right when conflicts are common or the work is expensive to redo. Redesigning the data so nothing conflicts is the best answer and is rarely available.
How often do they actually collide, and how much logic does the decision need?
when The decision is expressible as a WHERE clause: decrement if positive, transition if in the expected state.
cost Logic must fit in SQL; you lose the ability to inspect intermediate values in application code (Atomic Operations).
when Conflicts are rare, the decision needs real application logic, and retrying is cheap.
cost A version column, a retry loop, and a conflict surfaced to the caller when retries are exhausted (Optimistic Concurrency).
when Conflicts are frequent, or the work between read and write is too expensive to repeat.
cost Serialised access to the row, a held connection, and deadlock risk if lock order is inconsistent (Pessimistic Locking).
when The contention is on an aggregate: a counter, a total, a balance.
cost A schema change — append to a ledger and sum, or shard the counter — plus reads that now aggregate (Atomic Operations).
when Last-write-wins is genuinely acceptable: a display preference, a cached view, a non-authoritative field.
cost You must be certain, and the certainty decays as the field acquires new callers.
Where backend races actually appear
Abstract discussion of interleavings makes this feel theoretical. In practice the same handful of endpoints race in every product, and knowing the list is worth more than knowing the theory — the theory lives in the Concurrency & Parallelism domain and is worth reading there (Reasoning About Races: A Method, Not an Instinct).
Note how many of them are check-then-act on something the database could enforce directly. That is the most common avoidable version: an invariant expressed in application code where a constraint would express it once, correctly, for every code path.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Book the last seat | Capacity exceeded | Count read, then insert | Conditional update on remaining capacity, or a unique constraint per seat (Atomic Operations) |
| Deduct from a balance | Balance goes negative | Balance read, then written | UPDATE ... SET balance = balance - $1 WHERE balance >= $1 and check rows affected |
| Redeem a single-use coupon | Used many times in one burst | Usage checked in application code | Unique constraint on (coupon, user), or an atomic state transition |
| Create a user by email | Two accounts, same email | SELECT existence, then INSERT | Unique index; treat the violation as the duplicate path (Database Constraints) |
| Edit a shared document | One editor's change silently disappears | Both loaded, both saved whole objects | Version column; zero rows updated means conflict (Optimistic Concurrency) |
| Claim a job from a queue table | Two workers run the same job | Selected pending, then updated to running | UPDATE ... WHERE state = 'pending' returning the row, or FOR UPDATE SKIP LOCKED (Pessimistic Locking) |
| Increment a rate-limit counter | Limit exceeded during bursts | Read, compare, write | Atomic increment, and compare the returned value (Rate Limit Algorithms) |
| Retry of a payment | Charged twice | Two executions of one intent | Atomic claim on an idempotency key (The Idempotency Key Flow) |
How to build it
Most important first.
- Find the critical section first: the read that informs a decision, and the write that acts on it. If they are separate statements, you have a race and the only question is how often (Finding the Critical Section).
- Prefer collapsing it into a single statement the database evaluates atomically —
UPDATE ... SET n = n - 1 WHERE n > 0decides and acts at once (Atomic Operations). - Where the decision needs application logic, use a version column and treat zero rows updated as a conflict (Optimistic Concurrency).
- Where conflicts are frequent and retrying is expensive, take a row lock and hold it for the shortest possible span (Pessimistic Locking).
- Push invariants into database constraints wherever they can be expressed there. A unique index enforces "one booking per seat" against every code path, including the ones written next year (Database Constraints).
- Never coordinate in process memory. A module-level
Map, an in-process mutex or a local counter is invisible to every other instance (Stateless Services). - Test for it deliberately: fire N concurrent requests at the endpoint and assert on the resulting state. A single-threaded test suite proves nothing here (A Test Strategy Chosen by What Each Layer Can Prove).
What can go wrong
- Lost update: two requests read a record, each modifies a different field, and the second write overwrites the first field with a stale value. No error, and the data is quietly wrong.
- Double-spend: two requests both find sufficient balance and both deduct.
- Phantom insert: a uniqueness check in application code passes for both requests because neither has committed yet, and two "unique" rows are created (Concurrency Anomalies).
- A fix that moves the race rather than closing it — wrapping the read and the write in a transaction without raising the isolation level or taking a lock, which changes nothing at read-committed.
- A distributed lock in Redis with a TTL shorter than the work it protects, so the lock expires mid-operation and a second holder appears.
- Retry-on-conflict without a bound, turning a contention spike into a retry storm (Retry Storms).
- Read-modify-write on a shared row — the canonical lost update (Optimistic Concurrency).
- Check-then-act on existence, capacity or state, where the check and the act are separate statements (Duplicate Detection).
- Two requests inserting a row that should be unique, neither visible to the other until commit.
- A cache read and a database write interleaving so the cache is repopulated with a value that is already stale (Cache Invalidation).
- Request and retry of the same intent running simultaneously (Idempotency in Backends).
- Races are exploitable on purpose. Submitting a coupon, a withdrawal or a redemption many times in parallel is a standard technique, and the balance check that looks fine sequentially is bypassed by concurrency.
- Rate limits implemented as read-check-write have the same race, so a burst can exceed the limit by the width of the window (Rate Limit Algorithms).
- Any "check the user has not already done this" logic in application code is a race unless the database enforces it. Enforce uniqueness with a constraint, not with a
SELECT(Database Constraints). - The window is small and attackers automate. "It would need perfect timing" means "a script will hit it within a minute".
- "It is in a transaction, so it is safe." A transaction gives atomicity and a consistent read view. At read-committed it does not stop two transactions reading the same value and both writing (Isolation Levels).
- "Node is single-threaded, so there are no races." One thread means no data races on memory. It does not mean no races on the database, because every
awaitis an interleaving point (Blocking the Event Loop). - "We have a mutex." In-process mutexes coordinate one process. Backends are deployed with several (A Mutex on Server A Does Nothing About Server B).
- "It has never happened." It has never been noticed. Lost updates leave no error, only wrong data.
- "Serializable isolation makes it go away." It converts silent corruption into serialization failures you must catch and retry. That is a large improvement and it is not free (Isolation Levels: The Mechanism Behind Each).
Operating it
- Count business invariant violations directly: bookings beyond capacity, negative balances, duplicate rows that should be unique. That query is the only thing that measures the outcome rather than the mechanism.
- Track update statements affecting zero rows on tables with version columns; that count is your conflict rate (Optimistic Concurrency).
- Database deadlock and lock-wait metrics, which rise when contention shifts from lost updates to blocking (Hold Time, Wait Time, and the Ratio Between Them).
- Correlate incidents with concurrency: if the bug appears only at peak, the window is being hit more often, which is itself the diagnosis (Correlation Ids That Survive Every Hop).
- Collision probability rises with concurrent requests touching the same row, not with total traffic. A hot row at 100 rps is worse than a cold table at 10,000 (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- At 10x, races that "never happened" happen daily. Nothing changed except the number of samples drawn from the same distribution.
- At 100x, per-row contention becomes the ceiling, and the fix stops being a lock and becomes a data model change — sharding a counter, appending to a ledger instead of updating a total (Atomic Operations).
- More instances make in-process coordination worse in a specific way: it appears to work on one instance and degrades silently as you scale out.
- Every mechanism here costs something: atomic statements constrain the logic to what SQL can express, optimistic concurrency costs retries, pessimistic locking costs throughput, constraints cost schema changes.
- Designing for concurrency up front adds complexity to endpoints that may never see contention. The judgement is about which rows are hot, not about the whole schema.
- Correctness under concurrency generally costs latency: an atomic single-row update serialises access to that row, which is the point and also the bottleneck.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALRead-decide-write over shared state races in every language and every framework.
- DATABASE-SPECIFICPostgres uses MVCC with row-level locks and detects deadlocks by waiting for
deadlock_timeoutthen running a cycle check; its default read-committed re-evaluates theWHEREclause after a blocking write releases. InnoDB also uses MVCC but takes next-key (gap) locks on indexed ranges under repeatable-read, so it blocks phantom inserts that Postgres permits at the same nominal isolation level, and it detects deadlocks immediately from its wait-for graph. The same application code has different anomaly and blocking behaviour on the two engines. - RUNTIME-SPECIFICNode interleaves handlers only at
awaitpoints on one loop thread, so shared in-memory state is safe between awaits and unsafe across them; a JVM or Go service runs handlers on real threads, so in-memory state needs actual synchronisation. Both are equally exposed to races on the database, which is where backend races almost always are.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — concurrency testing as a first-class practice: N parallel requests against one endpoint, asserting on final state rather than on responses.