The question this answers
Why do two individually correct atomic operations still break the invariant that spans them?
Transferring 50 from account A to account B, and a reporting thread that reads both balances to check the books still balance.
Two atomic balances, a and b — and, implicitly, the relationship between them that no variable holds.
a + b is constant across every transfer, and neither balance ever goes negative.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Two atomics, one dead invariant
Make both balances atomic. Every read is indivisible, every write is indivisible, no value is ever torn, and a race detector will report nothing. The books still fail to balance, because the invariant was never about a balance — it was about the *sum*, and no location holds the sum.
A transfer is two operations. Between them, the system is in a state that violates the invariant by construction: the money has left A and has not arrived at B. That window is not a bug in the atomics; it is inherent to representing one logical change as two physical ones. Making each half indivisible narrows the window to nanoseconds and leaves it wide open.
The second failure in the same code is check-then-act. if (a >= 50) a -= 50 is an atomic read followed by an atomic write, with a fully preemptible gap between them where another thread can also read, also decide yes, and also subtract. Both threads observed a legal state and both transitions were individually legal; the composition is not. This is the shape Reasoning About Races: A Method, Not an Instinct teaches you to enumerate.
| # | Transfer thread | Reporting thread | State |
|---|---|---|---|
| 1 | atomic fetch_sub(a, 50) -> 200 | · | a=150 b=150 a+b=300 |
| 2 | · | atomic load a -> 150 | a=150 b=150 T2.a=150 |
| 3 | · | atomic load b -> 150 | a=150 b=150 T2.a=150 T2.b=150 |
| 4 | · | report total = 150 + 150 = 300 | T2.total=300 |
| 5 | atomic fetch_add(b, 50) -> 150 | · | a=150 b=200 a+b=350 ✕ The sum is now 350. T1's two atomic operations are correct individually; the pair briefly created — and now permanently reports — money that does not exist relative to the reader's snapshot. |
| 6 | · | [alternative schedule] load a -> 150, load b -> 150 after fetch_sub but before fetch_add | T2.total=250 ✕ The other ordering: the reporting thread sees 250 and the books appear to have lost 50. No thread did anything wrong. |
Drawing the atomicity boundary where the invariant is
The fix is not a better atomic. It is to make the *whole transition* the unit that other threads cannot observe half-done. Concretely: take one lock covering both balances, do both updates, release. The critical section is chosen by the invariant, not by the variables — which is the argument in Finding the Critical Section.
The reporting thread has to participate too. A reader that does not take the lock still sees the intermediate state, so a "safe" writer plus an unsynchronized reader is still a broken system. If the reader only needs an approximate total, say so explicitly and document the tolerance; if it needs a consistent snapshot, it needs the lock or a snapshot mechanism.
There is a narrow case where one atomic genuinely does cover a two-field invariant: pack both fields into one word and CAS the word. A 32-bit count plus a 32-bit version in one 64-bit atomic is the standard trick, and it works exactly as far as the width the platform can CAS. Past that width you are back to a lock, and the packing itself is a maintenance cost — see The ABA Problem: The Value Came Back for what versions are usually for.
1std::atomic<int64_t> a{200}, b{100};2 3bool transfer(int64_t amount) {4 if (a.load() < amount) return false; // check5 a.fetch_sub(amount); // ... and act, in a different instant6 b.fetch_add(amount); // ... and the pair is not atomic either7 return true;8}9 10int64_t total() { return a.load() + b.load(); } // may observe the gap1std::mutex m;2int64_t a = 200, b = 100; // plain ints; the mutex provides both3 // mutual exclusion and the happens-before edge4 5bool transfer(int64_t amount) {6 std::lock_guard<std::mutex> g(m);7 if (a < amount) return false; // check and act are now one indivisible step8 a -= amount;9 b += amount;10 return true;11}12 13int64_t total() {14 std::lock_guard<std::mutex> g(m); // readers must participate15 return a + b;16}The mutex is not protecting a and it is not protecting b. It is protecting the statement "a + b is constant and neither is negative", which is why both the check and both writes are inside it and why the reader takes it too. Note also that the plain int64_t fields are correct here precisely because the mutex supplies the happens-before edge — see Happens-Before: The Edge That Makes a Write Visible.
The patterns that look atomic and are not
Almost every real instance of this bug is one of a handful of shapes. Recognising them by sight is more useful than any general rule, because the general rule — "the atomic unit must be the invariant, not the variable" — is easy to agree with and hard to apply under deadline pressure.
The tell is grammatical: if the specification of the operation contains the words "and", "if", "then", or "unless", it spans more than one instant, and a single atomic cannot express it. "Increment the counter" is one instant. "If the counter is below the limit, increment it" is two.
Note where this reasoning already exists elsewhere in your stack. A database gives you a transaction precisely so a multi-row invariant has an atomic unit — see Isolation Levels and The Database Solves Concurrency For Its Data, Not For Your Memory. An HTTP API gives you If-Match for the same reason. Application memory is the one layer where nobody hands you transactions, so you draw the boundary yourself.
| What the code does | What the atomic gives | What is still broken | What it actually needs |
|---|---|---|---|
counter.fetch_add(1) | The whole increment, indivisibly | Nothing — this is the case atomics are for | Nothing |
if (n < limit) n.fetch_add(1) | An indivisible read and an indivisible write | Both threads read n == limit-1 and both increment; the limit is exceeded | CAS loop on n, or a semaphore — see Semaphores: Counting Permits as a Resource Limit |
a.fetch_sub(x); b.fetch_add(x) | Two correct single-account updates | Any reader between them sees a sum that never legally existed | One lock over both, or a single owning thread |
if (!p.load()) p.store(new T()) | An indivisible pointer read and write | Two threads both see null and both construct; one object leaks or is used by nobody | call_once / a function-local static — see Double-Checked Locking: The Canonical Cautionary Tale |
ratio = hits.load() / total.load() | Two indivisible reads | The two reads are from different instants; the ratio can exceed 1 | A lock, or accept and document an approximate value |
if (q.size() > 0) q.pop() | A consistent size and a correct pop | Another consumer pops in between; this pop hits an empty queue | A try_pop that fuses the test and the removal |
Key points
- Atomics protect locations. Invariants relate locations. When the invariant spans two, the atomics are individually correct and the system is wrong.
- Check-then-act is two instants no matter how atomic each instant is: both threads can observe a legal state and both transitions can be legal while the composition is not.
- The atomicity boundary is chosen by the invariant, not by the variables — which usually means a lock over the whole transition.
- Readers must participate in whatever the writers use, or they observe the intermediate state that the writers were careful to make brief.
- Grammatical tell: if the operation's specification contains "and", "if" or "then", it needs more than one atomic.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Write down the invariant as a sentence about state — "a + b is constant", "in-flight never exceeds the limit".
- • List every location the sentence mentions. If it is more than one, no single atomic can enforce it.
- • Identify the transition that takes the system from one state satisfying the invariant to another, and make that whole transition the unit no other thread can observe half-done.
- • Give every reader of the invariant the same mechanism, because an unsynchronized reader observes exactly the window the writers eliminated for each other.
- • Only if the invariant fits in one CAS-able word — count plus version, head plus tag — can a single atomic replace the lock.
- • T1 fetch_sub(a, 50); T2 loads a and b; T1 fetch_add(b, 50) — T2 reports 250 for a system whose total is 300. Every operation was atomic.
- • T1 checks a >= 50 (true); T2 checks a >= 50 (true); T1 fetch_sub(a, 50); T2 fetch_sub(a, 50) — a is now negative from a balance of 50. Check-then-act with atomic reads and atomic writes.
- • T1 loads hits (=90); T2 records a hit and a request (hits=91, total=100); T1 loads total (=100) — T1 computes 0.90 from state that never coexisted. Harmless here; the same shape decides whether to shed load elsewhere.
- • With one mutex over both: T1 acquires, subtracts, adds, releases; T2 acquires, reads 150 and 150, releases. Every observable state satisfies the invariant. No schedule breaks it.
- • Promises: each atomic operation is indivisible, and no individual value is ever torn or lost.
- • Promises: with a lock covering the whole transition, no thread that also takes the lock can observe an intermediate state.
- • Does NOT promise: that a sequence of atomic operations is itself atomic. There is no composition rule that makes it so.
- • Does NOT promise: that "no data race" means "no race condition". A race detector checks unsynchronized memory access; it cannot know your invariant. See Data Race Is Not Race Condition.
- • Does NOT promise: that narrowing the window helps. A window of ten nanoseconds hit a thousand times a second is hit every day in production.
- • The correct fix — one lock over the transition — serialises the whole transition rather than each variable, so the contended region is larger than the two atomics were. That is the price, and it is the right price.
- • Readers taking the lock turn a read-mostly workload into a contended one; a read-write lock or a snapshot pointer is the usual next step. See Read/Write Locks, Honestly and Copy-on-Write as a Concurrency Strategy.
- • Packing two fields into one CAS-able word keeps the contention at a single line but concentrates it there — every reader and every writer now fight for one line. See What a Shared Write Costs.
- • Lost update — check-then-act where both threads pass the check.
- • Race condition without a data race — every access synchronized, invariant still broken. The most under-diagnosed failure in this module, because every tool reports clean.
- • Torn logical read — a reader assembling a snapshot from several individually atomic reads, producing a state that never existed.
- • Limit overrun — a counter checked against a bound and then incremented, letting N threads past a limit of one.
- • Double initialization — two threads both observing "not yet initialized". See Initialization Races.
- • Recognising the pattern early: the cheapest moment to notice a two-location invariant is before the atomics are written, not after the incident.
- • Code review — "which invariant does this atomic enforce?" is a question that finds this bug in seconds and has no polite way to be answered wrongly.
- • Deciding deliberately that an approximate value is acceptable, and writing that down, so the next reader does not "fix" it into a lock.
- • When the reasoning is applied so broadly that every atomic gets a lock "just in case" — a genuine single-location counter does not need one.
- • When the response is a single global lock over unrelated invariants, converting a correctness bug into a contention bug. See Concurrency Anti-Patterns.
- • When the packed-word trick is used past the point of readability to avoid a mutex nobody measured.
- • Add an assertion that checks the cross-location invariant from a thread that takes the same lock, and run it under stress. An invariant checked only in tests that never interleave proves nothing. See Stress Testing: A Test That Passed Once Proves Nothing.
- • Reconcile against an independent source: sum the accounts against the ledger, compare in-flight counts against the load balancer. Drift that scales with concurrency is this bug.
- • Note explicitly that a thread sanitizer will not find it. TSan reports unsynchronized access; here every access is synchronized. This is the case where tooling gives a false all-clear.
- • Log the pair of values together under the lock rather than separately, so the log itself cannot show a state that never existed.
- • You accept a real critical section with a real contention cost, in exchange for an invariant you can state and check.
- • Every reader of the invariant now has an obligation, which means the lock becomes part of the type's public contract and cannot be an implementation detail.
- • The alternative — accepting approximation — has its own complexity: someone must own the tolerance, document it, and defend it the next time a number looks wrong.
- • One mutex over the whole transition. The default answer, and correct far more often than lock-free reasoning is. See Mutexes: What They Protect and What They Do Not.
- • A single owning thread reached through a queue: the invariant is enforced by there being exactly one writer. See The Actor Model and Message Passing.
- • An immutable snapshot swapped in with one atomic pointer store — the multi-field state becomes one location by construction. See Immutability as a Concurrency Strategy and Copy-on-Write as a Concurrency Strategy.
- • Move the invariant to a layer that already has transactions: a database row with a CHECK constraint, or a conditional write. See Transactions and ACID.
- • Redefine the requirement. "Approximately balanced, reconciled hourly" is a legitimate specification and is much cheaper than the alternatives — but it must be a decision, not an accident.
Two increments, twenty schedules: find the one that loses an update
| # | Task A — counter++ | Task B — counter++ | State |
|---|---|---|---|
| 1 | rA ← counter | · | counter=0 rA=0 rB=0 |
| 2 | rA ← rA + 1 | · | counter=0 rA=1 rB=0 |
| 3 | counter ← rA | · | counter=1 rA=1 rB=0 |
| 4 | · | rB ← counter | counter=1 rA=1 rB=1 |
| 5 | · | rB ← rB + 1 | counter=1 rA=1 rB=2 |
| 6 | · | counter ← rB | counter=2 rA=1 rB=2 |
if (balance >= 100) withdraw(100) — drive it until it overdraws
balance = 100
withdraw(amount): # both tasks run this concurrently
b = read(balance) # 1
if b >= amount: # 2 <- decided on a value that may already be stale
debit(amount) # 3| # | Withdrawal A (100) | Withdrawal B (100) | State |
|---|---|---|---|
| 1 | rA ← read balance | · | balance=100 paidOut=0 |
| 2 | if rA >= 100 | · | balance=100 paidOut=0 |
| 3 | debit 100 | · | balance=0 paidOut=100 |
The lost update, step by step
| # | Task A — counter++ | Task B — counter++ | State |
|---|---|---|---|
| 1 | rA ← counter | · | counter=0 rA=0 rB=— |
| 2 | · | rB ← counter | counter=0 rA=0 rB=0 |
| 3 | rA ← rA + 1 | · | counter=0 rA=1 rB=0 |
| 4 | counter ← rA | · | counter=1 rA=1 rB=0 |
| 5 | · | rB ← rB + 1 | counter=1 rA=1 rB=1 |
| 6 | · | counter ← rB | counter=1 rA=1 rB=1 ✕ 2 increments completed, counter = 1 |
What people believe, and what is true
I made every shared variable atomic, so the code is thread-safe.
Thread-safety is a property of operations against an invariant, not of variables. Every variable atomic and every invariant broken is a completely ordinary state for a program to be in.
The thread sanitizer found no races, so there is no race.
It found no *data* races. A race condition over correctly synchronized accesses is invisible to it, because it does not know what must stay true.
The window is only a few nanoseconds, so it will not happen.
At ten thousand transfers a second, a few nanoseconds is hit thousands of times a day. Narrow windows produce rare bugs, and rare bugs at scale are daily bugs.
Go deeper
Overview
An atomic makes one variable safe. If what must stay true involves two variables, you need something bigger — usually a lock.
Practical
Say the invariant out loud before choosing a primitive. Count the locations in the sentence. One location means an atomic may be enough; more than one means it is not, and no combination of atomics will change that.
Advanced
The only escape is to make the multi-field state a single location: pack it into one CAS-able word, or make it an immutable object published by one atomic pointer store. Both convert a composition problem into a publication problem, which is Safe Publication: Handing Over a Finished Object.
Internals
There is no hardware primitive that atomically updates two arbitrary addresses. Double-width CAS updates two *adjacent* words, which is why version-tagged pointers work and why "atomically update these two unrelated counters" does not. Hardware transactional memory attempted the general case and is not something to build on today.