The question this answers
A detector reports my program is race-free. What class of bug does that actually rule out?
A bank transfer implemented as two separately-locked account updates, plus a shared statistics counter written by every request thread.
The counter (unsynchronized, written by many threads) and the two account balances (each guarded by its own lock, but with no lock spanning the pair).
The counter equals the number of completed transfers, and the sum of all account balances is unchanged by any transfer.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Two conflicting accesses with no ordering between them
The definition a detector implements is narrow and exact. Two memory accesses race if they touch the same location, at least one is a write, they are performed by different threads, and there is no happens-before relationship ordering them. That last clause is the entire mechanism — the detector is not looking for "simultaneous", it is looking for *unordered*, which is a much more useful notion because two accesses a full second apart still race if nothing synchronizes them. Happens-Before: The Edge That Makes a Write Visible is the underlying relation, and Data Race Is Not Race Condition the definition.
Under most language memory models a data race is undefined behaviour or, at minimum, gives no guarantee about what values are observed. That is why detectors matter: a data race is not "probably fine most of the time", it is a construct the language declines to define — see What a Memory Model Defines. The compiler is entitled to assume you have none, and optimizes accordingly.
The narrowness cuts the other way too. counter++ from two threads with no synchronization is a data race and will be reported. Two threads each correctly locking their own account and transferring money in an order that leaves the *pair* inconsistent is not a data race at all — every access was properly synchronized. The detector will be silent, and the money will still be wrong.
1function transfer(from, to, amount):2 # BUG 1 — a DATA RACE. A detector reports this one.3 # two threads write stats.transfers with no synchronization at all.4 stats.transfers = stats.transfers + 15 6 # BUG 2 — a RACE CONDITION, not a data race.7 # every access below is correctly locked. A detector sees8 # perfectly ordered accesses and reports nothing.9 lock(from.mutex)10 if from.balance < amount: unlock(from.mutex); return INSUFFICIENT11 from.balance -= amount12 unlock(from.mutex)13 14 # <-- here the money exists nowhere. Another thread reading both15 # accounts sees a total that is short by 'amount'.16 17 lock(to.mutex)18 to.balance += amount19 unlock(to.mutex)20 21 return OK22 23# Detector verdict: 1 data race found (stats.transfers)24# Reality: 2 bugs. The one it missed is the one that loses money.How a detector observes, and what that costs
There are two broad categories, and knowing which you are using changes what a clean run means. Dynamic detectors — the sanitizer family, and equivalents built into some runtimes — instrument every memory access at compile time or run time, maintain a happens-before structure (commonly vector clocks) plus shadow metadata per memory word, and report a race the moment two unordered conflicting accesses are observed. They are precise about what they see and see only what executes. Static analysers reason about code without running it, catch some races on paths that never execute in a test, and produce false positives that dynamic detectors largely do not.
The cost of dynamic detection is substantial: it is normal for instrumented builds to run several times slower and to use several times the memory, because every access is intercepted and every word carries shadow state. That cost is why detectors run in CI and in dedicated soak environments rather than in production, and it is also why a detector build changes timing enough to alter which schedules occur — the Heisenbugs: The Bug That Leaves When You Look at It problem, applied to the tool itself.
A crucial and underappreciated property: dynamic detectors do not need the race to actually *manifest*. If two unordered conflicting accesses occur in an execution, the race is reported even if the resulting values happened to be correct that time. That is what makes them far stronger than "run it a lot and see if it breaks" — but it also means they only report races on the paths your test actually drove.
WARNING: data race
Write of size 8 at 0x55f0a2c41008 by thread T7:
#0 recordTransfer() stats.c:41 stats.transfers = stats.transfers + 1
#1 transfer() bank.c:118
#2 handleRequest() server.c:206
Previous write of size 8 at 0x55f0a2c41008 by thread T3:
#0 recordTransfer() stats.c:41 stats.transfers = stats.transfers + 1
#1 transfer() bank.c:118
#2 handleRequest() server.c:206
Location: global 'stats' of size 32 at 0x55f0a2c41000
No happens-before edge between T3's write and T7's write.
(T3 released no lock, sent no signal, and joined no thread that T7 observed.)
SUMMARY: 1 data race in recordTransfer
--- what this report does NOT say -----------------------------------
* nothing about bank.c:124-131, where 'from' and 'to' are locked
separately. Those accesses are ordered. The detector is satisfied.
* nothing about paths this run did not execute.
* nothing about whether any wrong value was ever actually observed.What a clean run proves, stated precisely
A clean dynamic-detector run proves: on the executions that were performed, no two conflicting unordered accesses occurred. That is a real and valuable guarantee — it eliminates an entire class of undefined behaviour on the exercised paths. It is emphatically not a proof of correctness, and the gap has four named parts, laid out in the matrix below.
The practical consequence is that detectors belong in a *pair* with something that explores schedules. A detector answers "is any access unsynchronized?"; stress testing with randomized scheduling answers "does any schedule break the invariant?" — see Stress Testing: A Test That Passed Once Proves Nothing. Neither subsumes the other, and shipping with only one is the common mistake. Deterministic replay is the third leg, because it makes whatever either tool finds reproducible — Deterministic Replay: Making the Schedule Reproducible.
One further honest note: the absence of data races is precisely what makes a race condition *hard*. If your program is data-race-free, the remaining timing bugs are all at the level of multi-step invariants, which no memory-access-level tool can see. Fixing the data races that a detector finds moves your bugs up a level; it does not remove them.
| Gap | Why the detector misses it | What closes it |
|---|---|---|
| Race conditions with correct locking | Every access was ordered. There is no data race to find — the bug is in the size of the critical section, not in synchronization. | Invariant-based reasoning and schedule exploration — Reasoning About Races: A Method, Not an Instinct, Finding the Critical Section |
| Unexecuted code paths | Dynamic detection only observes what ran. An error branch never taken has never been checked. | Coverage-directed tests plus static analysis |
| Timing-dependent liveness bugs | Deadlock, livelock and starvation involve no conflicting memory access at all. | Deadlock detection, wait-graph analysis, timeouts — Deadlock, Livelock |
| Logic that is wrong even single-threaded | A detector says nothing about whether the computation is right. | Ordinary tests |
Key points
- A detector reports two conflicting accesses to the same location by different threads with no happens-before edge between them. That is the whole definition.
- Data races and race conditions are different: the first is about unsynchronized memory access, the second about logic depending on timing. Detectors find only the first.
- The correctly-locked-but-still-wrong transfer is the canonical case: every access ordered, detector silent, money missing.
- Dynamic detectors report a race even when the run produced correct values — the race is the unordered pair, not the wrong answer.
- They observe only executed paths, and cost several times the runtime and memory, which is why they live in CI rather than production.
- A clean detector run moves your bugs up a level to multi-step invariant violations; it does not remove them.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Instrument every load and store, at compile time or via binary rewriting, so the tool observes each memory access with its address, size and thread.
- • Maintain a vector clock per thread and shadow metadata per memory word recording which threads last read and wrote it, at which clock.
- • Advance clocks on synchronization events — lock acquire and release, thread start and join, atomic operations — because those are what create happens-before edges.
- • On each access, compare the accessing thread's clock against the shadow metadata; if the previous conflicting access is not ordered before this one, report a race with both stacks.
- • Static analysers instead reason over the control-flow graph and a lockset model, catching unexecuted paths at the price of false positives.
- • T3 executes
stats.transfers = stats.transfers + 1and T7 executes the same line with no lock between them. Even when the result happens to be correct, the shadow metadata shows T3's write unordered with T7's, and the race is reported. - • T1 locks
from, debits 100, unlocks. Before T1 locksto, T2 reads both balances and computes a total 100 short. Every access was ordered by a lock; the detector reports nothing; the audit report is wrong. - • A race exists on an error-handling path. The test suite never triggers the error, the path never executes, the shadow metadata is never consulted, and CI is green for two years.
- • The instrumented build runs 8x slower, which widens every window and causes a *different* interleaving to be explored than the one production hits — the tool changed the schedule it was measuring.
- • Dynamic detection guarantees soundness for what it observed: a reported race is (barring rare tool bugs) a real unordered conflicting pair, not a heuristic guess.
- • It guarantees nothing about paths not executed, schedules not explored, or invariants above the level of a single memory location.
- • It does not require the bug to manifest — an unordered pair that produced the right answer is still reported. This is its main advantage over stress testing alone.
- • It says nothing about deadlock, livelock, starvation or lost updates through properly-locked operations, because none of those involve an unordered access.
- • A clean run is a statement about an execution, never about a program.
- • Shadow metadata roughly multiplies the memory footprint, which itself changes cache behaviour and therefore timing.
- • Every synchronization event must update the tool's clock structures, adding work to exactly the acquire and release paths that are already contended.
- • The slowdown is uneven — heavily-shared data pays most — so an instrumented run can shift where the contention appears to be.
- • False confidence: a green detector run treated as proof of thread safety, shipping a lost-update bug that was never in scope for the tool.
- • Terminology collapse: the report says "data race", the team says "race condition", and the fix targets the wrong level.
- • Suppression rot: a suppression file added for a noisy third-party library grows until it hides real findings.
- • Coverage illusion: the detector runs only under unit tests that use a single thread, so it observes nothing and reports nothing.
- • Timing displacement: the instrumented build never reproduces the production schedule, so the specific bug being hunted never occurs under the tool.
- • Languages where a data race is undefined behaviour, because there the class of bug is not merely wrong but unbounded — see C++: Threads, Atomics and a Memory Model With Teeth.
- • CI on any codebase with shared mutable state, where the cost is paid by machines and the finding is precise and actionable.
- • Reviewing new concurrent code, where an unsynchronized field is easy to write and nearly invisible in review.
- • Confirming that a lock-free construction actually established the ordering it claims — detectors understand atomics and will report missing edges.
- • As the only concurrency test. A codebase with correct locking everywhere and a lost-update bug gets a clean bill of health.
- • On latency-sensitive integration tests, where the slowdown causes unrelated timeouts and the run becomes uninterpretable.
- • In languages and runtimes where the memory model already precludes the class being detected, making the cost pure overhead.
- • Number of distinct reported races, tracked over time — the useful metric is new races per week, not the absolute count.
- • Fraction of the test suite executed under the detector, since unexecuted paths are unchecked paths.
- • Suppression-file size and age, as a proxy for how much of the tool has been switched off.
- • Whether known-race regression tests still trip the detector, which is how you prove the tool is actually running.
- • A second build configuration to maintain, with its own dependencies, flags and CI lane.
- • Test suites need genuine multi-threaded exercise or the detector observes nothing — writing those tests is itself work.
- • Report triage requires reading two stacks and reasoning about the missing happens-before edge, which is a real skill.
- • Suppressions become a governance problem: every entry is a permanent blind spot that needs an owner.
- • Eliminate shared mutable state so the question cannot arise — message passing or immutable snapshots, see Message Passing and Immutability as a Concurrency Strategy.
- • Make the shared field atomic, which both fixes the race and removes it from the detector's scope — Atomics: What Is Actually Indivisible.
- • Stress testing with randomized scheduling, which finds invariant violations that no memory-access tool can see — Stress Testing: A Test That Passed Once Proves Nothing.
- • Type-system or ownership enforcement, where the language prevents unsynchronized sharing at compile time and the detector becomes unnecessary for that class.
- • Code review against a written invariant, which catches the correctly-locked-but-wrong-scope bug that tooling misses entirely.
Race detector lab
| task | access | location | holding |
|---|---|---|---|
| A | read | count | — |
| A | write | count | — |
| B | read | count | — |
| B | write | count | — |
| # | Task A | Task B | State |
|---|---|---|---|
| 1 | r1 = count | · | count=0 done=0 |
| 2 | · | r2 = count | count=0 done=0 |
| 3 | count = r1 + 1 | · | count=1 done=1 |
| 4 | · | count = r2 + 1 | count=1 done=2 ✕ count equals the number of increments that have completed — broken here |
What people believe, and what is true
The race detector is clean, so the code is thread-safe.
It is free of unordered conflicting accesses on the paths that ran. A perfectly locked transfer that exposes an inconsistent intermediate state is thread-unsafe and completely invisible to it.
A data race just means you might read a stale value.
Under memory models that call it undefined behaviour, the compiler may assume it cannot happen and transform the code accordingly. The observable result is not bounded to "stale value".
If the output was right, there was no race.
A dynamic detector reports the unordered access pair regardless of the values produced. That is exactly why it beats running the test a thousand times.