The question this answers
My concurrency test passes. What have I actually learned?
A test that spawns two threads, each incrementing a shared counter a thousand times, and asserts the total is two thousand — plus the same scenario driven under randomized scheduling and injected delays.
The counter under test, and — in the harness — the scheduling decisions themselves, which the harness treats as a controllable input rather than an accident.
The counter equals the number of completed increments, under every schedule the harness is capable of producing.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The arithmetic of why one pass means nothing
Two threads each performing three indivisible operations can interleave in twenty ways. Two threads of ten operations: 184,756. Two threads of a thousand operations, which is a modest test: a number with six hundred digits. Your test executed exactly one of them. Passing tells you that one schedule was fine.
Worse, the schedule it executed was not sampled uniformly — it was whichever one the scheduler produces on an idle machine with warm caches and no contention, which is close to "each thread runs to completion in turn". That is systematically the *least* interesting region of the space. The schedules that break invariants cluster around preemptions inside a critical window, and an idle test machine almost never preempts there.
This is why concurrency tests are the classic false negative. Running the test a thousand times on the same machine mostly re-samples the same region. What changes the outcome is changing the *distribution*: force preemptions at random points, inject delays at suspension points, run under load so the scheduler is actually making choices, vary core counts and thread counts.
1# LEVEL 0 -- proves nothing2def test_counter():3 c = Counter()4 t1 = spawn(lambda: [c.inc() for _ in range(1000)])5 t2 = spawn(lambda: [c.inc() for _ in range(1000)])6 join(t1); join(t2)7 assert c.value == 2000 # passes ~always on an idle machine8 9# LEVEL 1 -- repetition. Better, still weak: same machine, same10# scheduler, same warm cache => largely the same region of schedules.11def test_counter_repeated():12 for _ in range(10_000):13 test_counter() # finds coarse races, misses narrow windows14 15# LEVEL 2 -- schedule perturbation. This is the one that finds bugs.16def test_counter_stressed(seed):17 rng = Random(seed)18 hooks.set_yield_policy(lambda: rng.random() < 0.05) # random preempt19 hooks.set_delay_policy(lambda: rng.choice([0, 0, 0, 1, 50, 500])) # us20 hooks.set_thread_count(rng.choice([2, 3, 8, 33])) # incl. > cores21 background_load(cpu_hogs=rng.choice([0, 4, 16])) # real contention22 23 c = Counter()24 ts = [spawn(lambda: [c.inc() for _ in range(1000)]) for _ in range(n)]25 for t in ts: join(t)26 assert c.value == 1000 * n, failure_report(seed) # SEED is the artifact27 28# The seed is the whole point: a failing seed is a reproducible bug report.29# Store every failing seed as a permanent regression case.The four knobs, and what each one surfaces
Randomized preemption inserts yields at random points, which explores interleavings the scheduler would not naturally choose. Delay injection sleeps at chosen points — the widened-window technique from Heisenbugs: The Bug That Leaves When You Look at It, used deliberately — and is the most targeted knob, because a delay placed exactly at a suspected window converts a one-in-forty-thousand bug into a one-in-two bug. Thread-count variation past the core count forces genuine preemption rather than parallel execution, which surfaces a different family of bugs — see Oversubscription. Background load makes the scheduler actually make decisions instead of running everything immediately.
They surface different things. Randomized preemption is good at check-then-act and lost updates. Delay injection is good at confirming a specific hypothesis. Oversubscription is good at lock convoys, priority inversion and lost wakeups. Load is good at pool exhaustion and queue behaviour. Running only one knob and calling it stress testing leaves whole families uncovered.
The critical output is not "passed" or "failed" — it is the *seed*. A harness that fails without recording the seed and the schedule has produced an unreproducible failure, which is worse than useless because it will be dismissed as flakiness. Every failing seed becomes a permanent regression test, and this is how a concurrency suite accumulates real coverage over time. Where seeds are insufficient, this is exactly the hand-off point to Deterministic Replay: Making the Schedule Reproducible.
| # | Thread A | Thread B | Stress harness | State |
|---|---|---|---|---|
| 1 | read counter (41) | · | · | counter=41 A local=41 |
| 2 | · | · | inject 5ms delay at suspension point in A | counter=41 |
| 3 | · | read counter (41) | · | counter=41 A local=41 B local=41 |
| 4 | · | compute 42, write counter | · | counter=42 A local=41 |
| 5 | delay expires; compute 42, write counter | · | · | counter=42 ✕ The counter equals the number of completed increments. Two increments completed; the counter reads 42, not 43. One increment was lost. |
| 6 | · | · | assertion fails; record seed=0x4c19, delay site, thread count | counter=42 seed=0x4c19 |
Reading the results honestly
The output of a stress campaign is a distribution, not a verdict. "Ten thousand runs, zero failures" is a bound on failure probability under the schedules that harness produced — not a proof of correctness, and specifically not a statement about schedules it cannot produce. If the harness never runs more threads than cores, it has said nothing about oversubscription. If it never injects a delay inside the critical section, it has said nothing about that window.
So report coverage of *knobs*, not just run counts, and treat every failure as a permanent artefact. The read-out below is what a useful campaign summary looks like: seeds, the knob settings that produced each failure, and the reproduction rate per seed. A reproduction rate of 1.00 means the seed is a reliable regression test; a rate of 0.02 means the seed captured something narrower and may need delay injection to become stable.
The last honest note concerns flaky tests. A concurrency test that fails one time in fifty in CI is not a flaky test — it is a passing test that occasionally tells the truth. Retrying it until green is a decision to ship the bug. The correct response is to take the failure, feed it to a stress harness with the same knobs, and either reproduce it deterministically or record the seed.
CAMPAIGN counter-invariant duration 42m runs 240,000 KNOB COVERAGE random preempt probability 0.00 .. 0.20 covered injected delay sites 14 of 14 covered injected delay magnitudes 0, 1us, 50us, 500us, 5ms covered thread count 2, 3, 8, 33 covered (cores = 8) background cpu load 0, 4, 16 hogs covered NUMA / core pinning not exercised GAP clock skew between threads not exercised GAP FAILURES (each seed is now a permanent regression case) seed knobs repro rate 0x4c19 delay 5ms @ site 7, threads 2 1.00 <- reliable 0x4c19 (same seed, delay removed) 0.00002 0x91af preempt p=0.20, threads 33, load 16 0.41 0xe007 preempt p=0.05, threads 3 0.02 <- narrow 0x1d3c delay 500us @ site 11, threads 8 0.96 CONCLUSION 4 distinct invariant violations, all lost updates on the counter. Zero failures under the default (level 0) test, across all 240,000 runs of the unperturbed configuration. This is what "the test passes" was worth.
Key points
- Two threads of a thousand operations have a schedule space with hundreds of digits. Your passing test executed one of them.
- The schedule a normal test executes is not a random sample — it is the least interesting one, because an idle machine rarely preempts inside a critical window.
- Four knobs matter: randomized preemption, injected delays, thread counts above core count, and real background load. Each surfaces a different family of bugs.
- A delay injected at a suspected window converts a one-in-forty-thousand bug into a one-in-two bug. It is the most targeted tool in the set.
- The deliverable of a failing run is the seed, not the failure. A failure without a reproducible seed will be dismissed as flakiness.
- A concurrency test that fails one time in fifty is not flaky. It is a passing test that occasionally tells the truth, and retrying it is a decision to ship the bug.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Parameterize the run by a seed, and derive every scheduling decision the harness controls from that seed so failures are reproducible.
- • Insert yield points at suspension sites and take them with a seeded probability, forcing interleavings the scheduler would not produce.
- • Inject delays of varied magnitude at enumerated sites, widening specific windows so that racing actors land inside them reliably.
- • Vary thread count above and below core count, and add genuine background CPU load, so the scheduler is actually making choices under contention.
- • Assert the invariant, not the output: "counter equals completed increments" catches schedules that a fixed expected value would miss.
- • On failure, persist the seed and knob settings as a regression case and measure its reproduction rate over repeated runs.
- • Unperturbed: A performs all 1,000 increments, then B performs all 1,000. Total 2,000, test passes, nothing learned.
- • With a 5ms delay at A's read: B reads the same value, both compute the same successor, and one increment is lost on nearly every run.
- • 33 threads on 8 cores: preemption happens constantly and the lost-update rate rises without any injection at all — oversubscription as a stress knob.
- • Preempt probability 0.20 with 33 threads under load reveals a *different* failure — a lost wakeup in the pool's condition predicate — that no delay-injection configuration produced, because it needs many waiters, not a wide window. See Lost Wakeups: The Notify That Arrived Before the Wait.
- • A campaign guarantees that the invariant held across the schedules the harness produced. It guarantees nothing about schedules outside the knobs it exercised.
- • A recorded failing seed guarantees reproduction only for the knob settings recorded with it; the seed alone is not enough.
- • Zero failures across 240,000 runs is a probability bound under that distribution, not a correctness proof — and the distribution is one you chose, not a uniform one.
- • It does not identify the cause. It produces a failing execution; understanding it still requires reasoning or a trace.
- • It cannot find bugs the assertion does not check. An invariant nobody wrote down is an invariant nobody is testing.
- • The harness itself must not synchronize with the code under test, or it introduces happens-before edges that suppress the very bugs it hunts.
- • Background load competes with the test for cores, which is the point, but it also makes wall-clock duration unpredictable and CI budgets hard to set.
- • Injected delays inside critical sections serialize everything behind them, so a campaign with aggressive injection runs far slower than the sum of its parts.
- • False confidence from repetition alone: ten thousand runs of the same unperturbed configuration re-sample one region and prove almost nothing.
- • Unreproducible failure: the harness reports an assertion failure without the seed and knob settings, so the finding is unactionable and gets closed as flaky.
- • Assertion too weak: checking only the final total misses intermediate invariant violations that a stricter check would catch.
- • Harness-induced synchronization: a shared log or counter inside the harness orders the threads and hides the race.
- • Retry-until-green in CI, which converts a real finding into a permanently invisible bug.
- • Knob monoculture: only delay injection is used, so lost wakeups and convoys — which need many waiters rather than wide windows — are never surfaced.
- • Any shared-state component with a stated invariant: counters, caches, pools, queues, state machines. The cost is machine time and the finding is concrete.
- • Confirming a hypothesized window — inject a delay exactly there, and a jump in failure rate is strong evidence you found the right one.
- • Building a regression suite over time, since every failing seed becomes a permanent test for a bug that would otherwise be unreproducible.
- • Before a concurrency refactor, to establish that the existing behaviour actually holds under perturbation rather than by luck.
- • On code with no shared mutable state, where the schedule space is irrelevant and the campaign burns CI budget to prove nothing.
- • As a substitute for reasoning: a harness can only test the invariant you wrote, and the hard part is usually knowing what the invariant is — Invariants: Name It Before You Lock It.
- • When failures are recorded without seeds, at which point the campaign generates noise that trains the team to ignore concurrency failures.
- • In tight CI loops, where a 42-minute campaign cannot run per-commit and belongs on a nightly or pre-release schedule instead.
- • Failure rate per seed and per knob configuration — the shape of that table is the real result.
- • Knob coverage, explicitly listing what was not exercised, since gaps are where the untested schedules live.
- • Reproduction rate of each stored failing seed; below about 0.1 the seed needs delay injection to become a usable regression test.
- • Whether an unperturbed configuration ever fails at all — usually it does not, which is the point being demonstrated.
- • New distinct invariant violations found per campaign hour, as the signal for whether to keep extending the harness.
- • Yield and delay hooks must be threaded through the code under test, or the harness can only perturb at coarse boundaries.
- • Seeds must fully determine harness behaviour, which requires discipline about every source of randomness inside the harness itself.
- • CI needs a place for long campaigns that is not the per-commit path, plus a policy for what a failure blocks.
- • A growing library of failing seeds is a maintenance surface: each one needs to keep reproducing as the code changes, or be retired deliberately.
- • A race detector, which reports unordered accesses without needing the failing schedule to occur at all — strictly better for data races, blind to invariant violations. See Race Detectors: What They Find, and What They Structurally Cannot.
- • Deterministic replay, once a failure has been captured, to make it reproducible without relying on a seed reproducing the same schedule — Deterministic Replay: Making the Schedule Reproducible.
- • Exhaustive schedule exploration or model checking on a small critical section, when the state space is genuinely small enough to enumerate.
- • Designing the shared state away — an atomic counter, an immutable snapshot, or a single owner — so the schedule space stops mattering. See Atomics: What Is Actually Indivisible and Immutability as a Concurrency Strategy.
- • Property-based testing over operation sequences, which explores logical orderings rather than physical schedules and is cheaper to run.
What people believe, and what is true
The concurrency test passes, so the code is thread-safe.
It passed one schedule out of a space with hundreds of digits, and specifically the least adversarial one. That is not evidence of thread safety.
Running the test 10,000 times is stress testing.
Repetition without perturbation re-samples the same region. What finds bugs is changing the distribution — preemption, delays, thread counts, load.
That test is flaky, we retry it.
A concurrency test that fails occasionally is reporting a real schedule in which the invariant broke. Retrying until green is a decision to ship it.