Flaky Tests
A test that passes and fails on identical input destroys the verdict for every other test in the run, because it teaches people to re-run until green.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why is an intermittent test failure more dangerous than a consistent one?
CI's only product is a trustworthy verdict. A test that is right most of the time converts that verdict into a probability, and people respond to probabilities by resampling.
That test is flaky, everyone knows. Just re-run the job. We will fix it when we get a chance, or we will add an automatic retry so it stops bothering people.
Automatic retry does not remove the defect, it removes the evidence. The intermittent behaviour continues and nothing reports it.
- Automatic retry does not remove the defect, it removes the evidence. The intermittent behaviour continues and nothing reports it.
- The trained response generalises. Once re-running is the reflex, a genuine intermittent regression gets re-run too, and it merges.
- Flakes are contagious to attention: a run with one flaky test is red, so the other forty checks in that run also go unread.
- Many flakes are real defects. A test that fails when two things happen close together is describing a race that exists in the code, and production has far more concurrency than CI (Data Race Is Not Race Condition).
- Cost compounds quietly — re-runs consume runner minutes and wall-clock time on every affected change, and nobody attributes the spend to the flake.
- And a suite past a certain flake rate cannot be green at all: with enough independently flaky tests, the probability that a run passes falls below the point where anyone expects it to.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A test asserts a property of the code. A flaky test asserts a property of the code *and* of the environment it happened to run in, and the second half is not held constant.
- The sources are a short list, and they are all forms of undeclared input: wall-clock time, ordering, shared state, real concurrency, external services, resource limits, and unseeded randomness.
- Flakiness has a rate, and the rate is what makes it tractable. A test that fails one run in fifty is measurable, and measurement lets you rank flakes rather than argue about them.
- Detection is mechanical: the same commit producing different verdicts is definitionally a flake, and CI already has that data if anyone records it.
- Quarantine is what keeps trust intact while the fix is pending. Moving a test out of the blocking set and into a tracked list means its signal stops corrupting everyone else's verdict — provided quarantine has an owner and an expiry, or it becomes deletion with extra steps.
- The mitigation has its own failure mode. A blanket retry policy is quarantine applied invisibly to everything, forever, with no list and no owner (The Automation Trap).
What a flake costs that a failure does not
A deterministic failure and an intermittent one look similar in the run log and are opposite in effect. One adds information; the other subtracts it from every other check in the run.
The last row is the one that matters. A team that resamples verdicts has replaced a decision procedure with a slot machine, and the slot machine is also how genuine regressions get merged.
| Deterministic failure | Flaky failure | |
|---|---|---|
| What it tells the author | This change broke this behaviour | Something is wrong somewhere, possibly not here |
| Correct response | Fix the code or the assertion | Classify, quarantine, investigate the race |
| Actual response | Fix it | Re-run |
| Effect on other checks in the run | None; they still read | They go unread — the run is red for a reason nobody trusts |
| Cost visibility | Visible: someone spent an hour | Invisible: five minutes each, spread across everyone |
| What it teaches | Read the failure | Resample until green |
| Relationship to production | Caught a bug | Often *is* a bug, deferred to production to find (Data Race Is Not Race Condition) |
Where flakes come from
Every row is an undeclared input: something the test depends on that is not held constant. Naming the category is most of the fix, because each category has a standard remedy.
The response column is deliberately specific. "Make the test more robust" is not a remedy; injecting the clock is.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Test runs near midnight, or in another timezone | Fails at specific times of day, passes otherwise | Reads the wall clock, or assumes a timezone or a date boundary | Inject the clock as a dependency and set it explicitly (Production Time Is UTC, Timezone and DST Failures) |
| Test order changed by a shard or a runner update | Passes alone, fails in the suite, or vice versa | Order dependence — relies on state another test left behind | Randomise order deliberately, print the seed, isolate setup per test |
| Two tests run concurrently | Unexpected rows, missing records, wrong counts | Shared database, shared temp directory, shared global | One database or namespace per worker; no shared mutable fixtures (Parallelising CI) |
| Runner under load | Timeouts, and only sometimes | A fixed sleep or a timeout tuned to a fast machine, or a genuine race in the code | Wait on a condition, not a duration. If the code races, fix the code (Reasoning About Races: A Method, Not an Instinct) |
| External service called for real | Fails during someone else's outage or rate limit | The test depends on a system you do not control | Stub it; keep a small separate suite for real-dependency checks, off the blocking path |
| Unseeded randomness or generated data | Fails on rare inputs, unreproducibly | Random input with no recorded seed | Seed deterministically, print the seed on failure, keep failing seeds as fixtures (Determinism: Same Input, Same Output?) |
| Cache warm on one run, cold on another | Different behaviour with no code change | The test observes cache state it did not set up | Clear or control the cache in setup; treat cache state as an input (Caching in CI) |
A policy that actually removes them
The reason flake policies fail is almost always the missing deadline. Detection and quarantine are easy to build and easy to leave running forever, at which point the quarantine list is just a slower way of deleting tests.
Every step below has an owner and an exit condition. The step people skip is the last one.
- 1Detect
Record every same-commit verdict disagreement and compute a per-test rate.
fails by Relying on people to report flakes; they re-run instead, and the event is never recorded.
evidence A ranked list of tests with rates, generated from run history rather than from memory.
- 2Quarantine
Automatically remove a test above the threshold from the blocking set; keep running it and reporting it.
fails by Quarantine that also stops running the test, which loses the data needed to confirm a fix.
evidence The blocking suite's pass rate rises immediately, and re-run rate falls.
- 3Assign
Give it an owner — usually the owning team of the code under test, not the person who last touched the test.
fails by Assigning to a CI or platform team, who cannot fix a race in someone else's service (Shared Ownership).
evidence Every quarantine entry names a team that currently exists.
- 4Diagnose
Identify which undeclared input varies, using the table above.
fails by Adding a sleep or a retry, which changes the rate and not the cause.
evidence The test fails reliably when the identified input is forced, and passes when it is held.
- 5Exit
Fixed and returned to the blocking set, or deleted with the coverage gap recorded.
fails by Neither. The entry ages, the list grows, and quarantine becomes permanent.
evidence Entries leaving the list. If nothing has left in a month, the policy is not running.
A test deleted for being flaky is a coverage decision and should be recorded as one — otherwise the gap is invisible and nobody knows the behaviour is unasserted.
How to do it properly
Most important first.
- Measure first: record every case where the same commit produced different verdicts, and compute a per-test flake rate.
- Quarantine on a threshold, automatically. The test leaves the blocking set, keeps running, and appears on a list with an owner and a deadline.
- Give quarantine an expiry. A test still quarantined after the deadline is deleted or fixed — leaving it in limbo indefinitely is the same as deleting it, but without the honesty.
- Fix the cause, not the symptom: seed randomness, inject the clock, isolate state per test, wait on conditions rather than durations, stub external services (Determinism: Same Input, Same Output?).
- Randomise test order deliberately in CI. Order dependence found by design is cheap; found by a shard reassignment during an incident it is not.
- Treat concurrency flakes as production defects until proven otherwise. The test found a race; production will find it too (Reasoning About Races: A Method, Not an Instinct).
- If retries are used at all, scope them to the specific quarantined test, log every retry, and alert on the rate — never a suite-wide default.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
One flaky test is contained. A team culture of re-running until green is not: it removes the containment that CI was supposed to provide for every change.
What can go wrong
- A permanent global retry setting, which is the most common "fix" and the one that permanently removes the signal.
- Quarantine with no expiry, growing into a large set of tests that run, fail and are ignored — pure runner cost for no information.
- Deleting the test instead of fixing it, removing coverage of a real behaviour because the test was inconvenient.
- Adding a fixed sleep to make a timing flake pass, which slows every run and fails again on a slower runner.
- Blaming the CI system for flakes that are genuine races, and "fixing" them by reducing parallelism (Parallelising CI).
- A flake rate measured but never acted on, so the dashboard becomes a record of a decision nobody is making.
- "Flaky tests are a test problem." Many are code problems. A test that fails under concurrency has found a concurrency defect, and deleting the test does not delete the defect (Heisenbugs: The Bug That Leaves When You Look at It).
- "Retries make CI more reliable." They make CI *appear* more reliable while removing the measurement of how reliable it is.
- "We will fix them when we have time." Flake rate compounds: as the suite grows the probability of a fully green run falls, and past a threshold the pipeline stops being usable at all.
- "It only fails in CI, so it is a CI problem." CI is more concurrent, more resource-constrained and less warm than a laptop. Production is more so than CI (Why Local Success Predicts So Little).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Per-test flake rate exists as a number, computed from same-commit disagreements.
- The quarantine list is short, every entry has an owner and a date, and entries leave it.
- Re-run rate is trending down, and you can point at the tests whose fixes caused the drop.
- A randomly selected recent green run would still be green if re-run — spot-check this rather than assume it.
- Retries, where they exist, are logged and counted; a rising retry count triggers investigation rather than silence.
- Quarantine is itself the rollback: it is reversible in one step and restores trust in the rest of the suite immediately.
- If a de-flaking fix does not hold, put the test back in quarantine rather than adding a retry around it — retries hide whether the fix worked.
- Reverting a global retry policy will make the pipeline visibly redder. That is the policy working: the redness was always there, it was being absorbed.
- Automate detection and quarantine on threshold — humans are poor at noticing an intermittent pattern spread across weeks and people.
- Automate the report: which tests, what rate, whose, how long quarantined.
- Do not automate the fix, and do not automate a global retry. The first is impossible and the second is the failure mode (The Automation Trap).
- Quarantine removes a blocking check, so a real regression in that area can now merge. That is a genuine coverage gap accepted deliberately in exchange for a readable verdict.
- Making tests deterministic — injected clocks, seeded randomness, isolated state — costs design effort and makes some tests less faithful to how the system actually runs.
- Measuring flake rates needs run history storage and some analysis, which is infrastructure nobody asked for until they need it.
- Randomised order finds order dependence and makes individual failures harder to reproduce unless the seed is printed — so print the seed (Triaging a CI Failure).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALFlakiness sources are properties of tests and of the code under test, so they appear in every language and on every CI system. What differs is detection support: some runners ship flake detection and quarantine, others leave you to build it from run history.
- TOOL-SPECIFICTest-level retry is a runner feature — Jest and Vitest have retry options, JUnit needs an extension, pytest needs a plugin. Whether a retry is visible in the report differs too, and an invisible retry is the dangerous configuration regardless of which tool provides it.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — test isolation and determinism as design properties, and what a quarantined test costs in coverage terms.