The question this answers
What is chaos engineering actually for, and what separates it from causing an outage on purpose?
The practice guarantees nothing about the system. It guarantees something about your knowledge: after a completed experiment you know whether one specific assumption held, at one specific blast radius, on one specific day. That is a much smaller claim than "the system is resilient", and it is the only one the method supports.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
Nothing about this practice is visible to a node — an injected fault is indistinguishable from a real one, which is the entire point. The experimenter knows what was injected and when; the system does not, and must not, because a system that behaves differently when it knows it is being tested is not being tested. This is also why "the fault was injected" must be evidenced from the target’s own signals rather than from the runner’s intent.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Four words, each of which is load-bearing
Deliberate: you chose this fault, at this time, because you had a specific question. Randomly killing things is not chaos engineering; it is a fuzzer with no oracle, and when something breaks you learn "something broke" rather than "our assumption about replica promotion is wrong".
Controlled: bounded blast radius, bounded duration, an automatic abort condition, an announced window, a tested revert. Control is what makes the risk priceable, and pricing the risk is what makes it permissible to do this where real users are.
To test: there is a hypothesis, stated before the injection, that a single run can refute. Without it the exercise cannot fail, because any outcome will be rationalised afterwards.
An assumption you already hold: the target is a belief the team acts on. "Losing a zone is transparent to users." "The breaker opens before the pool saturates." "The replica promotes within thirty seconds." "The retry budget caps amplification at 1.2x." Every one of those is written in a design document somewhere, believed by everyone, and unverified.
| Chaos engineering | Breaking production | |
|---|---|---|
| Target chosen becausetypical | A specific stated assumption | It seemed interesting or it was easy |
| Baselinetypical | Measured and recorded beforehand | Judged from memory afterwards |
| Blast radiustypical | Declared, minimal, ratcheted up over runs | Whatever the fault happens to reach |
| Stop conditiontypical | Automatic threshold plus a maximum duration | Somebody notices |
| A bad outcome istypical | A result — the finding you were looking for | An incident, and a reason the practice gets banned |
| Outputtypical | Hypothesis confirmed or refuted, with evidence | A war story |
Why test assumptions rather than look for bugs
The practice is not a bug hunt. It is aimed at a specific epistemic problem: distributed systems accumulate beliefs that were true when written and are no longer checked. The failover was tested at launch, three years and two hundred deploys ago. The runbook describes a topology that changed. The circuit breaker’s threshold was set for a dependency that has since become four times faster. Nothing failed in the interim — that is exactly why nobody noticed.
Ordinary testing cannot reach these. Unit and integration tests check the code you wrote against the behaviour you specified; they cannot check whether your production topology still matches the assumption that made the specification correct. Load tests check capacity, not failure response. Only injecting the failure into the real system, with real traffic and real shared infrastructure, tests the assumption as it actually stands today.
Which is why the highest-value experiments are not exotic. They are the boring ones aimed at claims everybody is certain about — and the value is highest precisely where certainty is highest, because certainty is what stops anybody checking.
- "We can lose an availability zone." — tested at launch, never since.
- "The database fails over automatically in under a minute." — true of the version you ran two upgrades ago.
- "The circuit breaker protects us." — threshold set against a dependency whose latency profile has changed.
- "That dependency is optional." — became load-bearing when someone added a call in the fallback path.
- "The runbook covers this." — refers to a dashboard that was deleted.
The discipline, and why it is the whole practice
Chaos engineering has a reputation problem, and it is deserved: the practice is remembered for a tool that kills instances at random, and that memory has cost more programmes than any technical difficulty. When an engineer proposes it and a director hears "randomly break production during business hours", the answer is no, and the answer is correct given what was heard.
The way through is to lead with the constraints rather than the injection. Smallest radius that can still refute the hypothesis. Automatic abort. Announced window. Deploy freeze. Tested revert. Staging first where the hypothesis permits. Escalate the radius only after the hypothesis holds at the smaller one. Presented that way, the request is not "may we break production" — it is "may we spend a bounded, reversible, four-minute risk to find out whether a belief we are already betting the business on is true".
And there is an ordering rule that saves programmes: do not run experiments you are confident will fail. If you already believe the failover is broken, you do not need evidence, you need a fix. Experiments are for beliefs you hold and cannot verify — running one to prove a colleague wrong is politics with an outage attached.
exp-2026-041 "single zone loss is transparent to checkout"
radius 1 of 3 zones, 6 min, off-peak
abort checkout_success < 99.0 OR p99 > 1500ms (edge-sourced)
result REFUTED at t+94s — abort fired on p99
finding connection pools did not re-balance; 1/3 of connections
pointed at the drained zone for 4 more minutes
action DS-2291 (pool health-check interval), owner: platform
rerun after DS-2291 ships
exp-2026-042 "checkout survives loss of the fraud service"
status NOT RUN — team is confident it fails closed today.
reason An experiment we expect to fail is a fix request, not an experiment.
action DS-2304 (decide fail-open policy), owner: risk + paymentsWhere the value actually comes from
Teams expect the value to be in the refutations, and some of it is. But three other sources are consistently larger and rarely anticipated.
The preparation finds more than the run. Writing the hypothesis forces someone to state the assumption precisely, and the act of stating it precisely is frequently enough to reveal that it is false. Measuring the steady state finds that the metric is an average, or mixes two products, or is scraped too coarsely to see the effect. Neither of those findings requires injecting anything.
Confirmations are results. A hypothesis that holds converts a belief into evidence with a date on it. That is what lets you say "we can lose a zone" to an auditor, a customer or a new team member and mean something by it. It also creates a regression baseline: when the same experiment fails in six months, you know the change is recent and roughly where to look.
The human path gets tested too. Did the right page fire? Did the runbook match reality? Could on-call find the dashboard? Did anyone know who owned the component? A game day with no injection at all tests all of this, which is why it is the correct first step for most teams — and why "we are not ready for chaos engineering" usually means "we should start with the parts that carry no risk".
Key points
- Deliberately introduce controlled failures to test assumptions — not randomly break production.
- The target is a belief the team already acts on but has not verified.
- Ordinary tests cannot reach these assumptions: they check code against specification, not topology against belief.
- Lead with the constraints — radius, abort, window, freeze, tested revert — because the reputation problem is what kills programmes.
- Never run an experiment you are confident will fail; that is a fix request wearing an experiment’s clothes.
- Most of the value is in the preparation and in the confirmations, not in the dramatic refutations.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Identify an assumption the team acts on and cannot currently evidence.
- • State it as a hypothesis a single run can refute.
- • Measure and record steady state; if you cannot, stop and report that.
- • Declare radius, duration, abort thresholds and revert, and dry-run the revert.
- • Announce the window and freeze deploys for it.
- • Inject, observe against the baseline, and let the runner stop the experiment.
- • Record the conclusion, the action item and the owner; schedule the re-run.
- • The injection does not reach the target and a false confirmation is recorded.
- • The blast radius exceeds the declaration through a coupling nobody drew.
- • The abort path depends on the injected component and cannot fire.
- • An unrelated incident coincides with the window and the result is unattributable.
- • The programme produces refutations that are never fixed, so the same experiment fails repeatedly and the practice loses credibility.
- • Injected fault that never landed: the operator sees the experiment complete with no deviation whatsoever, and later finds the target instance had been out of rotation the whole time — the "hypothesis held" record is worthless.
- • Radius overrun: the operator sees a second team paged for a service that was not in the declared scope, because the target shared a node pool with it.
- • Abort that could not fire: the operator sees the injection run for its full duration through a severe deviation, because the abort condition read a metric served by the broken component.
- • Unattributable window: the operator sees a large deviation during the experiment and a deploy that landed ninety seconds earlier, and cannot separate them because there was no deploy freeze.
- • Findings backlog: the operator sees the same experiment refuted for the fourth quarter in a row, with the fix still unassigned — at which point the programme is generating incidents and no improvement.
- • Announcing the window and freezing deploys are coordination with other humans, and they are the cheapest risk reduction available.
- • Authority to accept the risk must be explicit and held by someone who owns the consequence; without it, the first surprise ends the programme.
- • The experiment itself needs no coordination with the target system, and must not have any — a fault the target cooperates with does not model the target being gone.
- • Findings need an owner at declaration time, not at review time, or the backlog accumulates and the practice becomes pure cost.
- • During the injection the system provides whatever guarantee it genuinely has, which may be weaker than the documented one — measuring that gap is the purpose.
- • The abort condition bounds the duration of any deviation, not its depth within the threshold.
- • State that drifted during the window is real drift and needs the same reconcile step as any incident.
- • If the experiment escapes its declared radius, it is an incident and should be handled as one, including a postmortem — the fact that you started it does not change its nature.
- • Detect: the runner watches the declared metrics against the recorded baseline continuously.
- • Contain: the abort fires and reverts the injection; the maximum duration is the second net for when abort logic fails.
- • Recover: confirm the target is back in service and any backlog has drained before ending the window.
- • Reconcile: repair derived state that drifted, and re-run any skipped checks, exactly as after a real incident.
- • Verify: re-measure steady state against the pre-experiment baseline, and record the result with its evidence.
- • Evidence that the fault actually applied, sourced from the target rather than from the runner.
- • Deviation from the recorded baseline in the hypothesis metric, sampled finer than the expected effect.
- • Whether the blast radius stayed inside the declaration, measured by which services deviated, not by which were targeted.
- • Time from abort threshold crossing to injection reverted — the number that tells you whether the bound is real.
- • Programme-level: ratio of findings closed to findings raised. A falling ratio means the practice is generating risk without buying safety.
- • Systems with resilience mechanisms that are rarely exercised — failover, replica promotion, breakers, multi-zone routing — where the only proof they work is that they have not been needed.
- • After an incident, to verify that the fix actually changes the behaviour that caused it.
- • Organisations where operational knowledge is concentrated in a few people: a game day distributes it faster than documentation does.
- • Systems with a known backlog of reliability defects: you already know what will break, and the experiment adds risk without adding information.
- • Teams that cannot yet measure steady state — start there, since that finding is free and the experiment would be uninterpretable anyway.
- • Where there is no authority to accept the risk, or no capacity to fix what is found; both turn the programme into a source of incidents.
- • Game days: walk through the scenario with humans and runbooks and no injection. Zero risk, and it finds missing runbooks, unclear ownership and stale dashboards — the majority of what first experiments find anyway.
- • Failure-injection tests in CI against a subset of the system: repeatable, safe, and they catch regressions in code-level failure handling, though not in production topology.
- • Mining incident history: every past incident is an experiment that already ran, at full blast radius, for free. Most teams have not extracted the assumptions those incidents refuted.
- • Dependency and fault-domain review on paper, which is cheaper than any experiment and tells you which experiment is worth the risk.
Experiment, or outage with a fashionable name?
What people believe, and what is true
Chaos engineering means randomly killing things in production.
Random killing with no hypothesis produces "something broke" rather than an answer. The practice is deliberate injection of a chosen fault to test a stated assumption, under bounds.
You need a chaos tool to start.
The first valuable exercises need no injection at all: state an assumption precisely, try to measure the steady state that would test it, and walk the scenario with a runbook. Most teams find their first three findings there.
An experiment where nothing broke was a waste of time.
It converted a belief into dated evidence and created a regression baseline. That is the majority of the practice’s long-run value.
We do chaos engineering — we have a tool that kills a pod every hour.
With no hypothesis and no baseline, that is a background source of noise. It may build resilience habits; it does not test any assumption, and it cannot tell you which of your beliefs are false.
Production experiments are reckless.
Unbounded ones are. A four-minute injection at 1% radius with an automatic abort carries a smaller expected cost than the unbounded outage that occurs when an unverified assumption fails on its own schedule.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Deliberately introduce controlled failures to test assumptions you already hold. Not randomly breaking production — every word of that definition is doing work.
Practical
Start with the assumptions your design documents state and nobody has checked. Write one as a hypothesis, try to measure its steady state, and expect to stop there the first time. Lead every proposal with radius, abort, window and revert. Never run an experiment you expect to fail. Give every finding an owner before the run.
Advanced
The practice manages a specific risk: belief decay. Resilience mechanisms are exercised rarely, so their correctness is unobserved between incidents, and confidence in them grows with time while evidence for them does not. Experiments are how you pay a small, scheduled, bounded cost to keep that confidence attached to evidence — which is why cadence matters more than sophistication, and why an unrepeated experiment is worth much less than a boring one that runs every month.
Apply it
- 🔧 Open a design document your team wrote and extract every resilience claim it makes. Mark each as evidenced, unevidenced, or known false.
- 🔧 Design a game day for your highest-impact dependency that injects nothing at all, and list what it would still find.
- ⚡ A quarterly experiment that passed for three quarters fails this quarter. Nothing about the experiment changed. What is the first thing this tells you, and where do you look?
- 💬 A director hears "chaos engineering" and says no. What do you say next?
- 💬 Name three assumptions your team acts on that have never been verified. How would you test the cheapest one?
- 💬 Your experiment ran and nothing deviated. What do you have to check before recording that the hypothesis held?
- 💬 Why is an experiment you are confident will fail a bad experiment?