The Steady-State Hypothesis and the Abort Condition

An experiment that is written down has six parts, and skipping any of them turns it into something else. Hypothesis: a falsifiable statement about what the system will do. Steady state: the measurable normal behaviour, recorded *before* anything is touched. Injection: the specific fault, at a specific blast radius, for a specific duration. Observation: the metric that decides the hypothesis. Abort condition: the threshold at which the injection is automatically reversed. Conclusion: what was learned, including "the hypothesis held", which is a real result.

The hypothesis must be falsifiable and specific. "The system is resilient to a node failure" is not a hypothesis; it cannot be shown false by any measurement. "When one of the three API replicas is killed, checkout success rate stays above 99.5% and p99 stays under 800ms for the full five minutes" is a hypothesis, because a single run can refute it.

Note the direction of the hypothesis. You state what you *expect*, then try to break the expectation. An experiment run without a stated expectation cannot fail — whatever happens, someone will explain afterwards why that was the expected behaviour. This is the operational version of pre-registration, and it exists for the same reason.

The second step defeats more experiments than any other. To claim the system deviated, you need a number that describes normal, with its normal variance, before you touch anything. Not "it feels fine" — a measured distribution: checkout success rate at 99.7% ± 0.1 over the last fourteen days, p99 at 480ms ± 60, queue depth under 200.

Teams routinely discover at this step that no such number exists. The success-rate metric mixes two products; the latency metric is an average, so a doubled tail is invisible in it; the queue-depth gauge is scraped every 60 seconds and the interesting behaviour lasts 20. Stop there and report it. "We cannot measure our steady state" is a more valuable result than any injection would have produced, because it means that during a real incident nobody will be able to tell whether the system is recovering.

It is also the cheapest finding available. You reached it with zero risk to production, before injecting anything, and the work it generates — a real SLI, a percentile instead of a mean, a scrape interval shorter than the phenomenon — is work you needed anyway. This is the honest answer to "we are not mature enough for chaos engineering": the first experiment you cannot run is already telling you what to fix.

An abort condition is a pre-declared threshold, expressed in the same metrics as the steady state, that automatically reverses the injection when crossed. It must be automatic. A human watching a dashboard is not an abort condition — during the ten seconds it takes to notice, interpret and act, the thing you were worried about has already happened, and the human is the person least willing to admit their experiment is going badly.

The rule is uncomfortable but simple: without an automatic abort condition, you have not run an experiment, you have caused an outage on purpose. The distinction is not moral, it is operational: an experiment has a known worst case, and the abort condition is what makes the worst case known. It converts an unbounded risk into a bounded one, and bounding the risk is the only reason it is acceptable to do this in production at all.

Two details matter. First, the abort path must be tested before the experiment — a revert that has never been exercised is exactly as reliable as an untested fallback, which is to say unknown. Second, hitting the abort condition is a *successful* experiment: you learned that the hypothesis is false, and you learned it in a bounded window at a time of your choosing with the whole team watching, rather than at 03:00 on a holiday weekend.

The other bound is scope. Every experiment declares how much of production it can affect, and the first run of any new hypothesis takes the smallest setting that can still refute it: one instance, one shard, one percent of traffic, one non-peak hour, staging first if the hypothesis is about a mechanism rather than about production scale.

The dial only turns up after the hypothesis has held at the smaller setting. This ordering is what makes the practice defensible to the people whose revenue is at stake — you are not asking permission to break production, you are asking permission to break 1% of it for four minutes with an automatic stop, and you have already shown the same test passing at a smaller radius.

The uncomfortable truth about scaling the radius: many failure modes only appear at scale, because they are about resource exhaustion, retry amplification or correlated capacity. A one-instance experiment cannot refute a hypothesis about [[cascading-failure]]. So the dial has to move eventually, and the discipline is that it moves one notch at a time with the abort condition re-derived at each notch.