SafetyGENERALSCALE-SPECIFIC
Blast Radius: If This Is Wrong, How Much Does It Affect?
The organising question of the whole domain — one test, one user, one tenant, one percent, one zone, one region, everyone — and why the honest answer is usually larger than the intended one.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
If this change is wrong, how much can it affect — and what, specifically, would stop it from affecting more?
The problem
Every production change carries a probability of being wrong and a consequence if it is. You can rarely reduce the probability much, and you can almost always reduce the consequence — but only if you can say what it currently is.
What teams do first
This is a small change, so it is low risk. It is a one-line config edit, or a flag default, or a library bump.
How it breaks
Size of diff and size of consequence are unrelated axes. A one-line change to a global configuration value applies to every instance at once; a thousand-line feature behind a flag applies to nobody until you say so (Change Size: Why Small Changes Are Safer, and When They Are Not).
How it breaks in production
- Size of diff and size of consequence are unrelated axes. A one-line change to a global configuration value applies to every instance at once; a thousand-line feature behind a flag applies to nobody until you say so (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Blast radius is a property of the change *plus its containment*, and most teams estimate it from the change alone.
- "Low risk" is an adjective, so two people can agree on it while meaning "affects one tenant" and "affects everyone". A scale makes the disagreement visible.
- The containment people assume is often the containment they have for a different kind of change — a canary contains code, and does not contain a schema migration or a shared cache format change at all.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Blast radius is the worst realistic scope of impact if a change is wrong and nothing stops it. Stated on a scale it becomes comparable across changes and across teams: one test → one user → one tenant → one percent of traffic → one zone → one region → everyone.
- Each step is roughly an order of magnitude of consequence, and the ladder is the actual subject of this domain. Canaries, flags, cells, staged regions, small changes and reversibility are all mechanisms for moving a change down it.
- The second half of the answer is what contains it. A scope with no named containment is
everyoneregardless of intent — an intended audience is not a mechanism. - Containment is per-mechanism, not global. Traffic weight contains the code path. A flag contains the behaviour it gates. A tenant rollout contains anything tenant-scoped. None of them contain shared state, and the shared surfaces are where the escalation lives.
- The dimension people forget is time. Blast radius is not static: it grows as a rollout proceeds and as bad data accumulates, and it stops growing when you stop the rollout — which is why time to detect and time to reverse are components of the radius rather than separate concerns.
The ladder
Read this as a measuring instrument. The value is that two engineers who disagree about a change will disagree about a rung, which is a conversation, rather than about the word "risky", which is not.
The third column is where most estimates go wrong. A containment mechanism that does not apply to the kind of change you are making is not containment.
| Scope | A change that sits here | What actually contains it | How it escalates one rung |
|---|---|---|---|
| one-test | A candidate receiving shadowed traffic whose responses are discarded | Nobody consumes the output (Shadow Traffic: Real Requests, Discarded Answers) | The candidate writes to the shared database, or emits a side effect |
| one-user | A flag enabled for a single internal account; a debug setting for one session | Targeting by identity, evaluated per request | The code path writes shared state that other users read |
| one-tenant | A feature released to one customer; a per-tenant configuration change | Tenant isolation in data and routing | The tenant shares a database, a cache, a rate limiter or a queue with others |
| one-percent | A canary step; a percentage flag rollout | Traffic weight, plus an abort that ends the step (Canary: One Percent, Then Five, Then Watch) | The change touches schema, shared cache format, or global configuration — none of which are routed |
| one-zone | A deploy staged to one availability zone; a zone-scoped infrastructure change | Zonal isolation of compute and, if you have it, data | A shared control plane, a cross-zone database primary, or capacity that cannot absorb the zone's traffic elsewhere |
| one-region | A regional rollout; a change to region-scoped infrastructure | Regional independence and the ability to serve from elsewhere (Region Failover) | Global services: DNS, identity, CDN configuration, a global database, the deployment pipeline itself |
| everyone | A schema migration; a global config change; a shared library release; a DNS or certificate change | Usually nothing — and saying so is the honest answer (DNS in Production, Renewal: Automating the Thing That Expires) | There is no rung above this one; what grows now is duration and the volume of bad data |
Changes that jump the ladder quietly
GENERALThese escalations follow from what a system shares rather than from any tool. The specific list of amplifiers differs per architecture — a single-tenant system has no tenant boundary to lose, a system with per-tenant databases loses far fewer of these — which is why enumerating your own shared surfaces is the practical version of this section.
Each of these is a change someone estimated at a low rung, in good faith, because the mechanism they were relying on did not apply. They are worth knowing as a list, because the pattern is not obvious from the diff.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Canary deploy that includes a migration | Every user affected while the canary is at one percent | Traffic weight routes requests; it does not route the schema | Stage the migration separately from the code, and estimate its radius on its own (A Migration and a Deploy Are One Event) |
| One-line change to a shared library | Every service that depends on it fails after its next build | The radius belongs to the dependency graph, not to the diff | Version and release shared libraries with their own staged rollout (Dependency Management) |
| Global configuration value edited | All instances pick it up nearly simultaneously | Configuration usually has no rollout mechanism at all | Treat config as a deployable with the same staging as code (A Config Change Is a Production Change) |
| Cache key or serialization format change | Errors on every instance, including ones running old code | The cache is shared and unversioned | Version cache values, or use a new key namespace so old and new coexist (Operating a Cache) |
| Feature flag default changed | Behaviour changes everywhere within seconds, with no deploy and no review | A flag flip has the reach of a deploy and none of the process (Feature Flags: Deploy Is Not Release) | Audit and stage flag changes like deploys; require review for global defaults |
| Identity provider or authentication configuration change | Nobody can log in, including the engineers responding | Authentication is a global dependency of everything, including your own tooling | Stage, keep an out-of-band access path, and rehearse recovery (Break-Glass Access) |
| Change to the deployment pipeline itself | A bad change cannot be reversed because reversal runs through the pipeline | The containment mechanism is the thing that changed | Treat control-plane changes as top-rung, and keep a manual path to deploy a known-good version |
Radius is a function of time, not a constant
The same change occupies several rungs over the course of one rollout. That is why detection time and reversal time belong inside the estimate: a change caught at step one never leaves the bottom of the ladder, and the identical change caught after full rollout reached the top.
The same defect, at each point in a progressive rollout
- Step 0changeCandidate running under shadowed traffic. Responses discarded. Radius: one-test.
- Step 1changeEnabled for internal accounts only. Radius: a handful of users who will tell you directly.
- Step 2changeOne percent of traffic. Radius: one-percent — and one percent of the data now being written in the new shape.
- Step 2 + 10msignalThe defect exists at every step, unchanged. What has changed is only how many people meet it. Detection has not happened yet.
- Step 3changeTwenty-five percent. Downstream dependencies now see meaningful load from the candidate; the radius includes their users too.
- Step 4changeFull traffic. Radius: everyone. From here it stops growing in width.
- Step 4 + 40msignalIt keeps growing in depth: every minute adds records written by the defective code, which the reversal will not undo.
- DetectionsignalAlert fires. Whatever rung the rollout is on at this moment is the radius you actually got (A Successful Deploy Is Not Evidence of a Healthy System).
- Reversal beginsactionWidth stops growing for new requests. Depth stops growing only when the last instance is reversed (Rollback: Only Useful If It Is Actually Safe).
- Reversal completerecoveryFinal radius: the rung at detection, times the duration, plus the residue that reversal did not touch (Roll Forward: When Going Back Is the Harder Option).
The steps are illustrative of a standard ramp rather than measured. The transferable point is the shape: exposure width is set by the rollout step, exposure depth by elapsed time, and both are ended by detection plus reversal — which is why those two durations are part of the radius rather than separate metrics.
changesignalactionrecovery
How to do it properly
Most important first.
- State the scope and the containment for every non-trivial change, in those words, at review time. "One percent, contained by the canary weight" is a sentence that can be checked; "low risk" is not.
- Check the containment against the change, not against the strategy: if the change touches the schema, a shared cache format or a global configuration key, the traffic-level containment does not apply (Version Coexistence: N and N+1, in Both Directions).
- Treat "contained by: nothing" as a valid and important answer. Some changes genuinely are all-or-nothing, and knowing that is the point (Destructive Changes: What a Rename Really Does).
- Include time in the estimate: how long before you would detect this, and how long to reverse it. A one-percent exposure for four hours is not the same radius as one-percent for four minutes.
- Rank shared surfaces explicitly. The database, the shared cache, the identity provider, DNS, the CDN configuration and the deployment pipeline itself are radius amplifiers — a change to any of them starts at the top of the ladder (Reducing Blast Radius).
- Match the ceremony to the scope. An
everyonechange deserves review, rehearsal and a window; aone-tenantchange deserves to ship quickly, and slowing it down teaches people that the process is noise (Guardrails, Not Gates).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Nothing contains a wrong estimate — it is the input to every containment decision downstream, so getting it wrong means the mechanisms chosen do not match the change. This lesson is the one whose failure mode is every other lesson in the module being applied to the wrong thing.
What can go wrong
Failure modes, including of the mitigation
- Radius estimated from the code and not from what the code touches — a small change to a shared library reaches every service that depends on it.
- Containment assumed from the deployment strategy while the change bypasses it entirely (schema, config, flags, DNS, IAM).
- A control-plane change: the thing that would have contained the blast — the pipeline, the flag service, the load balancer configuration — is itself what changed.
- A change scoped to one tenant that reads or writes shared state on that tenant's behalf, escalating silently.
- Detection so slow that a nominally small exposure accumulates a large amount of bad data before anyone reacts (A Successful Deploy Is Not Evidence of a Healthy System).
- Correlated containment: the "independent" zones share a control plane, a configuration source or a database, so the containment boundary is not where the diagram says it is.
Misreads this invites
- "Small change, small blast radius." Unrelated. Ask what it touches, not how large it is.
- "We canary everything, so our blast radius is one percent." One percent for the code path. Zero containment for the migration, the shared cache format or the config change that shipped with it.
- "It only affects one tenant." Only if nothing that tenant touches is shared. Check the database, the cache, the queue and the rate limiter before believing it.
- "Blast radius is about severity." It is about *scope*. A total outage for one internal test user and a subtle wrong answer for everyone are different points on this ladder, and the second is usually worse.
- "We reduced blast radius by deploying at 3am." You reduced the number of witnesses. Fewer people watching usually means longer detection, which increases the radius.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- For the last significant change, the recorded scope and containment — and, if it went wrong, whether the actual impact matched the estimate. Estimates that are never checked do not improve.
- Postmortems name the containment that failed as well as the defect that occurred (Postmortems).
- The shared surfaces in your system are enumerated somewhere, so "does this touch one" is a lookup rather than a memory test.
How you get back
- Reversibility is part of the radius, not a separate property: a change you can undo in seconds has a smaller consequence than an identical change you cannot (Rollback: Only Useful If It Is Actually Safe).
- A change with no reversal is at the top of the ladder regardless of how few users it was aimed at, because the consequence, once wrong, persists.
- Radius keeps growing until reversal completes, so rollback duration is part of the estimate rather than a footnote to it.
What to automate, and what stays human
- Automate the enforcement: a pipeline that refuses a schema change during a rollout, or requires an approval for a change to a shared surface, applies the estimate more consistently than a checklist (Policy as Code).
- Automate the labelling where the signal is mechanical — a diff touching migrations, IAM, DNS or global configuration can be flagged as high-radius automatically.
- Keep the estimate itself human. It requires knowing what a system's shared surfaces are and how a change interacts with them, which is exactly the knowledge automation lacks.
What this costs
- Reducing blast radius costs delivery speed: staged rollouts, cells, extra environments and flags all make each change slower and more complex.
- A per-change estimate is overhead, and applied to every trivial change it becomes ritual. Reserve it for changes that touch anything shared.
- Cellular and regional partitioning gives real containment and multiplies operational surface — more deployments, more monitoring, more drift, more cost (Operating in More Than One Region).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe ladder applies to any production system on any platform. Which rungs are physically available depends on architecture — a single-region single-tenant system has no
one-tenantorone-regionrung, so its ladder is shorter and its jumps are larger. - SCALE-SPECIFICWith one region, one database and a handful of instances, most changes are honestly
everyoneand the useful levers are reversibility and change size rather than partitioning. Cells, zones and regional staging become available — and worth their operational cost — only at a scale that already has them for other reasons.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Securitydatabase-privileges