Error Budgets: Unreliability You Are Allowed to Spend
If the objective is 99.9%, then 0.1% of failure is not a defect — it is a budget. Treating it as something to spend rather than something to avoid turns every reliability-versus-velocity argument into an arithmetic question, which is the only version of that argument anyone ever wins.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The arithmetic, and what it actually buys you
The budget is the complement of the objective: budget = 1 − SLO. At 99.9% over 30 days, 0.1% of events may fail. If traffic were uniform, that is about 43 minutes of complete outage — and the "if traffic were uniform" is doing real work in that sentence, which we will come back to.
The reframing is the whole point. Unreliability stops being a moral failure and becomes a resource with a quantity. You can spend it on a risky migration, on a deliberately fast release cadence, on a chaos experiment, or on nothing at all. Budget left over at the end of a window is not a prize; it is evidence you could have moved faster and chose not to. That framing is what converts an unwinnable argument about values into a decision about allocation.
The subtlety that trips teams up: the budget is denominated in events, not minutes. A 43-minute equivalence assumes failures are spread evenly across the window. Spend the same 0.1% during Black Friday peak and it may last four minutes of wall-clock time, because the event rate is an order of magnitude higher — and the number of users harmed is correspondingly higher too. This is exactly right, and it is why event-denominated budgets are better than time-denominated ones: they price an outage by how many people it hurt, not by how long a clock ran.
| Objective | Budget | Per 30 days | Per 28 days | Per 7 days |
|---|---|---|---|---|
| 99% | 1% | 432 min (7.2 h) | 403 min (6.7 h) | 101 min |
| 99.5% | 0.5% | 216 min (3.6 h) | 202 min | 50 min |
| 99.9% | 0.1% | 43.2 min | 40.3 min | 10.1 min |
| 99.95% | 0.05% | 21.6 min | 20.2 min | 5.0 min |
| 99.99% | 0.01% | 4.3 min | 4.0 min | 1.0 min |
Spending it on purpose
A budget only changes behavior if something happens when it runs low, and that something has to be agreed in advance. Deciding mid-incident what a depleted budget means produces a negotiation, not a policy. The usual structure is a small ladder: comfortable budget means ship normally and take reasonable risks; depleted budget means reliability work takes priority over feature work until the window recovers; and somewhere between the two, extra scrutiny on risky changes.
The policy has to bind both directions to be credible. If a healthy budget does not actually license faster shipping, engineers correctly conclude the budget is a stick rather than a tool, and start gaming the SLI definition instead. Teams that get real value from budgets are the ones that visibly *spend* them — running the risky migration in a window with room, shipping on Fridays when there is budget for it.
Who owns the decision matters too. The budget belongs jointly to engineering and product, because it is the exchange rate between the two things they each optimise for. An error budget owned solely by engineering becomes a veto; owned solely by product it becomes a formality. The version that works has both parties looking at the same number and agreeing in advance what each threshold triggers.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Budget remaining (28d rolling) | 18% | Low, but not gone — risky changes need justification | suspect |
| Burn rate, last 1h | 0.4x | Currently spending slower than the window allows | normal |
| Burn rate, last 24h | 3.1x | Yesterday consumed roughly 10% of the window budget | suspect |
| Days remaining in window | 11 | Budget recovers as the oldest days roll off | normal |
| Largest single consumer | Deploy v4.12 — 61% of budget | One change owns most of the spend; that is the review conversation | smoking gun |
When it runs out
Exhaustion should be undramatic, because it was planned for. The standard response is a freeze on risky changes — not on all changes, and specifically not on reliability fixes, which are the way out. A blanket freeze that blocks the fix is a policy that has confused the goal with the mechanism.
Recovery depends on the window type, and this catches people out. With a rolling window, budget returns gradually as bad days age out of the back — after a bad Tuesday, you are constrained for about four weeks, easing daily. With a calendar window, everything resets at once on the first of the month, which is simpler to explain and creates a visible cliff in behavior around month boundaries that some teams find distorting.
One honest caveat: error budgets are a coordination tool, and they work when the organisation actually respects them. If leadership overrides a freeze the first time it is inconvenient, the budget becomes theatre and everyone learns to treat it as such. It is better to have no budget policy than one that is publicly ignored, because the second teaches the team that reliability commitments are negotiable under pressure — which is precisely the belief you were trying to eliminate.
| Budget remaining | Release policy | What gets prioritised | Who decides |
|---|---|---|---|
| > 50% | Ship normally; take deliberate risks | Feature work; run the risky migration now | Team |
| 20–50% | Ship, with extra review on risky changes | Feature work plus known reliability debt | Team |
| 5–20% | Risky changes need explicit sign-off | Reliability work moves ahead of new features | Team + product owner |
| 0–5% | Freeze risky changes; reliability fixes still ship | Reliability only | Product owner + engineering lead |
| Exhausted | Freeze holds until the window recovers | Root-cause fixes and their validation | Escalation, agreed in advance |
Key points
- Budget = 1 − SLO. It is a resource to allocate, not a defect count to drive to zero.
- Budgets are denominated in events, not minutes: the same 0.1% spent at peak harms far more users and lasts far less wall-clock time.
- The policy must bind in both directions — a healthy budget licensing faster shipping is what makes the constraint credible.
- Exhaustion freezes risky changes, never reliability fixes; a freeze that blocks the fix has confused mechanism for goal.
- A budget policy that leadership overrides under pressure is worse than none, because it teaches the team that the commitment is negotiable.
Progressive depth
Overview
If you promise 99.9%, you have permitted 0.1% failure. That permitted fraction is the error budget: a quantity you can spend rather than a defect you must eliminate.
Practical
Publish remaining budget and burn rate as the headline numbers. Agree in advance what happens at each threshold, including what a healthy budget licenses. Exempt reliability fixes from freezes explicitly.
Advanced
Budgets are denominated in events, so spend at peak costs disproportionately more users per wall-clock minute. Attribute spend to deploys and incidents so the largest consumer is obvious. Rolling windows recover gradually; calendar windows reset at a cliff and distort behavior near month boundaries.
Internals
Remaining budget is 1 − (bad_events / (valid_events × (1 − SLO))) over the window — a ratio of ratios, which is why it can exceed 100% consumed and read negative. For low-traffic services the binomial confidence interval on the SLI can be wider than the entire budget: at 1,000 requests a day and a 99.9% objective, the window contains roughly 28 permitted failures, and ordinary variance alone moves that by a meaningful fraction. Below some traffic volume the budget is statistically indistinguishable from noise, and the honest response is a looser objective or a longer window, not a more precise dashboard.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Objective → budget: 99.9% over 28 days allows 0.1% of events to fail.
- 2Deploy → SLI: v4.12 raises the error rate to 1% for 90 minutes before rollback.
- 3Burn → budget: at 10x the allowed rate for 1.5 hours, roughly 61% of the window's budget is consumed in one afternoon.
- 4Budget → policy: remaining budget crosses below 20%, moving reliability work ahead of features by prior agreement.
- 5Window → recovery: with a rolling window the spend ages out over the following four weeks rather than resetting.
- • Treating leftover budget as a success metric to maximise, rather than as unused capacity for change.
- • Converting the budget to minutes and forgetting the uniform-traffic assumption, then being surprised when a peak-hour incident costs far more.
- • Reading a low burn rate right now as safety, without checking the 24-hour rate that reflects yesterday's spend.
- • Assuming budget exhaustion means the service is unusable — it may mean one bad deploy consumed the allowance while the service is currently healthy.
- • Blaming the on-call engineer for spend caused by an architectural single point of failure.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Publish remaining budget as a percentage of the window allowance, refreshed continuously, alongside burn rate over 1h and 24h.
- • Attribute spend to its causes — deploys, incidents, dependency failures — so the review conversation has a subject.
- • Track budget in events, and show the wall-clock equivalent only with the uniform-traffic assumption stated.
- • Record every freeze and every override, so the policy's actual observance can be audited later.
- • Agree the policy ladder in advance with product, including what a healthy budget licenses, and write it down.
- • Publish remaining budget and burn rate as the headline reliability numbers, above the raw SLI ([[slo]]).
- • Attribute spend to deploys and incidents automatically, so the largest consumer is visible without investigation.
- • Exempt reliability fixes from any freeze explicitly, in the policy text, before the first freeze happens.
- • Escalate deliberately when the budget is exhausted — a named decision, recorded, rather than an informal override.
- • After a window with a freeze, confirm the budget actually recovered and the freeze lifted on the agreed condition rather than on mood.
- • Check that at least one risky change was consciously scheduled into a healthy-budget period — evidence the policy works in both directions.
- • Audit the last quarter for overrides; more than an occasional one means the policy is not real.
- • Budget policy adds process to releases, and in a low-traffic service the budget is statistically noisy enough that the process may cost more than it buys.
- • Event-denominated budgets are harder to explain to non-technical stakeholders than "minutes of downtime".
- • Freezes have a real product cost; the policy only pays off if the reliability work done during them actually reduces future burn.
- • Alert on burn rate rather than on budget exhaustion, so the warning arrives while there is still budget to protect (Burn-Rate Alerts: How Fast Is the Budget Going?).
- • Re-derive budget arithmetic whenever the SLO target or window changes; stale budget figures outlive their objectives constantly.
- • Review budget attribution after each incident so recurring consumers become visible as a pattern rather than as isolated events.
Accuracy
Performance numbers are conditional. These are the conditions.
- ESTIMATEDAll budget-minute conversions are exact arithmetic on the stated window under uniform traffic: 30 days = 43,200 minutes, 28 days = 40,320, 7 days = 10,080. Real traffic is peaked, so wall-clock equivalence is an upper bound on time and an under-statement of user impact at peak.
- ILLUSTRATIVEThe policy ladder thresholds (50% / 20% / 5%) are a common starting shape, not a standard. Low-traffic services need wider bands because the SLI is noisier.