SLOssloobjectivetargetwindowreliability

SLOs: A Target, a Window, and a Reason

An SLO is an SLI plus a target plus a window: "99% of checkouts complete under 300ms over 28 days". The target is not an aspiration — it is a commitment about how much unreliability you are willing to pay to avoid, and 100% is always the wrong answer.

▶ Run the labFollow the diagnosis

Frame the diagnosis

Performance work starts from a symptom and a signal — never from a resource dashboard.

Diagnostic question
What target on this SLI, over what window, is both worth defending and actually achievable?
Symptom
Reliability discussions have no shared vocabulary. One person says the service is "pretty stable", another says it is "constantly breaking", and both are looking at the same graph. Every reliability-versus-features argument restarts from zero.
Signal
The SLI measured against a stated target over a stated window, with the remaining budget as the headline number. The misleading signal is a single bad day read in isolation — without a window, no measurement can say whether that day mattered.
SymptomSignalMeasurementHypothesisEvidenceRoot CauseChangeValidationRegression Check

The three parts, and why each is load-bearing

An SLO is an SLIs: Measuring What the User Actually Feels, a target and a window, and dropping any one of the three produces something unusable. "99.9% availability" without a window cannot be evaluated: over an hour it is trivial, over a year it is demanding. "Availability over 28 days" without a target is just a graph. And a target on an indicator nobody defined is an argument waiting to happen.

Window length changes the personality of the objective more than most people expect. A short window (7 days) reacts fast and forgives fast: a bad Tuesday dominates the number, then vanishes a week later. A long window (90 days) is stable and unforgiving: one bad afternoon can consume the whole budget and constrain the team for months. Twenty-eight days is the common compromise — long enough to smooth weekly traffic cycles, short enough that recovery is visible within a quarter, and it contains exactly four of every weekday so weekend-versus-weekday traffic patterns do not bias it.

Rolling windows and calendar windows behave differently too. A rolling 28-day window means every day an old day drops off the back, so budget recovers continuously and there is never a "fresh start". A calendar-month window resets on the first, which is operationally simpler to talk about but creates a perverse incentive to ship risky changes on the 2nd and freeze on the 28th. Most teams pick rolling for alerting and calendar for reporting, and say which is which.

One complete SLO, written the way it should be stored
Service:     checkout-api
Journey:     complete a checkout

SLI:         count(status != 5xx AND duration < 300ms)
             / count(requests excluding health checks and 4xx-from-client)
Measured at: load balancer access logs

Target:      99.0%
Window:      28 days, rolling

Budget:      1.0%  ->  approx 403 minutes of full outage per 28 days
                       (28d = 40,320 min; 1% = 403.2 min)  [ESTIMATED]

Owner:       payments-platform
Reviewed:    quarterly, and after any incident this SLO failed to catch

Why 100% is always the wrong target

A 100% target is wrong for three independent reasons, and it is worth being able to state all three. First, it is unachievable: your dependencies have their own failure rates, and the network between you and the user is not yours. Second, it is unmeasurable — at some level of reliability your monitoring is less reliable than the thing it monitors, and you are measuring instrumentation noise. Third, and most importantly, it removes the ability to make trade-offs: a team with a 100% target must treat every risk as unacceptable, which in practice means shipping nothing or ignoring the target entirely. Teams always choose the second.

The cost of each additional nine is not linear — it is closer to an order of magnitude in engineering effort, and eventually it stops being an engineering question at all. Going from 99% to 99.9% is usually retries, timeouts and a health check. Going from 99.9% to 99.99% means multi-zone deployment, tested failover and eliminating single points. Going beyond that means multi-region active-active, and the coordination cost of that architecture shows up as latency for every user on every request — you have traded speed for reliability whether or not anyone decided to.

The right target is the one where users stop noticing improvements. Above that line, additional nines cost real money and buy nothing a user can perceive. This is the argument that makes reliability a product conversation instead of an engineering preference — and it is much easier to have when you can put a number on what the next nine costs.

What each nine costs and buys — ILLUSTRATIVE orders of magnitude, not a price list
TargetBudget / 30 daysTypically requiresUser-perceptible gain
99%~432 min (7.2 h)Health checks, restarts, a runbookBaseline — outages are noticed and remembered
99.9%~43 minRedundancy, sane timeouts and retries, on-callOutages become rare enough to forgive
99.95%~22 minMulti-zone, tested failover, load sheddingMarginal for most consumer products
99.99%~4.3 minMulti-region, no single points, automated failoverUsually imperceptible; the network is now the limit
100%0Impossible — and unmeasurable before it is impossibleNothing. It buys paralysis or cynicism.

Pick the first target from behavior, not from a wish

The reliable way to get a first SLO wrong is to choose the target in a meeting. Someone proposes 99.9% because it sounds serious, nobody checks what the service currently does, and the objective is either violated on day one (so everyone stops looking at it) or so loose it never fires (so it teaches nothing).

Do it backwards instead. Take the last 28 days of the SLI you just defined, look at what the service actually delivered, and set the initial target slightly below the current performance — tight enough that a genuine regression trips it, loose enough that normal operation does not. If the service delivered 99.4%, a 99% target is a real constraint that today's system passes. You now have an objective that means something on the day you adopt it, and you can ratchet it upward as the system improves.

The latency threshold deserves the same treatment, with one refinement: pick it from the distribution, not from a round number. If the histogram has a natural shoulder at 280ms where the fast path ends and the slow path begins, a 300ms threshold sits just past it and cleanly separates "worked" from "degraded" — and it will move decisively when the fast path breaks. A 250ms threshold placed mid-shoulder produces an SLI that swings on ordinary variance and teaches the team to ignore it.

Checkout latency over 28 days — the shoulder at ~280ms is where the threshold belongsILLUSTRATIVE
120050
9800100
21400150
17600200
7900250
1800300
720500
3101000
1402500
60+Inf
p50 158 ms — The typical checkoutp95 247 ms — Still on the fast paththreshold 300 ms — Just past the shoulder — separates worked from degradedp99 690 ms — The slow path; this is what the budget is spent on

Key points

  • An SLO is SLI + target + window; without the window the target cannot be evaluated, and without the target the SLI is just a graph.
  • Window length sets the objective's personality: short windows forgive quickly, long windows let one bad afternoon constrain a quarter.
  • 100% is wrong three ways — unachievable, unmeasurable, and it destroys the ability to trade reliability against anything else.
  • Each additional nine costs roughly an order of magnitude more effort, and above some line buys nothing a user can perceive.
  • Set the first target slightly below current measured behavior, then ratchet — a target chosen in a meeting is either violated on day one or never fires.

SLO & Error Budget Calculator

Change an input and watch which number moves — and which one does not.

What the target actually buys you
ESTIMATED
error budget
43.2 min
failed requests
1,296,000
Burn it in one incident — A single outage lasting 43.2 min consumes the entire window's budget.
Burn it slowly — Or 0.100% of requests failing continuously — invisible to anyone watching a dashboard, and exactly as expensive.

A common starting point. Note how little room this leaves for planned work: a single bad deploy can consume most of the window.

Follow the diagnosis

The causal chain, hop by hop — and the readings that invite the wrong conclusion.

  1. 1
    Team → objective: picks 99.9% in a planning meeting without checking current behavior.
  2. 2
    Service → SLI: has been delivering 99.4% for months; the objective is violated on the first day.
  3. 3
    Budget → dashboard: reads negative permanently, so the number carries no information about today.
  4. 4
    Team → dashboard: stops looking, because a metric that is always red is indistinguishable from a broken metric.
  5. 5
    Incident → response: a real regression arrives and nothing changes on the wall, because the wall was already red.
What this evidence makes people conclude — wrongly
  • Reading a violated SLO as "the service is bad" when it may only mean the target was never achievable for this architecture.
  • Treating the window as a formality and comparing this week's number to a 90-day target.
  • Assuming a met SLO means users are happy — it means the fraction you chose to measure stayed above the line you chose to draw.
  • Believing more nines is strictly better, without pricing the latency and complexity that multi-region reliability adds to every request.
  • Reading a single bad day as budget exhaustion without checking what fraction of the window it actually consumed.

Measure, fix, validate

An optimization is not finished until the metric that motivated it has moved.

How to measure it
  • • Compute the SLI over the full window continuously, and publish remaining budget rather than the raw percentage — "62% of budget left" is actionable, "99.4%" is not.
  • • Record the target, window, vantage point and owner alongside the number so the objective can be audited later.
  • • Before adopting a target, back-test it against the last 28 days of real SLI data and count how many times it would have fired.
  • • Pick the latency threshold from the histogram's shoulder, not from a round number ([[histograms]]).
What actually fixes it
  • • Set the initial target from measured behavior, slightly below current performance, and ratchet upward as the system genuinely improves.
  • • Choose a 28-day rolling window unless there is a specific reason not to, and state whether it is rolling or calendar.
  • • Publish remaining error budget as the headline number, with the raw SLI as supporting detail ([[error-budgets]]).
  • • Write the full objective down — SLI, vantage point, target, window, owner, review cadence — in one reviewable place.
  • • Re-derive the latency threshold from the current distribution whenever the service's performance profile changes materially.
How you know it worked
  • • Back-test the target against the last 28 days: it should have fired for real incidents and stayed quiet otherwise.
  • • Confirm the budget number moves during a controlled failure injection and recovers as the window rolls forward.
  • • Check after one full window that the objective produced at least one useful conversation — an objective nobody has ever discussed is decoration.
What it costs
  • • A tighter target constrains release velocity by design — that is the point, and it must be a decision someone made deliberately.
  • • Rolling windows are harder to explain to stakeholders than calendar months, and "how much budget is left" changes every day.
  • • Per-journey objectives multiply maintenance; every SLO needs an owner, a review and a runbook or it becomes noise.
Stop it coming back
  • Review targets quarterly against actual behavior; an objective the system now beats by a wide margin has stopped constraining anything.
  • Alert on budget burn rather than on the raw SLI crossing the target (Burn-Rate Alerts: How Fast Is the Budget Going?).
  • Treat every incident the SLO failed to catch as a defect in the SLO, and record the resulting definition change.

Accuracy

Performance numbers are conditional. These are the conditions.

What these numbers depend on
  • ESTIMATEDBudget-minute figures are exact arithmetic on the stated window: 30 days = 43,200 minutes, so 1% = 432 min, 0.1% = 43.2 min, 0.05% = 21.6 min, 0.01% = 4.32 min. A 28-day window gives 40,320 minutes instead. They assume uniform traffic; with peaked traffic the same budget buys less user-visible downtime if spent at peak.
  • ILLUSTRATIVEThe "typically requires" column and the latency distribution are teaching shapes drawn from common architectures, not measurements of any specific system.

Misconceptions

Claim
“We should aim for as many nines as we can get.”
Reality
Each nine costs roughly an order of magnitude more and eventually buys nothing perceptible, while multi-region reliability adds coordination latency to every request. Past a point you are making the product slower to make a number bigger.
Claim
“The SLO is violated, so we failed.”
Reality
A violated SLO is information, not a verdict. It may mean the service regressed, or that the target was never achievable given the dependencies, or that traffic patterns changed. All three are worth different responses.
Claim
“A 99.9% SLO means at most 43 minutes of downtime.”
Reality
It means at most 0.1% of *events* fail. Spread evenly that resembles 43 minutes of total outage over 30 days; concentrated at peak traffic the same budget is consumed in a fraction of that wall-clock time, because far more users were affected per minute.

Apply it