Availability, SLOs and Error Budgets

A Service Level Indicator is a measurement: the fraction of HTTP requests in the last 30 days that returned a non-5xx status within 500 ms. A Service Level Objective is the internal target for that SLI: 99.9%. A Service Level Agreement is a contract with a customer, with a looser number and financial consequences: 99.5%, with service credits below it. The SLO is stricter than the SLA so that you notice and act before the penalty clause does. The three terms are confused constantly; the fix is to remember that only the SLI is a *fact*, and the other two are *decisions* about it.

The properties themselves are distinct. Availability is the fraction of time — or, better, of requests — during which the service does what it should. Durability is the probability that stored data is not lost: S3 promises eleven nines of durability and four nines of availability, which is exactly right — a temporarily unreachable object is fine, a lost one is not. Reliability is the broad term: the service does the right thing, correctly, consistently. A search that returns instantly but wrong is available and unreliable. Define the SLI so that the failure your users actually care about counts — a login that returns 200 in 4 s is not "up" to the person waiting.

Availability is almost always quoted as a percentage, and percentages hide the number that matters: how long you are allowed to be down. Each extra nine divides the allowance by ten. From 99.9% to 99.99% is the step from "one bad afternoon a year" to "52 minutes a year, in total, including every deploy that goes wrong" — and that step typically costs more than all the previous ones combined, because at four nines a human cannot be in the loop for recovery: 52 minutes is roughly one page, one diagnosis, one rollback. Five nines means nobody is ever in the loop; every failure must be detected and mitigated automatically in seconds.

Availabilities are probabilities, and they compose like probabilities. A request that must pass through a load balancer, a service, and a database in series succeeds only if all three do: 0.9999 × 0.999 × 0.999 ≈ 0.998, or 99.8% — worse than the weakest component, which is why a service cannot promise more nines than the dependency it cannot avoid. Ten dependencies at 99.9% each, all in series, give 0.999^10 ≈ 0.990: a 99% system built from 99.9% parts. This is the hidden cost of a synchronous call chain in Microservices and the reason Request/Response vs Event-Driven matters for availability, not just for latency.

Redundancy composes the other way. Two independent instances in parallel, either of which can serve the request, fail only if both do: 1 − (0.01 × 0.01) = 0.9999, so two 99% servers behind a load balancer give 99.99% — *if* their failures are independent, which shared power, a shared deploy, or a shared bad config make untrue. The pattern that actually delivers nines is therefore: remove serial dependencies from the critical path (cache, degrade, queue), and add independent parallel capacity to what remains.

An SLO of 99.9% over 30 days means you are *allowed* 43 minutes of failure. That allowance is the error budget, and treating it as a budget changes behaviour. While budget remains, the team ships: feature releases, risky migrations, chaos experiments — all are spending from a balance that exists to be spent. When the budget is exhausted, releases stop except for reliability fixes, until the 30-day window rolls enough failure out. The policy replaces the argument between "ship faster" and "be more careful" with a number both sides agreed to in advance.

Alerting follows from the budget too. Alert on burn rate: the multiple of the sustainable rate at which the budget is being consumed. At 99.9%, a burn rate of 14.4 over one hour means 2% of the monthly budget has gone in that hour — page someone. A burn rate of 1 over three days means the budget will be exactly spent by month end — open a ticket. Symptom-based alerting on the SLI ("users are seeing errors") replaces cause-based alerting ("CPU is at 90%"), because a cause without a symptom is not an incident and a symptom without a known cause still is.

A server that reports 100% success while DNS is broken, the CDN is misconfigured, or the load balancer is routing to a dead pool is honestly reporting that every request *it received* succeeded. The SLI that matters is measured as close to the user as possible: synthetic probes from outside your network that exercise the real path (DNS → TLS → LB → app → DB) every minute from several regions; client-side telemetry that reports request outcomes from the browser or mobile app, including the ones that never reached you; and load-balancer logs as the server-side measurement of record, because they see the request before any service does. A request the user sent and you never saw is still a failed request from the SLI’s point of view.

Define the SLI over requests, not time: "99.9% of requests succeed" is more honest than "up 99.9% of the time", because a system that is up but returning errors for one region or one endpoint is partially down, and time-based availability rounds that to zero. The exercise in Logs, Metrics and Traces shows what the histograms behind a latency SLI look like.