SLIs: Measuring What the User Actually Feels
An SLI is a ratio: good events over valid events. The hard parts are not the arithmetic — they are deciding what counts as good, what counts as valid, and where in the request path you measure, because each choice moves the number by more than most outages do.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Good events over valid events
Every SLI has the same shape: count the events that went well, divide by the events that should have gone well, express it as a proportion. The shape matters because it forces two definitions into the open. What is a *good* event — a 200? A 200 in under 300ms? A 200 in under 300ms carrying a non-empty result? And what is a *valid* event — every request that arrived, or only the ones the service was actually responsible for?
Availability and latency are the two SLIs almost every request-driven service needs, and they are the same formula with different predicates. Availability counts non-5xx responses over all requests. Latency counts requests faster than a threshold over all requests. Note what the latency SLI is *not*: it is not "average response time" and it is not p99. It is a count of fast-enough requests, which is a proportion — the same units as availability, so both can share a budget. That single decision is what makes the rest of this module compose.
A latency SLI stated as a proportion also sidesteps the aggregation trap in Percentiles: Which One, and How Many Users Is That?: proportions from different instances, regions and time buckets can be summed and re-divided honestly, while percentiles cannot. If you have ever tried to compute "the p99 across our twelve pods" and produced a number that was not any pod's p99, this is the fix.
1availability = count(status_code != 5xx) / count(all requests)2 3latency = count(duration < 300ms AND status_code != 5xx)4 / count(all requests)5 6quality = count(checkout completed AND payment confirmed)7 / count(checkout attempts that reached payment)8 9# note what is NOT here:10# avg(duration) -> hides the tail entirely (see averages-lie)11# p99(duration) -> a number, not a proportion; cannot share a budget12# count(5xx) -> a count, not a rate; meaningless without trafficWhere you measure moves the number more than most outages do
The same service measured at four points produces four different availability numbers, and the gap between them is not noise — it is the part of the user experience each vantage point cannot see. Server-side instrumentation is blind to every request that never arrived: DNS failures, TLS handshake timeouts, load-balancer 502s, and the mobile client whose connection died in a lift. Those are exactly the failures users describe as "the app is broken".
Move outward and you see more, but you pay for it. Load-balancer logs catch backend failures the service never logged. CDN or edge telemetry catches regional network problems. Real-user monitoring in the client catches everything including the user's own terrible café Wi-Fi — which is honest, but it also means your SLO now includes failures you cannot fix, and an error budget you cannot control is a budget nobody will respect.
The usual resolution is to measure at the load balancer or edge for the primary availability SLI, and to keep client-side RUM as a separate, unbudgeted signal that tells you when the gap between the two widens. What matters far more than picking the "right" point is writing down which point you picked, because six months later someone will compare your 99.95% to another team's 99.9% and the comparison will be meaningless unless both state their vantage point.
| Measured at | Availability reads | Catches | Blind to |
|---|---|---|---|
| Application code | 99.98% | Handler exceptions, dependency errors | Requests that never arrived; LB 502s; TLS failures |
| Load balancer | 99.94% | Backend 5xx, connection refusals, timeouts | DNS failures, client network, regional routing |
| CDN / edge | 99.91% | Regional outages, origin unreachability | Last-mile client conditions |
| Real-user monitoring | 99.7% | Everything the user experienced | Nothing — including failures you cannot fix |
Choosing valid events is the half everyone skips
The denominator decides what the SLI is willing to blame you for, and getting it wrong produces an indicator that either flatters you or punishes you for other people's problems. Include health checks and you have diluted the ratio with thousands of trivially successful requests until real failures cannot move the number. Include requests that returned 400 because a client sent malformed JSON and you are now spending budget on other teams' bugs. Exclude too much and the SLI quietly stops covering the failures that matter.
The workable default: valid events are requests the service was responsible for serving correctly. That excludes synthetic health checks and excludes 4xx caused by client error — but not 429 and not 401-storms caused by your own token service, both of which users experience as breakage. It also excludes traffic during a declared maintenance window only if your users were genuinely told; otherwise you are just hiding.
Then there is the aggregation question. One SLI over all traffic lets a healthy high-volume endpoint mask a completely broken low-volume one — checkout can be 100% down while /health and /search keep the aggregate at 99.9%. Splitting per critical user journey costs you more SLOs to maintain but buys an indicator that actually moves when something users care about breaks. Split by journey, not by endpoint: "complete a checkout" is a journey; POST /api/v2/cart/items is an implementation detail.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Aggregate availability (all routes) | 99.91% | Inside the 99.9% objective — nothing to see | normal |
| Request volume | 48k/min, steady | No traffic anomaly to explain a complaint | normal |
| Availability, checkout journey only | 71.4% | Better than one in four checkouts is failing | smoking gun |
| Checkout share of total traffic | 0.6% | Too small to move the aggregate by more than a rounding error | suspect |
| Server-side 5xx rate | 0.02% | The failures are 200s with an empty cart — not errors at all | suspect |
Key points
- Every SLI is
good events / valid events— a proportion, which is why latency SLIs are stated as "fraction of requests under X", not as a percentile. - Where you measure changes the number by more than most incidents do; the vantage point is part of the SLI definition, not a footnote.
- Server-side error rate is blind to every request that never arrived — the exact failures users describe as "the app is broken".
- The denominator is a policy decision: exclude health checks and genuine client errors, never exclude 429s or your own auth failures.
- One aggregate SLI lets a high-volume healthy endpoint mask a fully broken critical journey; split per user journey, not per endpoint.
Progressive depth
Overview
An SLI is a number between 0 and 1 that says what fraction of user interactions went well. Good events divided by valid events. That is the whole idea.
Practical
Pick two per critical journey: availability (non-5xx / all) and latency (under-threshold / all). State the vantage point. Exclude health checks and genuine client errors from the denominator; do not exclude 429s or your own auth failures.
Advanced
Split by user journey rather than endpoint, because aggregates hide low-volume critical flows. Track the gap between client-side RUM and server-side measurement as its own signal — a widening gap means failures are occurring outside your instrumentation entirely.
Internals
Proportions compose and percentiles do not: sum(good)/sum(valid) across instances and time buckets is exact, whereas averaging p99s across pods produces a number that is nobody's p99. This is why SLIs are ratios of counters, why histogram buckets rather than pre-computed quantiles are the right storage (Histograms: A Distribution You Can Afford to Keep Forever), and why the latency SLI threshold must be chosen at bucket boundaries or the count is an interpolation.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1User → client: request fails at TLS or DNS; nothing reaches the load balancer, so nothing is counted.
- 2Client → support: "the app does not work"; support has no signal that corresponds to the complaint.
- 3Service → metrics: handler-level error rate reads 0.02%, because only handled requests are in the denominator.
- 4Aggregate SLI → dashboard: 99.91%, comfortably inside objective, because checkout is 0.6% of traffic.
- 5Team → conclusion: "must be the user's network" — a conclusion the SLI was constructed to be unable to disprove.
- • Reading a green aggregate SLI as "no user is affected", when it only means no *large fraction* of users is affected.
- • Treating average latency as a latency SLI; the average is inside objective while the slowest 5% of sessions are unusable (The Average Was Fine and Users Were Not).
- • Assuming a low 5xx rate means low failure rate — 200-with-empty-body and client-side timeouts are invisible to it.
- • Comparing two teams' availability numbers without checking whether they measure at the same point.
- • Excluding all 4xx from valid events, thereby excluding 429s that your own rate limiter produced during an incident.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Compute `good/valid` per user journey over a rolling window from the same telemetry the user's request produces — not from a separate synthetic probe.
- • State the vantage point explicitly in the SLI definition ("measured at the load balancer, excluding health checks").
- • Track the RUM-versus-server gap as its own series: a widening gap means failures are happening outside your instrumentation.
- • Break availability down by journey and check that the lowest-volume critical journey is visible in its own series.
- • Define SLIs per critical user journey with an explicit numerator, denominator and vantage point, written down where reviewers look.
- • Move the primary availability measurement outward to the load balancer or edge so requests that failed before the service are counted.
- • State latency SLIs as proportions ("99% of checkouts complete under 300ms") so they share units and a budget with availability.
- • Add RUM as a separate unbudgeted signal and alert on the divergence between it and the server-side number.
- • Review the denominator explicitly: list what is excluded and why, and re-review it after every incident that the SLI failed to catch.
- • Replay the last three incidents against the proposed SLI: if it does not visibly dip during each one, it is not measuring what users felt.
- • Confirm the new journey-level SLI moves during a deliberate synthetic failure of that journey in staging.
- • Check that aggregate and per-journey SLIs disagree in the expected direction during a low-volume failure — if they never diverge, the split is not doing any work.
- • Journey-level SLIs multiply the number of definitions to maintain, and each one needs an owner or it rots.
- • Measuring at the edge includes failures you cannot fix, which makes the budget less actionable even though it is more honest.
- • Client-side RUM costs egress and adds a privacy surface — you are now collecting per-session data that must be handled under the same rules as any other user data.
- • Alert on the SLI, not on the resources beneath it, so a change that breaks users fires regardless of which layer caused it (Alerts Worth Waking Someone For).
- • Review SLI definitions quarterly and after any incident the SLI missed; record the change in the same place as the SLO.
- • Add a test that fails if a new endpoint joins a critical journey without being included in that journey's SLI.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe four-vantage-point availability numbers show the direction and rough magnitude of the gap; the real gap depends entirely on client population, network conditions and where failures actually occur.
- ENVIRONMENT-SPECIFICWhich vantage point is even available depends on your stack: a managed edge may expose per-request logs, a bare load balancer may only expose counters.