The Average Was Fine and Users Were Not
Five requests at 50, 55, 52, 48 and 3000ms have a mean of 641ms — a number no single request experienced. The mean is the wrong summary for latency because one extreme value drags it away from everything, and it cannot distinguish "everyone is slightly slow" from "one user in a hundred is unusable".
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The five requests
Take the canonical example: five requests measured at 50, 55, 52, 48 and 3000ms. The mean is 641ms. Not one request took anything close to 641ms — four were around 50ms and one was three seconds. The mean has landed in a region of the distribution where no traffic exists, which is a good working definition of a useless summary statistic.
The median is 52ms and describes four of the five requests well. The maximum is 3000ms and describes the fifth. Between them they tell the whole story in two numbers; the mean tells a story that is true of nothing.
Scale this up and it gets worse rather than better. At a million requests a day, one percent hitting three seconds is ten thousand bad experiences — and because the other 99% are fast, the mean barely moves. The averaging that hid one bad request among five hides ten thousand among a million even more effectively.
Same mean, different systems
The deeper problem is that the mean is not injective: wildly different distributions produce identical averages, so a flat mean is not evidence that the distribution has not changed. Two services can both report a 91ms average while one is uniformly mediocre and the other is fast for everyone except a slice of users for whom it is broken.
These two systems need completely different responses. The uniformly-mediocre one has a systemic cost — every request pays it — and the fix is usually algorithmic or architectural, affecting everyone. The bimodal one has a specific population hitting a specific path: a cold cache, a shard that lost its replica, a customer with a hundred times more data than anyone else. Chasing the wrong one wastes the whole investigation.
This is why percentiles are not a refinement of the average but a replacement for it. p50 tells you about the typical experience, p99 tells you about the tail, and the gap between them tells you which of these two systems you are looking at — see Percentiles: Which One, and How Many Users Is That? for how to read them together and Tail Latency: Why p50 Being Fine Does Not Help for why the tail dominates at scale.
| Reading | System A: uniformly mediocre | System B: bimodal tail |
|---|---|---|
| mean | 91ms | 91ms |
| p50 | 88ms | 47ms |
| p99 | 140ms | 1,400ms |
| p99 / p50 ratio | 1.6 — narrow distribution | 30 — the tell for a bimodal split |
| Who is affected | Everyone, slightly | 1 in 100, severely |
| Likely cause | A cost paid on every request: serialization, an added hop, a slower algorithm | A subset hitting a different path: cold cache, degraded replica, oversized tenant |
| Right first move | Profile the common path (see Self Time, Total Time, and Where the CPU Went) | Sample traces from the slow tail (see Sampling Without Throwing Away the Evidence) |
What the average is genuinely for
The mean is not a broken statistic; it is the right tool for a different job. For anything you sum — total CPU seconds consumed, bytes transferred, cost per request, capacity required — the mean is exactly correct, because the sum is what you actually pay and the mean times the count is the sum. Capacity Planning: Traffic to Machines and Cost per Request: The Other Performance Metric both run on averages and should.
The rule is: use the mean for questions about aggregate resource consumption, use percentiles for questions about experience. "How many cores do I need" is a mean question. "Is the site fast" is a percentile question. Answering the second with the first is the error, and it is extremely common because the mean is the default in most tooling.
One more property worth knowing: means can be averaged, percentiles cannot. If you need to combine per-instance numbers, a weighted mean is valid arithmetic and a mean of p99s is not (see Histograms: A Distribution You Can Afford to Keep Forever for the aggregation that does work).
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| mean latency | 91ms (was 74ms) | Up 23% — looks like mild degradation, easy to dismiss | normal |
| p50 latency | 47ms (was 46ms) | Unchanged: the typical user is completely unaffected | normal |
| p99 latency | 1,400ms (was 180ms) | Up 7.8x — this is the incident | smoking gun |
| p99 / p50 ratio | 30 (was 3.9) | The distribution has gone bimodal; a subset is on a different path | smoking gun |
| request rate | 1,150/s (was 1,140/s) | Traffic is flat — this is not a load problem | normal |
Key points
- The mean of a latency distribution frequently describes no actual request, because one extreme value drags it into empty space.
- A flat mean is not evidence that the distribution is unchanged — very different distributions share the same average.
- The p99/p50 ratio distinguishes "everyone slightly slow" from "a subset severely broken", and those need opposite investigations.
- Use the mean for aggregate resource questions (capacity, cost, total CPU) where the sum is what you pay.
- Means can be averaged across instances; percentiles cannot — that requires merging distributions.
Percentile Explorer
Change an input and watch which number moves — and which one does not.
The tail is now heavy enough to drag the mean above p95 — the average has stopped describing any request that actually happened.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Subset → latency: 1% of requests start taking ~1.4s because of a path only they touch (cold cache, degraded replica, oversized tenant).
- 2Latency → mean: 99% of fast requests dominate the sum, so the mean rises only from 74ms to 91ms.
- 3Mean → dashboard: the average-latency panel moves within its normal band and triggers no alert.
- 4Dashboard → responder: no page fires; the incident is discovered through support tickets hours later.
- 5Responder → percentiles: p99 shows a 7.8x rise that was present in the data the entire time, unread.
- • "Average latency is fine, so the service is fine" — the average is specifically bad at detecting the failures that affect a minority severely.
- • "The mean went up 23%, everything got 23% slower" — nothing got 23% slower; 1% got 8x slower and the mean absorbed it.
- • "p99 is noisy, the mean is more stable" — the mean is stable because it is insensitive to exactly what you are looking for.
- • "We can add the p99s from each instance and divide" — that produces a number with no statistical meaning (see Histograms: A Distribution You Can Afford to Keep Forever).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Always graph p50 and p99 on the same panel; the gap between them is the diagnostic, not either number alone.
- • Track p99/p50 as its own series — a sudden rise means the distribution shape changed even if both endpoints look tolerable.
- • Compare against the same weekday-hour last week rather than against an hour ago, so daily traffic shape does not masquerade as a regression.
- • When the mean moves but p50 does not, look directly at the tail: the movement is coming from a small, severely affected population.
- • Replace average-latency panels with p50/p90/p99 on one chart; keep the mean only where an aggregate-consumption question is being asked.
- • Define SLOs on percentiles rather than averages, so the objective tracks experience (see [[slo]]).
- • Add p99/p50 ratio as a first-class series so distribution-shape changes are visible without reading two graphs.
- • When the tail moves, pull traces from the slow population specifically rather than sampling uniformly (see [[trace-sampling]]).
- • Confirm the fix moved p99 specifically — a change that only moves the mean has probably shifted load rather than fixed the tail.
- • Check p99/p50 has returned toward its historical ratio, not merely that p99 came down while traffic dropped.
- • Verify against the same weekday-hour baseline from before the regression, not against the incident hour.
- • Percentiles are noisier at low traffic — p99 of a 10 req/s endpoint moves on single requests and can generate false pages.
- • Storing distributions costs more than storing a mean (see [[histograms]]); the tail visibility is what you are buying.
- • The p99/p50 ratio is an extra series to maintain and explain, and it is meaningless when traffic is very low.
- • Alert on percentiles and on the p99/p50 ratio; never on average latency alone.
- • Add a p99 assertion to the load-test suite so a tail regression fails CI before it reaches production (see Regression or Tuesday? Telling a Real Change from Noise).
- • Keep the mean on capacity dashboards and out of experience dashboards, so the two questions are never confused again.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 50/55/52/48/3000ms example, the two 91ms-mean distributions and every percentile here are constructed to make the arithmetic verifiable. They are teaching shapes, not measurements from a real service.
- WORKLOAD-SPECIFICHow badly the mean hides a tail depends on the tail's size and severity. A 10% affected population moves the mean visibly; a 0.1% population moves it almost not at all, while still representing thousands of users at scale.