Benchmarking: Does This Number Answer My Question?
A benchmark is an experiment, and most benchmarks fail as experiments before they fail as measurements. Warm-up, environment, workload realism, repetition, variance and a baseline are the difference between a number you can act on and a number you can quote.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The question comes before the benchmark
Benchmarks go wrong at design time far more often than at run time. "Is this library faster?" is not a question a benchmark can answer — faster at what, on what data, at what concurrency, on what hardware, measured how? Each of those choices changes the answer, sometimes reversing it. Writing the question down precisely is most of the work, and it exposes immediately whether the planned benchmark can answer it.
The useful form of the question names the decision it informs. "Should we replace our JSON serializer given that it is 15% of request CPU and our payloads average 4 KB with deep nesting?" tells you the workload, the metric and what would count as enough of a win. "Is library X faster than library Y?" tells you nothing and will produce a number regardless.
The corollary is that a benchmark result is scoped to its question. A serializer that wins on 4 KB nested objects may lose on 200-byte flat ones. Reporting the win without the scope is how a benchmark result outlives the conditions that produced it and gets applied somewhere it is false.
| Question | What the benchmark must reproduce | Invalidated by |
|---|---|---|
| Is serializer A faster for our payloads? | Real payload shapes and sizes sampled from production | Synthetic flat objects when production data is deeply nested |
| Does this change reduce p99? | Concurrency, contention and the full request path | A single-threaded microbenchmark of one function |
| Can we handle 2× traffic? | Production-scale data, realistic key distribution, full stack | Anything with an empty database or a warm cache |
| Is the new runtime version faster for us? | Our workload, warmed, on identical hardware | A different instance type, or a cold JIT (JIT and Warm-Up: The First Thousand Requests Are a Different Program) |
| Did this release regress? | Same hardware, same data, same request mix, both versions | Comparing yesterday's run on different infrastructure |
Hygiene that decides whether the number means anything
Six practices separate a benchmark from a number. Warm up before measuring, because JIT compilation, cache population and connection establishment all make the first iterations unrepresentative (JIT and Warm-Up: The First Thousand Requests Are a Different Program). Repeat enough times to see the spread, because a single run gives you no way to know whether a difference is real. Interleave the variants — run A, B, A, B rather than all of A then all of B — so that drift in machine conditions affects both equally rather than being attributed to one.
Then: pin the environment, since a benchmark on a noisy shared machine measures the neighbours as much as the code. Change one thing at a time, or you learn only that the combination differs. And keep a baseline — the unchanged variant, measured in the same session — because comparing today's number to a number from a previous month compares two machines as much as two implementations.
The reporting matters as much as the running. A mean alone hides everything: report the median, a spread measure, the number of iterations and the environment. "30% faster" is a claim; "median 4.2 ms vs 6.1 ms, IQR 0.3 ms, n=200, interleaved, m6i.2xlarge, commit abc123" is a result someone else can check and reproduce.
1question: "Does batching the per-item lookup reduce end-to-end p992 for the checkout path at production payload sizes?"3 4setup:5 data = sample_from_production(n = 10_000) # real shapes, real sizes6 env = dedicated_host # not a shared runner7 variants = { baseline: current, candidate: batched }8 9warm_up:10 for 2_000 iterations: run(baseline); run(candidate) # JIT, caches, pools11 discard all measurements12 13measure:14 for round in 1..100: # interleaved, not grouped15 record(baseline, time(run(baseline, next(data))))16 record(candidate, time(run(candidate, next(data))))17 18report:19 median, p95, IQR per variant # not the mean alone20 n, environment, commit, data source21 difference stated against observed run-to-run spread22 23decide:24 if difference < 2 x observed spread: "no result yet, run more"Report the distribution, not the headline
The two reports below describe the same benchmark session. One is a claim and the other is evidence. The difference is not pedantry: the second version lets a reader see that the improvement is several times the run-to-run spread, that both variants were measured under identical conditions, and that the workload resembles the production case the change is meant to help.
Most importantly, the second version can be *disagreed with* productively. A reader can say "your payloads are 4 KB but ours average 300 bytes, so this may not transfer" — a useful objection that the headline version gives no purchase for. A benchmark report that cannot be challenged on its assumptions is not communicating a result.
One habit worth adopting: state the decision the number supports. "Median improves 31%, well outside run-to-run spread; at 15% of request CPU this predicts roughly 4–5% of end-to-end request time, which does not on its own justify the migration risk" is a benchmark that has finished its job (Microbenchmark or End-to-End: Why p99 Did Not Move is the arithmetic behind that last clause).
THE CLAIM
"The new serializer is 30% faster. We should switch."
THE RESULT
question Does serializer B reduce serialization time for our payloads?
workload 10,000 payloads sampled from production (median 4.1 KB, nested)
environment dedicated m6i.2xlarge, CPU pinned, n=100 interleaved rounds
warm-up 2,000 iterations discarded
commit abc123 (baseline) vs def456 (candidate)
median p95 IQR
baseline 6.10 ms 7.40 ms 0.28 ms
candidate 4.20 ms 5.10 ms 0.31 ms
difference -31 % -31 % spread ~0.3 ms
run-to-run spread across sessions: +/- 0.4 ms (difference is ~5x spread)
SCOPE Holds for nested payloads >1 KB. On flat payloads <500 B the
two were indistinguishable in a separate run.
DECISION Serialization is ~15% of request CPU, so this predicts ~4-5%
of end-to-end request time. Real, and probably not sufficient
on its own to justify the migration.Key points
- Write the question first, naming the workload, the metric and the decision it informs; most benchmarks fail at design time.
- Warm up, repeat, interleave variants, pin the environment, change one thing, and keep a same-session baseline.
- Report median, spread, iteration count and environment — a mean alone cannot be checked or challenged.
- A difference smaller than the run-to-run spread is not a result yet, however decisive the mean looks.
- Scope the conclusion to the conditions tested, and state what decision the number actually supports.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Question → design: "is X faster" without a workload produces a benchmark whose answer depends on choices nobody recorded.
- 2Design → warm-up: the first iterations include JIT compilation and cold caches, inflating whichever variant ran first.
- 3Warm-up → ordering: running all of A then all of B attributes any machine drift entirely to the variant that ran later.
- 4Ordering → repetition: a single run per variant gives a difference with no spread to compare it against.
- 5Repetition → conclusion: an 8% difference against 12% run-to-run spread is reported as a win and does not reproduce.
- • "The mean improved, so the change is better" — the mean is the statistic most distorted by a few slow iterations.
- • "We ran it three times and it was consistent" — three runs in one session on one machine measure that session, not the change.
- • "The benchmark is faster so the service will be faster" — only in proportion to that operation's share of request time.
- • "Results differ between engineers, so benchmarking is unreliable" — usually it means neither controlled the environment or reported spread.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Run-to-run spread for your benchmark on your hardware, before trusting any comparison — this is the noise floor.
- • Median and p95 per variant across interleaved repetitions, not the mean of a single grouped run.
- • Warm-up curve: iteration time against iteration number, to confirm the discard window is long enough.
- • Environment facts that belong in the report: instance type, isolation, commit, dataset provenance.
- • State the question, the workload and the decision before writing the benchmark; discard designs that cannot answer it.
- • Add warm-up, interleaving and repetition; report median and spread rather than a mean.
- • Pin the environment or accept that shared-runner numbers need a much larger effect size to be meaningful.
- • Establish the noise floor once, then require differences to clear it by a stated multiple before acting.
- • Publish the scope and the supported decision with the number, so it cannot be reused where it is false.
- • Re-run the benchmark in a fresh session and confirm the difference reproduces at a similar magnitude.
- • Have a second person reproduce it from the reported setup; if they cannot, the report was incomplete.
- • Confirm the predicted end-to-end effect appears in an integration or production measurement, not only in the microbenchmark.
- • Rigorous benchmarking takes far longer than a quick timing, and much of the time is spent proving there is no result.
- • Dedicated hardware for benchmarking costs money and sits idle between runs.
- • Interleaved repetition multiplies run time, which matters when the benchmark is in a CI path someone waits on.
- • Keep benchmarks in CI with thresholds derived from observed noise rather than from round numbers (Regression or Tuesday? Telling a Real Change from Noise).
- • Store the environment and commit with each result so historical comparisons can be checked for confounds.
- • Re-establish the noise floor when the CI hardware changes, since every stored threshold depends on it.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe reported medians, IQRs and percentages are teaching figures showing what a complete benchmark report contains. They are not measurements of any real serializer.
- ENVIRONMENT-SPECIFICNoise floors differ by orders of magnitude between dedicated hardware and shared CI runners. Every threshold in this lesson must be derived on the machine you will actually use.