Benchmark Fallacies: Confident Numbers That Are Wrong
Different environments, no warm-up, unrealistic payloads, averages without variance, several variables at once, and measuring something the system never actually does. Each produces a decisive number, and each is a reason to refuse to act on it.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Eight ways to produce a confident wrong number
These are not exotic mistakes; they are what happens by default when someone times something without treating it as an experiment. Each one is individually plausible and each produces a number that looks fine. Recognizing them on sight is a practical skill, because the results arrive as screenshots in review threads without their setup attached.
The common thread is that every fallacy makes the benchmark measure something adjacent to the question. The timing is real, the arithmetic is right, and the thing being timed is not the thing the decision is about. That is why arguing about the number is usually futile — the productive question is always "what exactly was measured, on what, and how many times?"
The last row is the one that survives the most scrutiny while being the most wasteful: a technically flawless benchmark of a code path that accounts for a negligible share of real work. It answers its question perfectly and the question was not worth asking.
| Fallacy | Why it happens | What it produces |
|---|---|---|
| Different environments | Baseline from an old run on other hardware | A hardware comparison presented as a code comparison |
| No warm-up | Timing starts at iteration one | JIT, cold caches and cold pools attributed to whichever variant ran first |
| Unrealistic payloads | Synthetic test data is easy to generate | A result that reverses on production data shapes |
| Mean without variance | The tool reports a mean by default | A difference indistinguishable from noise, stated as a fact |
| Multiple variables at once | Several changes shipped together | A combined effect that cannot be attributed or partially reverted |
| Grouped rather than interleaved runs | Run all of A, then all of B | Machine drift assigned entirely to the second variant |
| Averages hiding the tail | p99 is not reported | A change that improves the mean and worsens the tail passes review |
| Benchmarking the wrong thing | The hot path was assumed, not profiled | A flawless measurement of something that is 0.4% of request time |
The same change, benchmarked two ways
Below is one change measured twice. The left column contains four fallacies at once and reports a large, actionable-looking win. The right column measures the same change properly and reports a smaller, real, correctly scoped one.
Note that the fallacious version is not fabricated — every number in it was genuinely observed. That is what makes this class of error persistent: nobody is lying, and the numbers are reproducible in the sense that re-running the same flawed setup gives the same flawed answer. Reproducibility is not validity.
The practical response to a suspicious benchmark is not to argue about the conclusion but to ask for the setup: what data, what hardware, how many runs, what spread, warm-up or not, interleaved or grouped, and what share of real request time does this operation hold? Most contested benchmark results resolve themselves once those six answers are on the table.
1baseline: measured 3 weeks ago, c5.large, no warm-up2candidate: measured today, m6i.2xlarge, warm3data: synthetic flat objects, 200 bytes4runs: 1 each, grouped5reported: mean 8.4 ms -> 2.1 ms "4x faster!"6 7WHAT WAS ACTUALLY COMPARED8 two instance types, a cold run against a warm one,9 on data that does not resemble production payloads,10 with no spread and no baseline in the same session.1baseline: same session, same host, warmed2candidate: same session, same host, warmed3data: 10,000 payloads sampled from production (median 4.1 KB)4runs: 100 interleaved rounds each5reported: median 6.10 ms -> 5.05 ms (-17%)6 IQR 0.28 / 0.31 ms, spread across sessions +/- 0.4 ms7 8SCOPE 17% of serialization time; serialization is 15% of9 request CPU -> ~2.5% of end-to-end request time.10DECISION Real but small. Worth taking if the change is cheap;11 not worth a migration on its own.Both sets of numbers were really observed. The left one measured hardware, warm-up state and payload shape while believing it measured the code change — which is why "we re-ran it and got the same thing" is not a defence.
Benchmarking something the system never does
The most expensive fallacy is also the most rigorous-looking. A team profiles nothing, assumes a hot path, and benchmarks it beautifully — warm-up, repetitions, interleaving, spread, the lot. The result is trustworthy and irrelevant, because the function they optimized accounts for a fraction of a percent of real request time.
The guard is cheap: before benchmarking anything, get its share of real work from a profile (Self Time, Total Time, and Where the CPU Went, Reading a Flame Graph). If a function is 0.4% of request CPU, the best possible outcome from optimizing it is 0.4%, no matter how much faster you make it in isolation. That single number determines whether the benchmark is worth writing.
The same check applies to the workload itself. Benchmarking a code path with an input distribution the system never receives — all cache hits, all small payloads, all happy path — measures a scenario rather than the service. Sample the inputs from production and the benchmark inherits reality for free.
PROFILE SHARE OF REQUEST CPU (production sample, 1 hour) json serialization 15.2 % template rendering 11.8 % auth token validation 6.4 % date formatting 0.4 % <-- the one that was benchmarked everything else 66.2 % THE BENCHMARK THAT WAS RUN date formatting: 340 ns -> 95 ns (-72 %, rigorously measured) WHAT IT BUYS END TO END 0.4 % x 72 % = 0.29 % of request CPU WHAT THE SAME EFFORT ON SERIALIZATION WOULD BUY 15.2 % x 30 % = 4.6 % of request CPU (16x more, same effort) The benchmark was correct. The target was chosen without a profile.
Key points
- Every fallacy here produces a real, reproducible number that measures something adjacent to the question.
- Reproducibility is not validity — re-running a flawed setup reproduces the flaw exactly.
- Ask for the setup, not the conclusion: data, hardware, runs, spread, warm-up, ordering, and profile share.
- A rigorous benchmark of a 0.4% code path is bounded at 0.4%, however impressive the isolated speedup.
- Get the profile share before writing the benchmark; it decides whether the benchmark is worth writing at all.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Assumption → target: a hot path is assumed without a profile, and the benchmark is written for it.
- 2Target → setup: synthetic small payloads are generated because production data is inconvenient to sample.
- 3Setup → comparison: baseline comes from an older run on different hardware, so instance type enters the measurement.
- 4Comparison → statistic: a single run per variant reports a mean, with no spread to judge the difference against.
- 5Statistic → decision: a 4× headline drives a migration whose true end-to-end effect is a fraction of a percent.
- • "We re-ran it and got the same number" — a flawed setup is perfectly reproducible; that is not evidence of validity.
- • "It is 4× faster in isolation, so it will help a lot" — bounded by the operation's share of real work.
- • "Synthetic data is cleaner for benchmarking" — cleaner and unrepresentative; production shapes often reverse the ranking.
- • "The mean improved" — check p95 and p99, since mean-improving changes can worsen the tail.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Profile share of the operation in real request time, before designing any benchmark for it.
- • Run-to-run spread on the benchmark host, to know whether the observed difference clears the noise.
- • Input distribution of the benchmark against production inputs — payload sizes, cache hit ratio, branch mix.
- • Whether baseline and candidate were measured in the same session on the same host, warmed, interleaved.
- • Profile first and compute the share; refuse to benchmark anything whose ceiling is below the effort it would take.
- • Sample benchmark inputs from production traffic so the input distribution stops being a hidden variable.
- • Measure baseline and candidate in the same session, on the same host, warmed, interleaved.
- • Report spread and iteration count with every result; treat differences inside the noise floor as no result.
- • When a benchmark arrives without its setup, ask the six questions before discussing the conclusion.
- • Reproduce the corrected benchmark in a fresh session and confirm the magnitude holds.
- • Confirm the predicted end-to-end effect appears in an integration measurement, since that is what the profile share predicted.
- • Check tail percentiles as well as median, to catch changes that trade tail latency for average throughput.
- • Demanding full setup metadata slows down informal experimentation, which has real value early in an investigation.
- • Sampling production inputs adds a data pipeline and, sometimes, a privacy review.
- • Insisting on profile-share justification can discourage exploratory optimization that occasionally finds something unexpected.
- • Require setup metadata — data source, host, runs, spread, warm-up, ordering — alongside any benchmark result in review.
- • Keep profile shares for the top routes current, so target selection starts from evidence rather than assumption.
- • Alert on CI benchmark thresholds derived from the measured noise floor, and re-derive them when the runner changes.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe profile shares, timings and percentage improvements are teaching figures constructed to make the arithmetic legible. Real profiles are flatter and messier than this example.
- ENVIRONMENT-SPECIFICHow badly each fallacy distorts results depends on the hardware, runtime and workload. On a JIT runtime, missing warm-up alone can invert a ranking; on an AOT-compiled one it matters far less.