Regression or Tuesday? Telling a Real Change from Noise
p95 moved from 180 ms to 260 ms. Before filing the bug, establish that both numbers answer the same question: same traffic mix, same data, same environment, enough samples. Then compare the difference against the noise you already know your measurement has.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Before you file the bug: is it the same question?
Most apparent regressions are confounds, and they are cheap to rule out. Did the traffic mix shift — a batch job, a crawler, a new client version calling a heavier endpoint? Did the data grow past a threshold where a query plan changed (An Index Scan Is Not Automatically Faster)? Did the CI runner change class? Is the cache cold because the deploy just restarted everything? Is the comparison window a weekday against a weekend?
Each of these produces a genuine latency change with no code regression involved. Working through them takes minutes and prevents the common failure mode: a team bisecting commits for two days to find a regression that was a traffic-mix shift.
The discipline is to make the two measurements answer the same question before comparing them. Same route, same traffic composition, same data scale, same environment, same cache state, comparable time window. If any of those differ, fix that before reasoning about the code.
| Confound | How to check | Tell |
|---|---|---|
| Traffic mix shift | Per-route request share before and after | Overall p95 up while every individual route is flat |
| Data growth crossing a threshold | Row counts and query plans for the top queries | One query's plan changed; the rest are unaffected |
| Environment change | CI runner class, instance type, noisy neighbours | Baseline moved too, when re-run today |
| Cold cache after deploy | Cache hit rate over the minutes after restart | Latency recovers on its own within minutes |
| Comparison window | Day of week, hour of day, campaign activity | The "before" was a quiet period |
| Sample size | Request count in each window | p99 on a few hundred requests moves on its own |
| Client-side change | Client version distribution | A new app version calls a heavier endpoint more often |
Signal against noise, without a statistics lecture
You do not need hypothesis testing to make good decisions here; you need to know your measurement's noise floor. Run the identical benchmark five or ten times without changing anything and record the spread. That number — the run-to-run variation — is the bar any claimed difference has to clear.
A practical rule that works well: treat a difference as real when it is at least two to three times the observed run-to-run spread, and treat anything inside one spread as no result. If your CI benchmark varies ±8% between identical runs, an 8% "regression" is noise and a 40% one is not. The value of writing the noise floor down is that this becomes an arithmetic check rather than an argument.
For production percentiles, the equivalent discipline is comparing distributions rather than single numbers, over windows long enough to contain enough samples. A p99 computed from 300 requests is dominated by three requests; the same p99 over a day is stable. And prefer comparing the same hour on the same weekday, which controls for the daily and weekly shape that otherwise dominates everything.
CI benchmarks people actually keep
Performance tests in CI fail for one of two reasons: they are so noisy that everyone learns to re-run them, or they are so loose that they never catch anything. The fix for both is to derive thresholds from the measured noise floor rather than picking a round number, and to alert on sustained shifts rather than single runs.
Three practices make them survivable. Compare against a rolling baseline of recent runs rather than a fixed number, so gradual environment drift does not slowly turn every build red. Require consecutive breaches, or compare medians of the last N runs, so a single noisy run does not page anyone. And quarantine benchmarks whose noise exceeds their usefulness rather than leaving them to erode trust in the whole suite.
On the production side, the strongest tool is comparing deploy cohorts: run the new version alongside the old on a fraction of traffic and compare percentiles between them over the same window. This controls for traffic mix, data scale, time of day and environment simultaneously — every confound in the first section — because both cohorts experience all of them equally. Where it is available, it settles arguments that no amount of before-and-after dashboard comparison can.
1# Establish the noise floor once per runner class, and re-derive on change2noise_floor = spread_of(run_benchmark() for _ in range(10)) # e.g. +/- 8%3 4# Compare against a rolling baseline, not a hard-coded number5baseline = median(last_20_runs_on_main)6current = median(last_3_runs_on_this_branch) # medians, not single runs7 8delta = (current - baseline) / baseline9 10if delta > 3 * noise_floor:11 fail("regression: {delta:.0%} vs baseline, noise floor {noise_floor:.0%}")12elif delta > 1.5 * noise_floor:13 warn("possible regression: re-run to confirm before merging")14else:15 pass # inside the noise; not a result16 17# Quarantine rule: if a benchmark's own noise floor exceeds 15%,18# it cannot detect anything useful -> fix it or remove it.19# Leaving noisy benchmarks in place is what teaches people to ignore red.Key points
- Rule out confounds first — traffic mix, data growth, environment, cache state, comparison window, sample size.
- Measure your noise floor by running the identical benchmark repeatedly; that spread is the bar every claim must clear.
- Treat differences inside one spread as no result and differences above two to three spreads as real.
- CI thresholds should be derived from measured noise and compared against a rolling baseline, not a fixed number.
- Deploy-cohort comparison controls for every confound at once, because both cohorts experience the same conditions.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Dashboard → claim: aggregate p95 rises from 180 ms to 260 ms after a release, and the release is blamed.
- 2Claim → confound check: per-route p95 is flat everywhere; the aggregate moved because route mix shifted.
- 3Mix → cause: a new client version calls the report endpoint four times more often, and that endpoint was always slow.
- 4Cause → verification: the release did not change the report endpoint; the client did change its calling pattern.
- 5Verification → action: the finding is a client-side traffic change, not a server regression — a different fix and a different owner.
- • "p95 rose 40% after the deploy, so the deploy caused it" — correlation with a deploy is a lead, not a cause (Correlation Is Not the Root Cause).
- • "The benchmark is red, so we have a regression" — check the noise floor first; many CI benchmarks cannot resolve the difference they are asserting on.
- • "Aggregate latency rose, so something got slower" — a mix shift toward heavier routes raises the aggregate with nothing getting slower.
- • "It was fine yesterday" — yesterday may have been a quiet window, a different data scale, or a warmer cache.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Run-to-run spread of the benchmark on the target runner class, from at least ten identical runs.
- • Per-route traffic share and request counts in both comparison windows, to rule out mix and sample-size effects.
- • Query plans and row counts for top queries, to rule out data-growth threshold effects.
- • Percentiles per deploy cohort over the same window, where cohort-based comparison is available.
- • Work the confound checklist before bisecting: mix, data scale, environment, cache state, window, sample size, client versions.
- • Establish and record the noise floor per benchmark and per runner class, and re-derive it when infrastructure changes.
- • Set CI thresholds as multiples of measured noise against a rolling baseline, with a warn band that prompts a re-run.
- • Use deploy-cohort comparison in production where available, since it controls for every confound simultaneously.
- • Quarantine benchmarks whose noise exceeds their usefulness rather than letting them teach the team to ignore failures.
- • Re-run the comparison after controlling the identified confound and confirm the difference persists or disappears.
- • For a confirmed regression, bisect and confirm the difference appears at one commit and clears the noise floor at that commit.
- • After the fix, verify the metric returns to baseline for a sustained window, not just for one run.
- • Confound checks and repeated runs cost time in exactly the moment when a fast answer feels most valuable.
- • Noise-derived thresholds are looser than round numbers, so small genuine regressions can pass unnoticed.
- • Deploy-cohort comparison requires the infrastructure to run two versions simultaneously and enough traffic per cohort.
- • Keep the noise floor recorded next to each benchmark, and fail the build when a benchmark's own noise grows past a usable level.
- • Alert on sustained percentile shifts against the same weekday and hour, rather than on instantaneous comparisons.
- • Track per-route percentiles alongside the aggregate so mix shifts are visible immediately rather than diagnosed later.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe ten-run distribution, spread and threshold values are teaching figures. Your noise floor must be measured on your own benchmark and runner; it is frequently much larger than teams assume.
- ENVIRONMENT-SPECIFICShared CI runners commonly show run-to-run spreads of tens of percent, while dedicated hardware can be within a few percent. The same threshold rule produces very different absolute thresholds on each.