Beating the Baseline
A comparison is only a comparison on the same split, the same metric, the same threshold policy, with an interval — and the margin has to be read against what the winner costs to serve. A small gain that costs a GPU and eighty milliseconds is a loss.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The candidate beats the baseline. On what split, by which metric, at which threshold, with what uncertainty — and after the serving cost, is it still a win?
A search team has a neural re-ranker that beats their gradient-boosted ranker in an offline report. The re-ranker needs a GPU and adds tens of milliseconds to every search. The baseline runs on the existing CPU fleet. The director asks the only question that matters: "is the improvement real, and is it worth what it costs?"
The candidate's offline ranking metric is higher than the baseline's reported number. Higher is better. Ship it, and handle the latency and GPU cost as an infrastructure question afterwards.
The two numbers are from different holdouts. Re-scored on the same month, the gap halves; part of the "improvement" was the month.
- The two numbers are from different holdouts. Re-scored on the same month, the gap halves; part of the "improvement" was the month.
- The candidate was evaluated at its best cut-off and the baseline at the production cut-off. Under the same threshold policy — the same number of results shown — the gap narrows again.
- Resampling the holdout by query gives the gap an interval that includes zero. The candidate may be better; the evidence does not show it.
- The candidate is deployed anyway. The GPU fleet and the extra latency cost more per month than the measured lift in conversion is worth; the tail-latency increase loses more users than the ranking gains (Tail Latency: Why p50 Being Fine Does Not Help in the performance domain).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Rank the candidate results for a search query so that the results users click and convert on are at the top. The label is the click or conversion on a result, logged from production.
- The decision is which ranker serves production traffic, under a latency budget the product has already committed to (Designing Search Ranking).
- One example is one query with its candidate list and the observed engagement on each candidate; the training data is last quarter's logs.
- The boosted baseline was evaluated months ago on a different quarter's holdout; the neural candidate on last month's. The offline report compares the two numbers.
- Engagement labels are biased toward what the previous ranker showed at the top, which both models inherit (Feedback Loops).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A valid comparison fixes everything but the model: the same holdout rows, the same metric definition, the same threshold or cut-off policy, the same preprocessing, evaluated by the same code. Any difference in those is a difference in the measurement, and it shows up as a difference in the models.
- The gap between two models on a finite holdout is a random variable. Its interval comes from resampling the holdout — by query, by user, by whatever the independent unit is — and a gap whose interval includes zero has not been demonstrated (Metric Uncertainty). Paired evaluation, scoring both models on the same rows and looking at the per-row differences, tightens the interval considerably.
- The decision compares the demonstrated gain, in business units, against the incremental cost of the winner: hardware, latency and its effect on users, engineering and operations. A gain smaller than that cost is a loss with a better metric (Inference Cost).
Same split, same metric, same threshold policy
Two numbers from two reports are not a comparison. The baseline was scored on a quarter with an easier query mix; the candidate on last month at its own best cut-off. Each difference between the two measurements adds to the apparent gap, in a direction that usually favours whichever model was evaluated more recently, by the team proposing it.
The fix is mechanical: one harness that loads both models, scores them on the same rows, applies the same cut-off, and computes the same metric with the same code. The baseline is re-scored, not quoted. Whatever gap survives that is a gap between the models.
- 1Shared holdout
The same rows for both models, from the same period, with the same label construction.
fails by Baseline number quoted from an old report on a different quarter.
- 2Shared metric and cut-off
One metric definition and the production cut-off policy applied to both.
fails by Candidate evaluated at its best cut-off, baseline at the production one.
- 3Paired gap with interval
Per-query differences, resampled by query, interval on the mean gap.
fails by Two point estimates subtracted; the interval never computed.
- 4Business units and cost
Gap converted to conversions per thousand searches; incremental serving cost and latency effect on the same page.
fails by Cost handled "by infrastructure" after the decision.
- 5Online confirmation
Shadow for latency and errors, then split test for the business metric.
fails by Split test on an over-provisioned GPU pool that does not reflect full traffic.
The gap has an interval
A holdout is a sample. The gap between two models on it is an estimate of the gap on the population, and the estimate has a spread. Resampling the holdout by the independent unit — queries here, users for a churn model, days for a forecast — and recomputing the gap each time gives that spread. A gap whose interval includes zero has not been shown; it may be there, and the evidence is silent.
Paired evaluation helps. Scoring both models on the same rows and resampling the per-row differences removes the variation that is common to both — the hard queries are hard for both — and the interval on the difference is tighter than the intervals on the two absolute numbers would suggest. This is the offline version of what Bias and Variance teaches: the number moves with the sample, and the movement is the thing to measure before the number.
1import numpy as np2 3def paired_gap(metric_per_query_a, metric_per_query_b, rng, resamples=2000):4 # both arrays computed by the same harness on the same queries at the same cut-off5 diff = metric_per_query_b - metric_per_query_a # candidate minus baseline, per query6 n = len(diff)7 gaps = np.array([diff[rng.integers(0, n, n)].mean() for _ in range(resamples)])8 lo, hi = np.percentile(gaps, [2.5, 97.5])9 return diff.mean(), lo, hi10 11mean_gap, lo, hi = paired_gap(ndcg_baseline, ndcg_candidate, np.random.default_rng(0))12# report all three. if lo <= 0 <= hi, the improvement has not been demonstrated.13# then: gap_in_conversions_per_1k = mean_gap * exchange_rate14# incremental_cost_per_1k = gpu_cost_per_1k + latency_effect_per_1kThe resampling unit is the query, because queries are independent and their results are not. Resample by result row and the interval is far too narrow.
A small gain that costs a GPU is a loss
Suppose the paired gap survives: the candidate is better, and the interval excludes zero. The decision is still not made. The candidate needs a GPU per shard and adds tens of milliseconds to every search, and each of those is a cost in the same units as the gain once the exchange rate is agreed: conversions per thousand searches against dollars per thousand searches and users lost per millisecond of tail latency.
That arithmetic is the promotion decision. When the gain covers the cost with margin, ship it — through shadow, then a split test, because the offline cost estimate is also an estimate. When it does not, the baseline stays, and the finding is recorded so the next candidate knows the bar. The model that wins the metric and loses the arithmetic has lost.
Offline report: candidate's ranking metric above the baseline's on the report's holdout. Re-scored on one shared holdout at the production cut-off, paired and resampled: a small positive gap with an interval that just excludes zero.
Split test at full traffic: a conversion lift consistent with the offline gap on desktop; on mobile, tail latency under load is worse than in shadow and the lift is negative. Net, the candidate's incremental GPU cost exceeds the value of the lift.
- 1The serving cost was measured on a warm, under-loaded GPU pool in shadow; at full traffic the tail latency grew, and mobile users are the most latency-sensitive.
- 2The offline holdout under-represented mobile queries, so the slice where the candidate loses was a small part of the aggregate gap.
- 3The engagement labels favour what the incumbent ranker showed, which biases the offline comparison toward the incumbent — meaning the candidate's true offline gap may be larger than measured, which cuts the other way and does not rescue the cost arithmetic.
The candidate's demonstrated gain over the baseline, in business units, exceeds its incremental serving and latency cost, and continues to as traffic, hardware and the baseline change.
holds when The gain was measured online at full traffic, sliced; the cost was measured under production load; both are re-measured when the baseline is retrained or the traffic mix shifts.
breaks when The baseline improves on its own retrain; traffic grows and the GPU cost scales with it while the gain does not; a latency-sensitive segment grows.
respond When the margin closes, demote to the baseline — it was kept in the pipeline for exactly this — and record the bar the next candidate has to clear.
How to build it
Most important first.
- One evaluation harness, both models, one holdout, one metric, one cut-off policy; the baseline re-scored fresh, never quoted from an old report.
- Paired comparison with a resampled interval on the gap, by the independent unit; report the interval, not the point.
- Convert the gap to business units — conversions per thousand searches — and put the incremental serving cost and the latency effect next to it. That table, not the offline metric, is the promotion input (Promotion Is a Checklist, Not a Score).
- When the offline gap is small and positive, run the online test before committing to the infrastructure: shadow to check latency and errors, then a split test on live traffic with the business metric (A/B Testing Models).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The paired gap on the decision metric at the production cut-off, with its resampled interval, on the same holdout. That is "is it real".
- Conversions per thousand searches in the online test, against the incremental cost per thousand searches of the candidate's serving. That is "is it worth it".
- Do not measure two numbers from two reports and subtract them. They do not share a holdout, a cut-off or a month.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The holdout on which the gap was demonstrated resembles production traffic, including the query mix and the label bias, closely enough that the gap transfers.
- The serving cost and latency measured for the candidate are the ones production will see at full traffic, including tail behaviour under load.
- The gain, in business units, continues to exceed the incremental cost as traffic, hardware prices and the baseline itself change.
- Offline: both models scored by one harness on one holdout; paired, resampled gap with interval at the production cut-off; sliced by query type and language.
- Online: shadow deployment for latency and error rate at full traffic (Shadow Deployment), then a split test for the business metric with the same interval discipline.
- Over time: the baseline retrained and re-compared at each cycle; the cost side re-measured as hardware and traffic change; the margin tracked as a number in business units, not a metric.
What can go wrong
- The online test is run with the candidate on a warm GPU pool sized for the test; at full traffic the pool is under-provisioned, the tail latency is worse than tested, and the conversion lift disappears under timeouts.
- The gap is real and worth it on average and negative on a slice — a query language the neural model was undertrained on — that the aggregate hides (Evaluation Slices).
- The comparison is honest at promotion and never repeated; a year later the baseline has been retrained and improved, and the expensive candidate's margin over it is gone.
- The honest comparison is slower: a shared harness, resampled intervals, and an online test add weeks between "the candidate looks better" and "the candidate ships".
- Converting metrics to business units requires an agreed exchange rate — what a conversion is worth, what a millisecond costs — and the numbers are contested.
- Insisting on the cost side means that a genuinely better model is sometimes not shipped, which is correct and unpopular.
- "If offline AUC improved, ship it." Improved on what split, at which threshold, with what interval, and at what serving cost? Each of those has reversed a decision in practice.
- "The gap is small but positive, so the candidate is better." A positive point estimate with an interval that includes zero is not evidence of a difference. Run the paired evaluation before concluding anything.
- "Latency and cost are infrastructure questions, separate from model quality." They are the other side of the same decision. A model that wins the metric and loses users to latency has lost.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a comparison requires the same split, metric, threshold and an interval on the difference is true of any two predictors on any task; what changes across tasks is the independent unit for resampling and the business exchange rate for the metric.
- SCALE-SPECIFICAt low traffic the incremental cost of a heavier model is small in absolute terms and the latency effect may be unnoticed, so a small demonstrated gain can be worth shipping; at search-scale volume a GPU per shard and tens of milliseconds per query are large costs and the same gain is a loss.
- SIMULATEDThe shape of the argument here — a gap that halves on a shared holdout, narrows again under one cut-off, and loses significance under resampling — is illustrative and mirrors the bias-variance lab; the lab's curves come from a fitted polynomial on synthetic data, not from any search system.
- CONTESTEDA serious position holds that offline intervals are so wide on ranking problems, and offline labels so biased toward the incumbent, that the offline comparison should be skipped in favour of going straight to an online test with strict guardrails. That is right where online testing is cheap and safe; the reply is that an online test of a candidate that lost a fair offline comparison wastes the traffic and the GPU pool, and that the offline harness is what catches the cut-off and split mismatches before they reach users.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a split test read with an interval is a controlled experiment, and the discipline of pre-registering the metric, the cut-off and the stopping rule before looking at the data is an experimental-design question this domain applies rather than teaches.