Evaluation Leakage
The data is clean and the pipeline is ordered correctly. The leak is the engineer: tuning on the test set, peeking repeatedly, picking the best of many runs on one holdout.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Every feature, split and preprocessing step is correct. How does the test number still become an overestimate, and why is the cause a process rather than a column?
A search team has iterated on their ranking model for a year. Every candidate is scored on the same held-out query set, and the best candidate each quarter ships. The offline number has climbed steadily. Online A/B tests show the last three "improvements" did nothing measurable, and the most recent one was slightly negative.
Keep the test set fixed so results are comparable. Evaluate every candidate on it, choose the best, ship it. Comparing on the same data is the fair way to compare models.
Selecting the best of hundreds of candidates on one holdout selects, in part, for the candidate that happens to fit that holdout's noise. The winner's number is an overestimate — the maximum of many noisy measurements is biased upward — and the bias grows with the number of candidates (Metric Uncertainty).
- Selecting the best of hundreds of candidates on one holdout selects, in part, for the candidate that happens to fit that holdout's noise. The winner's number is an overestimate — the maximum of many noisy measurements is biased upward — and the bias grows with the number of candidates (Metric Uncertainty).
- Every look at the holdout informs the next candidate: features are added because they helped on it, thresholds are chosen because they scored well on it. Over a year the model has been shaped by the holdout as surely as if it had trained on it, and the holdout can no longer estimate anything about unseen queries.
- Online, the model meets queries that were not in the holdout, the fitted noise is absent, and the improvement evaporates. The offline number keeps climbing because it is measuring the team's familiarity with the holdout.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Rank results for a query so that the ones users click and stay on come first. The offline label is a historical click-and-dwell judgment on a fixed set of queries.
- The decision — ship or not — is made on the offline holdout number, and the holdout has been the same set of queries for a year.
- One example is a query with its candidate results and their historical judgments. The holdout is a few thousand queries, fixed when the project started and never changed so that numbers are comparable across quarters.
- Hundreds of model variants, feature sets and hyperparameter settings have been scored on it. Each quarter the best of those ships.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A holdout estimates the true metric with noise. A single evaluation of a single model gives an unbiased estimate. The maximum over many models evaluated on the same holdout is not unbiased: it picks out whichever candidate's noise was most favourable, and its number overstates the winner's true quality. This is the selection effect, and it does not require any intent to cheat.
- Repeated peeking is adaptive selection: each decision — which feature to keep, which threshold to use, whether to ship — is a small fit to the holdout. Individually negligible, collectively a model of the holdout's idiosyncrasies. The test set has become a validation set, and there is no test set left (Never Tune on the Test Set).
- The cleanest form is direct: tuning hyperparameters on the test set. The commonest form is indirect: the test number is glanced at during development, and development bends toward it. Both leak information from the test labels into the model through the engineer.
The winner's number is an overestimate
Suppose a hundred candidates are all in truth equally good. Each is scored on the holdout with some noise. The best score among a hundred draws is well above the true value, and the candidate that achieved it is no better than the others. Ship it, and it performs like the others. The offline process has selected for lucky noise.
Real candidates differ, so the selection is partly for genuine quality and partly for noise. The share that is noise grows with the number of candidates and shrinks with the holdout size. A year of daily experiments on a few thousand queries is deep in the noise-dominated regime, and the steady offline climb is the accumulation of favourable draws.
The holdout metric has risen steadily each quarter; every shipped candidate was the best of many on the same query set.
The last three A/B tests show no measurable change and the latest a slight regression in click-through and dwell.
- 1Selecting the best of hundreds of candidates on one fixed holdout picks the one that fits the holdout's noise; its number overstates its quality and the overstatement compounds across quarters.
- 2Features, thresholds and architecture choices were made by looking at the holdout, so the model has been shaped by the test labels through the development process.
- 3The reported gains were mostly inside the holdout's confidence interval, which was never computed; the online test detects differences the offline process could not resolve.
Peeking is training
The direct form — searching hyperparameters on the test set — is easy to forbid. The indirect form is the one that happens: the test number is visible on the dashboard, a feature is kept because it helped there, a threshold is moved because it looked better there. Each decision is a tiny gradient step on the test labels, taken by a person rather than an optimiser.
The defence is structural. The test set should be scored by a process the developer does not run casually, once per candidate that is a serious ship contender, with the count of evaluations recorded. If the count is large, the set is spent.
looks like A dashboard panel showing validation and test numbers side by side for every experiment run, which is convenient and feels transparent.
why it leaks Every development decision made while watching the test number is a fit to the test labels. The information travels from the labels, through the metric, through the engineer's choices, into the model — without any row of the test set ever entering training.
fix Hide the test metric during development; score it once per serious candidate through a recorded process; refresh the set when the count grows; use the validation set for every other decision.
1class TestSet:2 def __init__(self, X, y, budget=5):3 self._X, self._y, self.budget, self.uses = X, y, budget, []4 5 def score(self, model, reason):6 # every evaluation is recorded with why it was needed7 if len(self.uses) >= self.budget:8 raise RuntimeError("test set spent; draw a fresh holdout")9 m = metric(model.predict(self._X), self._y)10 self.uses.append((reason, m))11 return m12 13# validation is free; the test set is scored once per real ship candidate14val_score = metric(candidate.predict(X_val), y_val)15if val_score > champion_val_score + ci_half_width:16 test_score = test_set.score(candidate, reason="ship candidate Q3")The budget is arbitrary; the point is that it exists and is recorded. A test set whose use count nobody knows has no known meaning.
Read the number with its interval
A holdout of a few thousand queries carries a confidence interval that is not small. Most quarter-over-quarter offline gains in the year were inside it, which means the offline process could not distinguish the candidates it was choosing between. The online tests could, and did, and found nothing.
Reporting the interval with the number is the minimum. Correcting the selection — evaluating the chosen candidate on a holdout it was not selected on — is the fix. And checking that offline gains have predicted online gains recently is the assumption the whole process rests on.
A gain on the offline holdout, larger than its confidence interval, predicts a gain in the online metric.
holds when The holdout is fresh relative to development decisions, the number of candidates compared on it is small, and recent offline gains have been confirmed online.
breaks when The holdout has been fixed through many cycles of selection; gains are reported without intervals; the query mix has shifted since the holdout was drawn; the offline label no longer matches the online behaviour.
respond Retire the holdout, draw a fresh one from recent traffic, re-score the current champion and the candidate on it, and require gains outside the interval before an online test.
How to build it
Most important first.
- Separate the jobs: a validation set for every choice made during development, and a test set that is scored once, at the end, for the final number (Train / Validation / Test). If the test set has been looked at more than a handful of times, it is a validation set; get a new one.
- Refresh the holdout on a schedule, and when a refreshed holdout disagrees with the old one about the ranking of candidates, believe the new one — the old one has been fitted.
- Report uncertainty with every number, and treat a difference smaller than the confidence interval as no difference. Most of the "improvements" that did nothing online were inside the interval offline.
- Correct for the number of comparisons when selecting among many candidates, or evaluate the chosen candidate on a fresh holdout it was not selected on before believing its number.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The metric on a fresh holdout of queries never used for any decision, for the final candidate only. This is the number that resembles the online result.
- The online A/B result, which is the actual decision criterion and the number the offline process is supposed to predict (Offline vs Online Evaluation).
- The number on the year-old holdout, however high, measures fit to the holdout and is not an estimate of anything.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The number reported for the shipped model was computed on data that no development decision was made against, and the number of candidates compared on that data is known and small.
- Offline improvements that are reported as real are larger than the holdout's confidence interval, with the interval computed for the actual holdout size and the number of comparisons made.
- The relationship between the offline metric and the online outcome has been checked recently enough that an offline gain still predicts an online gain.
- Offline: score the shipped model on a holdout of queries drawn after the last development decision. Compare to the reported number; a large drop is the selection effect.
- Offline: compute the confidence interval on the holdout by bootstrap over queries, and check how many of the year's shipped "improvements" exceeded it.
- Online: correlate offline gains with A/B outcomes over the year. If the correlation has weakened, the offline holdout has stopped estimating the online metric.
What can go wrong
- The test set is refreshed, but it is drawn from the same query log the old one came from, and the team's intuitions — trained on the old set — transfer, so the fresh number is only slightly more honest.
- The offline gain is inside the confidence interval, the team ships anyway because "it cannot hurt", and the online test detects a small regression that is attributed to noise.
- A fresh holdout each quarter is drawn from recent queries and the metric drops, which is read as model decay rather than as the disappearance of the fitted noise.
- Refreshing the holdout breaks comparability across quarters, which is the very thing the fixed holdout was providing.
- A strictly once-only test set means most of the labelled data is spent on evaluation it cannot be reused for, and labelling is expensive.
- Reporting intervals and demanding gains outside them slows shipping and makes many small real improvements unshippable on offline evidence alone, pushing the decision to online tests, which are slower and costlier.
- "We never trained on the test set, so it cannot be leaked." Selection and adaptive development leak label information into the model through the engineer, without a single row of the test set entering a training loop.
- "The number went up every quarter, so the model kept improving." The number went up because the team kept selecting the best fit to a fixed holdout. Only a fresh holdout or an online test can say whether the model improved.
- "A fixed test set is the only fair way to compare models." It is fair for the first comparison. Every subsequent one is biased in favour of whatever fits the set's noise, and after a year the bias is the whole trend.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe maximum of many noisy estimates is biased upward under any metric, model family or task; adaptive development fits any holdout it is repeatedly measured against.
- SCALE-SPECIFICThe selection bias is small when the holdout is large relative to the number of candidates and the metric's variance, and large when a few thousand examples are used to choose among hundreds of variants; a team running many experiments on a small holdout is the worst case.
- CONTESTEDThe strongest case for a fixed holdout is that it is the only way to compare this quarter's model with last quarter's on equal terms, and that a refreshed set confounds model change with data change; benchmark suites in research are fixed for exactly this reason. That is a real cost. The reply is that comparability against a fitted holdout is comparability of overfits, and that the online test is the comparison that matters.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a test set with a recorded evaluation budget is a test fixture with a lifecycle, and the discipline of retiring and refreshing fixtures is a testing practice this domain borrows rather than defines.