Metric Uncertainty
A validation metric is a sample statistic with an interval around it. Two models compared on the same holdout need a paired comparison; a small test set cannot distinguish small improvements; and every comparison made against one holdout erodes it a little.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The candidate beats the incumbent by a small margin on the holdout. Is that a real improvement, how would you know, and what has the holdout already been used for?
A search team has iterated on its ranking model for a year. Each candidate is compared against the incumbent on the same ten-thousand-query holdout, and each improvement is small. The last several "wins" produced nothing online, and the team lead wants to know whether the holdout can still tell a better model from a lucky one.
Compute the metric for both models on the holdout, and if the candidate's number is higher, it is better. The holdout is large, the metric is standard, and a higher number is a higher number.
The metric is a mean over ten thousand queries, and a mean over a sample has a standard error. The candidate's margin is inside that error. The comparison is reporting noise with the sign that happened to come up.
- The metric is a mean over ten thousand queries, and a mean over a sample has a standard error. The candidate's margin is inside that error. The comparison is reporting noise with the sign that happened to come up.
- The two models were scored on the same queries, and their per-query scores are highly correlated — both get the easy queries right and both struggle on the hard ones. An unpaired comparison ignores that correlation and overstates the uncertainty; a paired one uses it and would have shown the margin was real, or not, with far more power. Nobody ran either.
- Dozens of candidates have been compared against this holdout. Each selection of the best candidate on it moved the shipped model a little toward whatever the holdout happens to reward — its particular hard queries, its particular labelling quirks — and the holdout has become part of the training signal (Evaluation Leakage).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Rank documents for a query so that the ones a user will engage with come first. The label is engagement on documents that were shown, judged per query.
- The decision is whether a candidate is better than the incumbent — a comparison of two models on one dataset, repeated many times.
- One example is one query with its ranked results and engagement labels. The holdout is ten thousand queries drawn once, a year ago, and reused for every comparison since.
- Query difficulty varies enormously. A few hundred hard queries account for most of the metric's variance; which of them a model gets right decides the number more than anything else.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A validation metric is a statistic computed on a sample from the population of future inputs. Its value would differ on a different sample of the same size, and the size of that variation is the standard error. For a mean over n examples the standard error shrinks with the square root of n; distinguishing a small improvement needs a sample size that grows with the inverse square of the improvement.
- The bootstrap estimates the interval without assuming a distribution: resample the examples with replacement, recompute the metric, repeat, and read the spread. For comparing two models on the same examples, resample the *pairs* and compute the difference each time — the paired bootstrap — so that the shared difficulty of each example cancels and the interval is for the difference itself.
- Each time a holdout is used to choose among candidates, the chosen one is the maximum of several noisy estimates, and the maximum is biased upward. Repeat the selection and the shipped model accumulates the holdout's idiosyncrasies. The holdout has not been trained on, but it has been optimised against, and its estimate of production drifts optimistic in proportion to the number of comparisons it has adjudicated.
A metric is a mean with a standard error
The holdout metric is the average of a per-query score over the queries in the holdout. Draw a different ten thousand queries and the average moves. The size of that movement is the standard error, and the candidate's margin over the incumbent has to be compared to it before the sign of the margin means anything.
The bootstrap gives the interval without a formula: resample the holdout with replacement many times, recompute the metric each time, and read off the spread. For two models on the same queries, resample the paired differences instead — the shared difficulty of each query cancels, and the interval for the difference is much narrower than the two separate intervals would suggest.
1import numpy as np2 3def paired_bootstrap(score_a, score_b, n_boot=2000, seed=0):4 """score_a, score_b: per-query metric for each model on the same queries.5 Returns the mean difference and a 95% interval for it."""6 rng = np.random.default_rng(seed)7 diff = score_b - score_a # per-query, paired8 n = len(diff)9 boots = np.empty(n_boot)10 for i in range(n_boot):11 idx = rng.integers(0, n, n) # resample queries, keep pairs12 boots[i] = diff[idx].mean()13 lo, hi = np.percentile(boots, [2.5, 97.5])14 return diff.mean(), (lo, hi)15 16# the decision is whether lo > 0, not whether diff.mean() > 017# if queries share sessions, resample sessions, not queriesThe pairing is the whole trick. Two separate intervals for the two models will overlap even when the paired difference clearly excludes zero, because both models are dragged by the same hard queries.
Small test sets cannot see small improvements
The standard error shrinks with the square root of the holdout size, so seeing an improvement half as large needs a holdout four times as big. There is a smallest improvement a given holdout can distinguish from noise, and asking it about anything smaller returns the sign of the noise.
This sets a floor on what is worth attempting. If the team cannot afford a holdout large enough to detect the improvements it is chasing, the honest conclusion is that those improvements cannot be evaluated offline — and the online experiment, which has its own power calculation, is the only place they could be.
Detectable margin ~ 2 x standard error of the paired difference Standard error ~ sd(per-query difference) / sqrt(n) sd(diff) n (queries) detectable margin (approx.) 0.20 1,000 0.013 0.20 10,000 0.004 0.20 100,000 0.0013 A candidate whose margin is below the row for your holdout has not been evaluated. It has been guessed at with the sign of the noise.
Every comparison spends the holdout
A holdout that has picked the best of fifty candidates has been optimised against fifty times. Each pick preferred whatever that holdout rewards — its particular hard queries, its labelling quirks — and the shipped model is a little more tuned to the holdout than to production with each round. It was never trained on, and it has still been learned.
The remedy is bookkeeping: count the comparisons, refresh the holdout on a schedule or a count, and hold a final test set that has adjudicated nothing and is read once. The count is a metric of the holdout's credibility, and it only goes down.
The holdout is a sample from the population the model serves now, and has adjudicated few enough comparisons that the shipped model's metric on it is not materially optimistic.
holds when The holdout was drawn recently from the current query mix, the number of comparisons against it is logged and small, and a separate test set exists that no selection has touched.
breaks when The query mix shifts; the comparison count climbs past what the interval width can absorb; candidates are tuned until one's interval excludes zero on this specific holdout.
respond Retire the holdout, draw a new one, reset the count, and re-baseline the incumbent on it before comparing anything else.
How to build it
Most important first.
- Report every metric with an interval, from a bootstrap over examples or from a closed form where one exists. A margin inside the interval is not a result (Beating the Baseline).
- Compare two models with a paired procedure on the same examples — a paired bootstrap on the per-query difference, or a paired test — and report the interval for the difference. This is where the power is.
- Size the holdout for the improvement you need to detect. If the smallest improvement worth shipping is tiny, the holdout needed to see it is large, and a holdout that cannot see it should not be asked.
- Ration the holdout. Keep a count of comparisons made against it, refresh it on a schedule, and keep a final test set that has adjudicated nothing (Never Tune on the Test Set).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The paired difference between candidate and incumbent, per query, with its bootstrap interval. The decision is whether the interval excludes zero, not whether the point estimate is positive.
- The number of comparisons the holdout has adjudicated since it was drawn, as a metric of its remaining credibility.
- Do not report the two models' metrics as separate numbers with no interval and read the sign of their difference.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The holdout is a sample from the population the model serves now; a holdout drawn a year ago is a sample from last year's queries, and its interval is about that population.
- The examples resampled by the bootstrap are independent at the level resampled — query, session or user — so the interval is not artificially narrow.
- The number of comparisons made against the holdout is small enough that selection bias in the shipped model's reported metric is within the interval's width; past some count that stops being true and the holdout has to be replaced.
- Offline: bootstrap the paired difference for the last several "winning" candidates against their incumbents. Count how many intervals actually excluded zero. That number, against the number shipped, is the calibration of the team's release rule.
- Online: for candidates whose interval excluded zero, compare the online outcome against the offline margin; a consistent shortfall is the holdout's accumulated selection bias.
- Over time: draw a fresh holdout, re-score the current incumbent on both, and measure the gap; a shipped model that looks better on the old holdout than the new one has learned the old one.
What can go wrong
- The bootstrap resamples queries, but queries from the same session or user are correlated, so the interval is too narrow; the resampling unit must be the independent unit (Group Split).
- The interval is reported and the release rule says "ship if the interval excludes zero", so candidates are tuned until one does — which is the same selection problem, now with an interval attached.
- The holdout is refreshed, and the new one differs in query mix from the old, so the incumbent's baseline number moves and a candidate is credited with an improvement that is the holdout changing.
- Intervals make most small improvements unshippable, which is correct and demoralising; the release cadence slows to the rate at which real improvements occur.
- Sizing the holdout to detect small effects can mean labelling far more queries than the team has budget for, and the honest response is to stop trying to ship effects that small.
- Refreshing the holdout breaks comparability with history; the metric time series gets a discontinuity that has to be explained.
- "The holdout has ten thousand queries, so the metric is precise." Precise enough for what? The standard error of the mean over ten thousand queries can still be larger than the margin between two good models. Precision is relative to the effect.
- "The candidate won on the holdout and lost online, so the holdout is stale." Possibly. Or the win was noise. Check the interval first; a margin inside it was never a win, and no holdout is stale enough to explain that.
- "We should use a bigger holdout." Bigger helps with the interval and does nothing for the selection problem. A holdout that has adjudicated a hundred comparisons is biased however large it is; it needs refreshing, not enlarging.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a metric on a sample has a standard error, that paired comparisons have more power, and that repeated selection against one holdout biases it, are statistical facts independent of task, model or domain.
- DATA-SPECIFICOn a holdout of a few hundred examples the interval is wide enough that only large improvements can be seen and the bootstrap is essential; on tens of millions of independent examples the interval is negligible and the selection problem dominates instead.
- CONTESTEDA serious position holds that formal intervals and paired tests give a false sense of rigour in ML, because the examples are rarely independent, the metric is rarely a simple mean, the bootstrap's assumptions are quietly violated, and the real check is the online experiment anyway — so teams should ship promising candidates to a small online test rather than argue about p-values. The reply is that an online test is far more expensive than a bootstrap, and a candidate whose offline margin is inside its interval has not earned one.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Statistics — multiple comparisons, the optimism of a selected maximum, the bootstrap's conditions and the corrected tests for comparing learners are the statistical literature behind every claim here.