Never Tune on the Test Set
The test set is touched once. Every look costs information; hyperparameter search, feature selection, early stopping and model selection all happen on validation. A team that picks the best of forty runs on the test set has shipped noise with a certificate.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Forty training runs, the best one chosen by its test-set score, and production well below that score. What did the test set stop being, and where should each of those forty decisions have been made?
A team building a document classifier ran a hyperparameter sweep of forty configurations, scored each on the test set, and shipped the best. The reported test score was the highest the project had seen. Production performance is well below it, below several of the runs that were not chosen, and nobody can say which of the forty was actually best.
The test set is the most representative data we have, so the best model on it is the best model. Score every configuration on it, pick the top, report the number. It is the same procedure as validation with a better dataset.
Forty scores on the same set are forty noisy estimates, and the maximum of forty noisy estimates is biased upward by an amount that grows with forty. The chosen run is partly the best configuration and partly the luckiest, and the reported score includes the luck.
- Forty scores on the same set are forty noisy estimates, and the maximum of forty noisy estimates is biased upward by an amount that grows with forty. The chosen run is partly the best configuration and partly the luckiest, and the reported score includes the luck.
- The test set has now been used to make a decision, which means the shipped model was selected for its performance on those specific documents. It has not been trained on them, but it has been chosen by them, and its score on them is no longer an estimate of anything except itself.
- There is no number left to report. Every set the team has was used for a decision; the honest estimate of production performance would need data no decision has touched, and the project has none (Evaluation Leakage).
- Production is a fresh sample, and it returns the configuration's true performance minus nothing — which is below the reported score by roughly the selection bias, and below some unchosen runs whose test-set luck ran the other way.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Classify incoming documents into a few dozen categories for routing. The label is the category a human assigned on receipt; the training set is last year's documents with their labels.
- The decision is which configuration to ship, and the reported number is supposed to be an honest estimate of what it will do on documents it has not seen.
- One example is one document with its assigned category. The test set is a few thousand documents held out at the start of the project, meant to be used once to report the shipped model's performance.
- The validation set exists too, and was used for the first few sweeps. When the test scores started looking better, the sweep was pointed at the test set instead, because it was larger and the numbers were "more reliable".
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Held-out data gives an unbiased estimate of performance exactly once: the first time a model is scored on it, before that score influences any decision. Use the score to choose — a hyperparameter, a feature set, a stopping epoch, a model — and the chosen thing is now correlated with the set's particular contents. The next score on the same set is optimistic by the amount of that correlation, and every further decision adds to it.
- This is why there are three sets. Training fits the parameters; validation adjudicates every decision that is not a parameter — hyperparameters, feature selection, early stopping, architecture, model family, the operating threshold; the test set is read once, at the end, to report what the chosen system will do on new data. Validation is expected to become optimistic; that is its job, and the test set exists to say by how much.
- Selecting the maximum of many runs makes the bias concrete. If each run's score is the true value plus noise, the best-scoring run has the largest positive noise among the candidates, and the expected size of that noise rises with the number of runs. Forty runs on a few thousand documents is enough for the winner's margin to be mostly noise.
What forty looks cost
Each of the forty runs has a true performance and a test-set score that is the true performance plus noise from which documents happened to be held out. Pick the highest score and you have picked, in part, the highest noise. The winner's reported score is its true performance plus the largest of forty noise draws, and that largest draw is not small.
The unchosen runs are informative in the other direction. Several had lower test scores and better production performance than the winner, because their noise draws went the other way. The sweep found the configuration the test set liked best, which is a different thing from the configuration that generalises best, and the difference is what production reported.
looks like A clean sweep: no test rows in training, no test labels in any feature, a held-out set scored the way it is supposed to be.
why it leaks The chosen configuration is the one whose noise on those specific test documents was most favourable. Information about which documents are in the test set flows into the choice of model, so the model is fitted to the test set by selection rather than by gradient.
fix Run the sweep on validation, choose there, and read the test set once for the winner. If the test set has already adjudicated, draw a fresh one from data no decision has seen.
Where each decision belongs
Every choice in a modelling project is a selection, and every selection needs data it can be optimistic on. Training data takes the parameters. Validation takes everything else — hyperparameters, feature selection, the early-stopping epoch, the model family, the threshold. The test set takes nothing; it reports.
The list is longer than most teams think, and early stopping is the one most often missed: monitoring the test loss to pick the epoch is a selection over hundreds of candidates, run every epoch, on the set that was supposed to be read once.
| Decision | Made on | Candidates | What goes wrong on the test set |
|---|---|---|---|
| Model parameters | Training | continuous | Memorisation; the classic overfit |
| Hyperparameters | Validation | tens to hundreds | Winner is the luckiest configuration; reported score inflated by the field size |
| Feature selection | Validation | many subsets | Features chosen for test-set noise; production sees the noise gone |
| Early-stopping epoch | Validation | one per epoch | The epoch where test noise peaked; hundreds of selections, silently |
| Model family | Validation | a few | Small field, small bias — still a selection |
| Operating threshold | Validation | continuous | Threshold tuned to the test set's class balance and score noise |
| Final reported score | Test, once | one | Nothing — this is the read the set exists for |
The set that nothing has touched
The value of the test set is a property of its history, not its contents: it estimates production because no decision has been conditioned on it. The moment one is, that property is gone, and no amount of care afterwards restores it. The only remedy is a set with a clean history — freshly held out from data that arrived after the decisions were made.
So the rule is procedural. Freeze it, log access, score it once, and treat any read that changed a choice as having spent it. A team that cannot produce the log cannot produce the number.
The winning configuration's test-set score, the highest the project recorded, presented as the expected production performance.
First month on new documents: well below the reported score, and below several configurations the sweep rejected; the ranking of the forty runs on production does not match their test-set ranking.
- 1The reported score was the maximum of forty noisy estimates on the same few thousand documents; the winner's margin was mostly its noise draw.
- 2Early stopping inside each run had also monitored the test set, adding hundreds of selections per run to the forty.
- 3The test set was drawn from last year alongside training; even a clean read would have estimated last year's documents, not this year's.
How to build it
Most important first.
- Freeze the test set at the start and log every access. It is scored once, for the final report, by a process that cannot feed back into any choice (Train / Validation / Test).
- Do every selection on validation: the sweep, the feature choice, the early-stopping epoch, the model family, the threshold. When validation is too small to make those decisions reliably, use cross-validation on the training portion and keep the test set out of it (Cross-Validation).
- When the test set has been used to choose, say so, discard it as an estimate, and hold out a fresh set from data that arrived after the decisions were made. The only honest number is one no decision has seen.
- Report the validation score of the winner with its interval, and the count of candidates it beat, so the reader can judge the selection optimism themselves (Metric Uncertainty).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The test-set score of the shipped configuration, read once, after every decision has been made on validation. This is the only number that estimates production.
- Validation score of the winner alongside the number of candidates and the spread across them — the size of the field is the size of the optimism.
- Do not report the best of many test-set scores as the expected performance. It is the expected performance plus the expected maximum of the noise.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The test set has adjudicated no decision — not a hyperparameter, not a feature, not an epoch, not a threshold, not which of several models to ship — and there is a log that would show it if it had.
- The test set is a sample from the population the model will serve, drawn independently of every set that fed a decision, and from a period that resembles deployment.
- The validation set's optimism is understood and bounded: the number of decisions made on it is known, and its score is reported as a selection result, not an estimate.
- Offline: for the forty runs, compute the validation score and the test score; if the test set was used for selection, the chosen run's test margin over the validation-best run is the selection bias, visible in the table.
- Online: compare the first month of production against the reported test score. A shortfall roughly the size of the spread across the sweep is the signature of a tuned test set.
- Over time: audit access to the test set; every read that fed a decision is a debit, and the balance decides whether the number it produces still means anything.
What can go wrong
- The rule is followed for the sweep and broken for early stopping: the training loop monitors the test set to pick the epoch, which is a selection on it with hundreds of candidates (Early Stopping).
- The test set is untouched but was drawn from the same time period as training, so it estimates last year rather than production; it was clean and still wrong about the future (Time-Series Validation).
- A fresh test set is drawn after the leak is discovered, from data the team has already looked at while debugging, and the "fresh" set has been seen by the people if not by the model.
- A test set read once is a few thousand labelled documents that contribute nothing to training or tuning — an expensive asset used for one number, which is why teams reach for it.
- Making every decision on validation means validation is smaller and noisier than the team would like, and the sweep's winner is chosen with less precision. That is the honest precision.
- Discarding a compromised test set and drawing a fresh one costs labelling time and delays the release; the alternative is a release with a number that is not true.
- "Use the test set to tune — it is bigger, so the numbers are more reliable." More reliable estimates of the wrong thing. The size of the set does not change what selecting on it does to the score; it only makes the optimism smaller per decision and the loss of the final estimate just as total.
- "We did not train on the test set, so the score is honest." Training is one way for information to flow from a dataset into a model. Selection is another. A model chosen by its test score has been fitted to the test set by the person choosing.
- "We only looked at it a few times." Each look that changed a decision is a selection. A few looks with ten candidates each is a few dozen selections, and the bias compounds; the honest response is to count them, not to minimise them.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a held-out estimate becomes optimistic the moment it influences a decision is a property of estimation, not of any task, dataset or model family; the rule has no exceptions that depend on context.
- SIMPLIFIEDThe forty-run story is stylised for the shape of the argument; the selection bias in a real sweep depends on the number of candidates, the size of the test set and the correlation between the candidates' errors, and the lesson does not compute it.
- CONTESTEDA serious position holds that in fast-moving applied work the three-way split is a luxury: labelled data is scarce, a separate test set that is read once wastes the most reliable data the team has, and the online experiment is the real test anyway — so a team should use cross-validation for everything and treat the first weeks of production as the test set. The reply is that this is a coherent choice only if the team actually reports no offline number as an estimate of production, and in practice the cross-validated number gets reported as one.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Research methodology — the reuse of public benchmark test sets across thousands of papers, and the adaptive-data-analysis results on how much information repeated queries extract from a holdout, are the formal version of the argument this lesson makes for one team.