Train / Validation / Test
Three sets with three jobs: learn parameters, choose between models, and estimate final performance once. The percentages are a consequence of the jobs, not a rule.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Which decisions is each set allowed to inform, and how big does each need to be for the number it produces to mean anything?
A marketing team wants a model to pick which of five discount offers to show each customer. The data science team has tried a dozen models and reported the best test score, and the finance team asks how much to trust it before budgeting the campaign.
Split 80/10/10, because that is what everyone does. Train on the first, tune on the second, report the third. The test score is the number the model will achieve in production.
The test set was consulted for the top three candidates, so it was used to choose. The reported score is the maximum of three noisy estimates and is optimistic by construction; the campaign under-delivers against the budget (Never Tune on the Test Set).
- The test set was consulted for the top three candidates, so it was used to choose. The reported score is the maximum of three noisy estimates and is optimistic by construction; the campaign under-delivers against the budget (Never Tune on the Test Set).
- Twenty thousand validation rows sound like plenty, but redemptions are a few percent, so the validation set holds a few hundred positives. The difference between the best and fifth-best model is inside the noise, and the "winner" was chosen by luck (Metric Uncertainty).
- The split was random over three years of campaigns, so validation contains offers from the same campaigns as training; the model learned campaign-specific quirks that the next campaign will not have (Time-Based Split).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the probability a customer redeems an offer, per offer type, from past behaviour. The label is redemption within 14 days of the offer.
- The decision downstream is a budget commitment based on the predicted redemption rate, so the final number has to be an honest estimate, not the best of twelve attempts.
- One example is one customer-offer pair from past campaigns, with the customer's history up to the offer date and whether they redeemed. Around two hundred thousand rows over three years.
- The team split it 80/10/10 at random, trained each model on the 80, compared them on the 10, and reported the best model's score on the last 10.
- Twelve models and several hundred hyperparameter settings were compared. The test set was consulted for the top three.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Any set used to make a choice becomes optimistic about that choice. Training data is used to choose parameters, so training error understates generalisation. Validation data is used to choose between models and hyperparameters, so the best validation score understates too — by more the more candidates were compared. The test set is honest only if it informed no decision.
- The size each set needs follows from its job. Training needs enough rows to fit the model — more for higher-capacity models. Validation needs enough positives that the difference between candidates exceeds the metric's standard error, which shrinks with the square root of the positive count. Test needs enough that the single reported number has a confidence interval the business can live with.
- So the percentages are context-dependent. With a million rows and a stable process, a 1% test set is plenty. With five thousand rows and rare positives, no split of a single holdout gives a trustworthy comparison, and cross-validation exists to reuse the data for validation while still leaving a test set alone (Cross-Validation).
Three jobs, three permissions
The three sets differ in what they are allowed to influence, not in their size. The table makes the permissions explicit. Anything in the "must not" column that happens anyway moves that set one column to the left, and its number stops meaning what it claims.
The last column is the part people skip: each set's size follows from its job and from the metric's noise, and the same 10% is generous on one dataset and useless on another.
| Set | Informs | Must not inform | Size follows from |
|---|---|---|---|
| Train | Model parameters; fitted preprocessing | Model choice; hyperparameters; the reported number | Model capacity — enough rows to fit without memorising |
| Validation | Model choice; hyperparameters; threshold; early stopping | The reported number | Positive count needed to separate candidates beyond the metric's standard error |
| Test | The reported number, once | Anything else, including feature decisions | The confidence interval the business needs on the final estimate |
Why the best of twelve is optimistic
Each candidate's validation score is the true score plus noise. Picking the maximum picks, in part, the largest noise. With a few hundred positives in validation, the standard error on a precision-like metric is a few percentage points, and twelve candidates whose true scores differ by less than that are indistinguishable; the winner is whichever got the luckiest draw.
The code estimates the standard error by bootstrap and asks whether the top two candidates are actually separated. When they are not, the honest report is a tie, and the choice should be made on cost, latency or simplicity.
1import random2 3def metric(y_true, y_pred): # any per-row metric, e.g. precision at k4 ...5 6def bootstrap_diff(y, pred_a, pred_b, n=1000, seed=0):7 rnd = random.Random(seed)8 idx = list(range(len(y)))9 diffs = []10 for _ in range(n):11 s = [rnd.choice(idx) for _ in idx] # resample validation rows12 ya = [y[i] for i in s]13 diffs.append(metric(ya, [pred_a[i] for i in s]) - metric(ya, [pred_b[i] for i in s]))14 diffs.sort()15 return diffs[int(0.025 * n)], diffs[int(0.975 * n)]16 17lo, hi = bootstrap_diff(y_val, preds_best, preds_second)18# If the interval straddles zero, the two models are tied on this19# validation set. Choosing "the best" is choosing the luckier one.Resampling rows rather than positives keeps the positive count random, which is what actually happens with a new draw of data. If the interval is wide, the fix is more validation positives or cross-validation, not a bigger model.
What the test number promises
A test score reported to finance is a promise: on data like this, the model will do about this well. The promise depends on the set having informed no decision, and on it resembling the campaign about to run. Both are assumptions, and both can be checked.
The second one is where most test sets fail quietly. A random 10% from three years of history estimates performance on the last three years. The campaign is next month.
The reported test score was computed once on data that informed no modelling decision and that resembles the traffic the model will score, in time period, entity mix and positive rate.
holds when The test set is loaded by a separate pipeline step that runs after the model is frozen; it is cut by the same strategy production implies — usually the most recent period; its score and the look count are recorded in the registry.
breaks when Someone checks "just one more model" against it; a feature is dropped because it hurt the test score; the test set is reused across many model versions; production drifts away from the period it was cut from.
respond Cut a fresh test set from the newest fully-labelled period and treat the old one as a second validation set. Do not report the old number again.
How to build it
Most important first.
- Assign each set one job and enforce it in the pipeline: the test set is loaded by a separate step that runs once, after the model is frozen, and its score is written to the registry with the model.
- Size validation from the positive count needed to separate candidates: estimate the metric's standard error and compare it to the differences you expect to care about; if the differences are inside the error, use cross-validation or accept that the candidates are tied.
- Choose the split *strategy* before the split *ratio*: time, entity and stratification decide whether the validation set resembles production at all (Choosing a Split Strategy).
- Report the test score with its uncertainty and the number of positives it was computed on, so the finance team sees an interval rather than a point.
- When the test set has been looked at, treat it as a second validation set and cut a fresh one from newer data.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The validation metric with its standard error and the positive count, for every candidate — the number that decides which model, and how confidently.
- The test metric, computed once, with the same uncertainty. This is the number that maps to the budget; it is not a better validation score.
- The gap between validation and test for the chosen model. A large gap means validation was over-used and the selection is optimistic.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The test set informed no decision — not model choice, not hyperparameters, not feature selection, not preprocessing — so its score is an unbiased estimate of performance on data like it.
- The validation and test sets resemble the traffic the model will score in production, in time, entity mix and positive rate, or the difference is known and stated.
- The number of candidates compared on the validation set was small enough, relative to its size, that the winner's margin exceeds the metric's noise.
- Offline: rerun the selection with a different random split seed; if a different model wins, the validation set is too small for the comparison and the winner is noise.
- Online: compare the first weeks of production redemption rate against the test estimate and its interval; a result outside the interval means the split did not resemble production.
- Over time: track how many times the test set has been consulted per model lineage, and cut a new one from recent data when the count grows.
What can go wrong
- The test set is separate, but the feature pipeline was tuned by looking at test rows — a normaliser fitted on everything, a feature dropped because it hurt the test score — and the number is quietly contaminated (Preprocessing Leakage, Evaluation Leakage).
- The test set is honest, but it was cut at random from the same period as training, so it estimates performance on the past, and production is the future.
- The team enforces a single test look, and then reuses the same test set for the next six months of model versions; each look is a small tune, and the set decays.
- A larger validation or test set is a smaller training set, and for a small dataset the trade is real: better estimates of a slightly worse model.
- A test set touched once is a number you cannot iterate on; the temptation to look again is the point, and resisting it costs the fast feedback that looking would give.
- Cross-validation gives better use of small data at the cost of training k models, and does not remove the need for a separate test set.
- "80/10/10 is the standard split." There is no standard. The ratio follows from how many rows fitting needs, how many positives the comparison needs, and how tight the final estimate must be, and those differ by orders of magnitude between problems.
- "We only looked at the test set three times." Three looks is three comparisons, and the reported number is the best of three noisy draws. Once means once.
- "The test score is what production will see." Only if the test set resembles production in time and entity mix. A random test set from the past is an estimate of the past.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a set used for a choice becomes optimistic about that choice is a statistical fact independent of task, data or model family.
- DATA-SPECIFICOn millions of rows a small fixed test set is fine and cross-validation is unnecessary; on a few thousand rows with rare positives a single holdout cannot separate candidates and k-fold validation with a small untouched test set is the only workable arrangement.
- CONTESTEDPractitioners disagree about whether a separate test set is worth its cost on small datasets. One side holds that nested cross-validation gives an honest estimate without sacrificing a tenth of the data, and that a small test set produces an interval too wide to be useful anyway. The other holds that a physically separate, once-touched test set is the only discipline that survives contact with a team under deadline pressure, because cross-validation folds get re-used the moment someone wants to check one more thing.
Where the depth lives
This domain teaches the model and hands the rest off by name.