Random Split
Shuffle the rows and cut. Correct when rows are independent and production looks like the training period. Wrong, and optimistic, whenever time or repeated entities are in the data.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Are the rows independent of each other and of time, so that a shuffled cut resembles the data production will send?
A manufacturer wants to predict which parts will fail a quality inspection from the sensor readings taken during machining. Each part is measured once; the process is stable; the team shuffled the parts and held out a fifth.
Shuffle and cut 80/20. Random assignment is unbiased, every row has the same chance of landing in either set, and the two sets are drawn from the same distribution — which is exactly what an evaluation wants.
For the machining data it does not break: the rows are independent, exchangeable and from a stationary process, and a random split is the right call. The shape of the failure is in the sibling team's copy.
- For the machining data it does not break: the rows are independent, exchangeable and from a stationary process, and a random split is the right call. The shape of the failure is in the sibling team's copy.
- On the funnel data the same user appears in hundreds of sessions, and a shuffle puts most users in both sets. The model recognises users rather than predicting behaviour; validation is excellent and the first new cohort of users scores badly (Entity Leakage).
- Behaviour on the funnel changes month to month. A shuffled validation set contains rows from every month, including months after some training rows, so the model is evaluated on a past it partly saw. Production is next month, and the number is optimistic (Temporal Leakage).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a part fails inspection from its machining telemetry. The label is the inspection result, recorded within the hour.
- The decision is whether to route the part to a slower manual inspection, and the cost of each mistake is a scrapped good part or a shipped bad one.
- One example is one part: a few dozen sensor aggregates and a pass/fail label. Eighty thousand parts over a year on the same line.
- Each part is machined once and never seen again; parts do not share an identity. The process has been under statistical control for the whole year with no tooling change.
- A sibling team building a model of an e-commerce funnel copied the same split for their session data, where each user has many sessions and behaviour changes monthly.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A random split is a claim: the rows are exchangeable, so any subset is distributed like any other, and production data is one more draw from the same distribution. When both halves of the claim hold, the validation score estimates production performance without bias.
- The claim breaks along two axes. Along entities: rows sharing an entity are correlated, so a shuffle leaks entity-specific information across the cut. Along time: a process that drifts makes later rows differently distributed from earlier ones, and production is always later, so a shuffled validation set is easier than the future.
- The symptom is never an error. It is a validation number that is too good, by an amount you cannot see from the number itself — only from comparing it against a split that respects the structure.
Two datasets, one split, one right answer
The machining data and the funnel data were split the same way. One split is correct and one is wrong, and nothing in the split code distinguishes them; the difference is entirely in the structure of the rows.
The compare states the check that separates them. It is cheap — two splits, two scores — and it is the only way to see a leak that produces a good number.
Shuffle all rows, cut 80/20, train, report the validation score as the expected production performance. Same code for the parts data and the sessions data.
Shuffle and cut; also cut by time (last two months held out) and by entity (users hashed to one side). Train once per split. Report the random score only if the three agree; otherwise report the structured score and explain the gap.
A random split's optimism is invisible from within the random split. Only a split that respects the structure production will impose — new time, new entities — can measure how much the shuffle leaked, and on exchangeable data the three simply agree at no cost but compute.
The cost of a shuffle on recurring entities
On the funnel data, the model was evaluated on sessions from users it had already seen. Users are consistent — the same person converts or does not at a fairly stable rate — so the model learned per-user tendencies through user-correlated features and was rewarded for recall of users rather than prediction of behaviour.
The offline number was strong. The first month's new users, who the model had never seen, converted at the base rate regardless of score.
Strong ranking of conversion on the shuffled 20% holdout; several user-history features near the top of importance.
On new-user traffic, the score barely separates converters from non-converters; on returning users it works about as well as offline. Overall lift far below the offline estimate.
- 1The shuffle placed nearly every user on both sides of the split, so the holdout measured how well the model recognises users it trained on, and production is dominated by users it has not.
- 2Sessions from later months sat in the holdout alongside training rows from the same months, so the temporal drift the model faces in production was absent from the evaluation.
- 3A smaller part of the gap is genuine drift in the funnel since training; that would show in feature distributions rather than in the new-versus-returning split.
What must stay true for the shuffle to keep being right
On the machining line the random split is correct today because parts are independent and the process is under control. Both are facts about the line, not about the split, and both can change: a tool replacement, a new material batch, an operator who runs the machine differently.
So a random split that is correct still ships with an assumption, and the assumption needs the same monitoring as a feature.
Each row is generated independently of every other, by a process whose distribution does not move between the training period and the serving period.
holds when Entities are measured once; there is no batch, shift or session structure that correlates neighbouring rows; feature distributions in production match training within noise.
breaks when A tooling or material change shifts the sensor baseline; a batch effect correlates consecutive parts; the pipeline is reused on data where entities recur.
respond If the temporal check diverges, switch to a time-based split and retrain on the post-change period; if batch correlation appears, split by batch rather than by part.
How to build it
Most important first.
- Ask two questions before shuffling: does the same entity recur across rows, and does the distribution move with time? If both answers are no, shuffle. If either is yes, do not (Choosing a Split Strategy).
- Where a random split is right, stratify it on the label when positives are rare, so folds have comparable positive counts (Stratified Split).
- Fix the seed and record it with the dataset version, so the split can be reproduced and a different seed can be tried to estimate the split's own variance (Random Seeds).
- Even when a random split is right, hold a final check against a later period of production data; a stationary process is an assumption, and this is the test of it.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The validation metric under a random split against the same metric under a time-ordered or group split. If they agree, the rows really are exchangeable and the random split is fine; if the random one is higher, the difference is the leakage.
- The metric's variance across a few split seeds — the spread that any single validation score is subject to.
- The distribution of a few key features in training against the latest production period; a drift here is the stationarity assumption breaking.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Rows are exchangeable: no entity, session, shift or batch links rows so that seeing one in training reveals another in validation.
- The process that generates the rows is stationary over the period the data covers and into the period the model will serve, so a random subset of the past resembles the future.
- The split seed is recorded so the exact validation set can be rebuilt and the score reproduced.
- Offline: compare the random-split score against a time-ordered split score on the same data; agreement supports exchangeability, and a gap measures how much the random split flattered the model.
- Online: compare the first production week's metric against the validation score; a shortfall with no serving bug is the split's assumption failing.
- Over time: monitor feature distributions against training and re-run the random-versus-temporal comparison on each retraining, since a process that was stationary can stop being so.
What can go wrong
- The machining process is stable for a year, then a tool is replaced, the sensor baseline shifts, and the random-split validation from before the change describes a process that no longer exists.
- The parts are independent, but the *shifts* are not — an operator or a batch of raw material affects a run of consecutive parts — and a shuffle spreads each shift across both sets, so the model learns the shift and the validation set rewards it.
- A random split is used correctly, and a new engineer copies the pipeline to a dataset where entities recur, because the split "worked last time".
- A random split is the simplest to implement and the easiest to get wrong without noticing, because it produces a clean number in every case.
- Where it is correct it gives the most training data per row of validation, since no rows are lost to a time gap or an entity boundary.
- It offers no information about temporal generalisation; even where it is right, a later production check is needed to confirm the stationarity it assumes.
- "Random split works for everything." It works when rows are independent and the process is stationary, which is a minority of production problems. The slogan survives because the failure is a good number, not a bad one.
- "Random is unbiased, so it cannot be optimistic." Unbiased over draws of independent rows. When rows share entities or drift with time, the independence assumption fails and the estimate is biased upward.
- "We stratified, so the split is fine." Stratification fixes the class ratio. It does nothing about the same user, or the same month, sitting on both sides of the cut.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a random split estimates production performance only under exchangeability and stationarity is a statement about sampling, independent of the model family.
- DATA-SPECIFICCorrect for independent, one-shot measurements from a stable process — parts on a line, images from a fixed collection, survey responses; wrong for user behaviour, transactions, sensor time series and anything where an entity recurs or the world drifts.
Where the depth lives
This domain teaches the model and hands the rest off by name.