Data Splitting

Train, validation and test as three different jobs, and the split strategy — random, temporal, grouped, stratified — as the decision that decides whether the metric means anything.

Train / Validation / Test

Three sets with three jobs: learn parameters, choose between models, and estimate final performance once. The percentages are a consequence of the jobs, not a rule.

Q · Which decisions is each set allowed to inform, and how big does each need to be for the number it produces to mean anything?
Random Split

Shuffle the rows and cut. Correct when rows are independent and production looks like the training period. Wrong, and optimistic, whenever time or repeated entities are in the data.

Q · Are the rows independent of each other and of time, so that a shuffled cut resembles the data production will send?
Time-Based Split
▶ lab

Train on the past, validate on the future. The only split that measures the thing production actually asks for — how well the model generalises to a period it did not see.

Q · Production will score data from a period after training. Does the validation set come from after the training set, with a gap that matches the label delay?
Group Split
▶ lab

When the same entity appears in many rows, all of its rows go to one side of the split. Otherwise the model is evaluated on recognising entities it already saw, and production is full of entities it has not.

Q · Does the same user, patient, device or document appear in more than one row, and will production ask about entities the model has never seen?
Stratified Split

When positives are rare, a plain random cut can leave validation with too few of them to say anything. Stratifying fixes the class ratio per set so every fold holds a known number of positives.

Q · Are positives rare enough that a random cut could leave the validation set with too few to measure the metric, and which variable should the split hold fixed?
Choosing a Split Strategy

Four questions decide the split: is there time in the data, do entities recur, are positives rare, and will production see new entities or a new period? The answers compose into one strategy.

Q · Given what production will look like relative to the training data, which combination of time, group and stratified splitting makes the validation number mean what it claims?