Data Splitting
Train, validation and test as three different jobs, and the split strategy — random, temporal, grouped, stratified — as the decision that decides whether the metric means anything.
Three sets with three jobs: learn parameters, choose between models, and estimate final performance once. The percentages are a consequence of the jobs, not a rule.
Shuffle the rows and cut. Correct when rows are independent and production looks like the training period. Wrong, and optimistic, whenever time or repeated entities are in the data.
Train on the past, validate on the future. The only split that measures the thing production actually asks for — how well the model generalises to a period it did not see.
When the same entity appears in many rows, all of its rows go to one side of the split. Otherwise the model is evaluated on recognising entities it already saw, and production is full of entities it has not.
When positives are rare, a plain random cut can leave validation with too few of them to say anything. Stratifying fixes the class ratio per set so every fold holds a known number of positives.
Four questions decide the split: is there time in the data, do entities recur, are positives rare, and will production see new entities or a new period? The answers compose into one strategy.