SplittingGENERALSCALE-SPECIFIC

Choosing a Split Strategy

Four questions decide the split: is there time in the data, do entities recur, are positives rare, and will production see new entities or a new period? The answers compose into one strategy.

Target & data What to measure What must stay true

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

Given what production will look like relative to the training data, which combination of time, group and stratified splitting makes the validation number mean what it claims?

The problem

A platform team is writing a shared training pipeline for a dozen model teams and wants one split function. Every team so far has copied a random 80/20 from a tutorial, and two of the resulting models have under-delivered badly on rollout.

The obvious approach

Provide train_test_split(rows, test_size=0.2, random_state=42) as the pipeline's split step. It is what every team is already doing, it is one line, and it has a seed so it is reproducible.

Why it breaks

The churn and fraud models were evaluated on customers and cardholders they had trained on, in periods they had trained on. Both rolled out to new customers in a new month and under-delivered by the size of the leak, which the offline number could not show.

How it breaks — usually after the offline metric looked fine

The churn and fraud models were evaluated on customers and cardholders they had trained on, in periods they had trained on. Both rolled out to new customers in a new month and under-delivered by the size of the leak, which the offline number could not show.
The forecast model was evaluated on interpolation between weeks it had seen, and over-ordered in the first month.
The document classifier was fine — its rows are independent and stable — which taught the other teams that the random split "worked", and the failures were blamed on the models.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Not one target: a churn model, a fraud model, a demand forecast, a document classifier and a medical-risk model will all use the pipeline. Each has a different answer to the four questions.
The platform's decision is what the pipeline should require callers to state, so that a split cannot be chosen by omission.

Data

The churn data is customer-months with recurring customers and drift. The fraud data is transactions with recurring cardholders, rare positives and fast drift. The forecast data is store-product-weeks. The document data is one row per document, independent and stable. The medical data is admissions within patients within hospitals.
Every one of them was split at random by its team, and every one of them produced a good offline number.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

A split is a claim about how production data relates to training data. The claim has independent parts: whether production is a later period, whether production contains entities not in training, and whether the class ratio is stable and rare. Each part, when true, imposes a constraint on the split.
The constraints compose rather than compete. Time decides the cut between periods. Grouping decides which side each entity lands on. Stratification decides the class ratio within each side. A dataset with all three properties needs all three, applied in that order: cut by time, assign entities within the evaluation period, stratify within entity assignment where positives permit.
The random split is what remains when every constraint is absent: no time, no recurring entity, no rare class. It is the base case, not the default.

Four questions, composed

The decision is not a choice of one strategy from four. Each question, answered yes, adds a constraint, and the strategy is whatever satisfies all the constraints present. The options below are the leaves; most real datasets land on a composite.

The last option is the random split, with its precondition stated. It is a correct choice for the document classifier and a wrong one for the other four.

Which split does production imply?

Is there time in the data? Do entities recur? Are positives rare? Will production see new entities, a new period, or both?

Time cut with a label-delay gap

when Snapshot dates exist and the distribution drifts, or production is always a later period. Applies even if entities do not recur.

cost Newest data cannot be trained on; a single cut is one draw and needs rolling repeats; the number is lower than a random split's.

Group split by the entity production is new on

when The same user, patient, device or document appears in many rows and production will meet entities not in training.

cost Whole entities move between sides, so the evaluation set is lumpier; positives concentrated in a few entities can be starved from validation.

Time cut, then group within it

when Both of the above: recurring entities and drift. Churn, fraud, medical risk, most user-behaviour data.

cost Two constraints, a smaller evaluation set, and separate reporting for new versus returning entities in the evaluation period.

Stratify inside whichever structure applies

when Positives are rare enough that a set could hold too few to measure, or a key segment's rate differs sharply.

cost Fiddly to implement inside a group or time split; can fragment small data into strata too small to cut.

Plain random split

when No time axis, no recurring entity, no rare class — independent one-shot rows from a stable process, and production draws from the same distribution.

cost Nothing, when the precondition holds; when it does not, a confident number for a task production never poses.

The same data, four numbers

The paired evaluation is the platform's most useful feature. For each model it reports the metric under the random split and under the chosen strategy. The random number is always at or above the structural one, and the gap is the leak that would otherwise have shipped.

The matrix gives the shape for the five datasets. The values are not measurements; they are the pattern of which number is trustworthy for which deployment question.

Dataset	Time?	Recurring entity?	Rare positives?	Production is new on	Strategy
Churn, customer-months	Yes	Yes	Somewhat	Period and some customers	Time cut + group by customer, stratify if positives permit
Fraud, transactions	Yes, fast drift	Yes, cardholder	Yes	Period; mostly known cardholders	Time cut + stratify; report new-cardholder slice separately
Demand, store-product-weeks	Yes	Yes, store-product	n/a (regression)	Period	Rolling time cuts; entities on both sides is correct — production forecasts known stores
Documents, one row each	No	No	No	New documents from the same distribution	Random split; verify with a later-batch check
Admissions within patients	Yes	Yes, patient and hospital	Somewhat	New patients, or new hospitals	Group by the level production is new on, then time cut; stratify within

The split encodes the deployment scope

Choosing "time cut plus group by customer" says: this model will serve a later period and some customers it has not seen. That is a deployment claim, and when the deployment changes — the model is reused for a new market, or a partner's customers — the claim and the number both expire.

The platform can enforce that by storing the strategy with the model and comparing it to the serving scope at promotion time.

must stay trueDeployment scope matches the split's claim

The model serves data whose relationship to training — later period, new entities, same class ratio — is the one the split strategy assumed, so the validation number is an estimate of production performance.

holds when The strategy and its parameters are recorded in the registry with the model; promotion checks the serving scope against them; retraining re-answers the four questions.

breaks when The model is deployed to a new site or market it was grouped within rather than across; the label delay grows past the gap; drift accelerates past what rolling cuts saw; entities start recurring in data that was one-shot.

how you would know First-period production metrics sliced by new versus returning entities against the structural validation scores; a promotion-time check that the stated serving scope matches the split strategy on record.

respond Re-split under the actual deployment scope and re-evaluate; retraining under the old split reproduces the old, wrong number with fresher weights.

Split strategy recorded with the dataset version

1dataset: churn_customer_month
2version: 2026-06-01
3split:
4  strategy: time+group
5  time:
6    snapshot_column: snapshot_date
7    cut: 2026-04-01
8    label_delay_days: 30        # gap between train end and validation start
9    validation_days: 30
10    test_cut: 2026-05-01        # single-use, after all validation cuts
11  group:
12    column: customer_id
13    val_frac: 0.2
14    seed: split-v1
15  stratify: none                # positives sufficient after grouping; see positive_counts
16positive_counts: { train: 3120, validation: 640, test: 610 }
17deployment_scope: later period, new and returning customers
18paired_random_split_metric_gap: recorded per model at evaluation

The deployment_scope line is the one that makes the record useful at promotion time. A model with this record proposed for a new-market launch is a mismatch the registry can flag, not a surprise the first month reveals.

How to build it

Most important first.

Make the pipeline require an explicit answer to each question — a snapshot-date column or none, a group-id column or none, a stratify column or none — and refuse to run with all three unset unless the caller asserts independence.
Derive the strategy from the answers with the decision below, and record the strategy and its parameters in the dataset version alongside the seed.
Ship a paired evaluation: for every model, compute the metric under the chosen strategy and under a random split, and surface the gap; a gap is a leak that would otherwise have shipped as a good number.
Add the test period as a separate, later cut, produced by the same strategy, and gate it behind a single-use step (Never Tune on the Test Set, Train / Validation / Test).
Document which production question each strategy answers — new period, new entity, both — so a model's validation number is reported with the question it answers.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

The metric under the chosen strategy — the number that predicts production for the stated deployment scope.
The gap between the chosen strategy and a random split on the same data; the size of what the random split would have hidden.
The positive count in each evaluation set under the chosen strategy, since the structural constraints can leave it small.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

The deployment scope stated when the split was chosen — new entities, new period, both, neither — is the scope the model actually serves, and a change in scope triggers a re-split rather than a reuse of the old number.
The columns the split keys on — snapshot date, entity id, stratum — are correct and stable: the date is the prediction moment, the id identifies one real entity, the stratum is available on every row.
The relationship between training and production data is of the kind the strategy modelled; a regime change or a new population is outside what any split can estimate.

How to verify — offline, online, and over time

Offline: run the paired evaluation and confirm the structural score is at or below the random score; check group disjointness and the time gap with assertions in the pipeline.
Online: compare first-period production metrics against the structural validation score, sliced by new versus returning entities.
Over time: re-answer the four questions at every retraining — data that had no drift can acquire it, and an entity that was unique can start recurring.

What can go wrong

Failure modes in production

The pipeline requires the columns, and teams pass a group column that is a surrogate key regenerated on each load, so the group split silently changes between runs.
The time cut is applied, but the label-delay gap is left at zero because the caller did not know the label window, and a small temporal leak survives.
The paired evaluation shows a gap, and the team reports the random number because it is higher and the reviewer does not know which to trust.

What the recommended approach costs

A pipeline that demands answers is slower to adopt than one line, and teams will pass none to make it run; the paired evaluation is the safeguard against that.
Structural splits produce lower, noisier numbers, and a platform that enforces them will be blamed for making models look worse.
Composing all three constraints on a small dataset can leave an evaluation set too small to use, and the pipeline has to say so rather than produce a number.

Misreads

"Random split works for everything." It works when there is no time, no recurring entity and no rare class. Each of those properties is a constraint the random split ignores, and most production data has at least one.
"We used a time split, so we are covered." A time split answers the new-period question. If production also contains new entities, the same entity across the cut still flatters the number, and a group split is needed alongside it.
"The structured split gave a worse number, so it is the wrong split." It gave the number for the task production poses. The random number was for a task — interpolation among seen entities — that production never poses.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThat time, entity recurrence and rare positives each impose a constraint on the split, and that the constraints compose, follows from what a validation set is for, whatever the model family.
SCALE-SPECIFICOn very large datasets every constraint can be satisfied at once with room to spare; on a few thousand rows the constraints collide and the honest answer may be that no split produces a usable comparison, which a small team must hear rather than work around.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringgrain snapshot-tables surrogate-keys