SplittingGENERALTASK-SPECIFIC

Time-Based Split

Train on the past, validate on the future. The only split that measures the thing production actually asks for — how well the model generalises to a period it did not see.

Target & data What to measure What must stay true ▶ Leakage Simulator

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

Production will score data from a period after training. Does the validation set come from after the training set, with a gap that matches the label delay?

The problem

A retailer wants weekly demand forecasts per store and product so the replenishment team can order stock. The first model, validated on a random holdout, was very accurate offline and over-ordered heavily in the first month of use.

The obvious approach

Shuffle the store-product-weeks and hold out a fifth. Every week is a row; rows are rows; a random holdout estimates performance on unseen rows.

Why it breaks

A held-out week whose neighbours are in training is interpolation, not forecasting. Trailing aggregates for the held-out week include sales the model also saw as training labels for the surrounding weeks. Offline error is tiny; forecasting the genuinely unseen future is much harder (Temporal Leakage).

How it breaks — usually after the offline metric looked fine

A held-out week whose neighbours are in training is interpolation, not forecasting. Trailing aggregates for the held-out week include sales the model also saw as training labels for the surrounding weeks. Offline error is tiny; forecasting the genuinely unseen future is much harder (Temporal Leakage).
The model was tuned to the average of three years and evaluated on a holdout that was also an average of three years. The last six months, which look like the next six, were a sixth of the evaluation instead of all of it.
When the team switched to holding out the last eight weeks, the score dropped sharply. The model had not changed; the random number had been measuring a task the model would never be asked to do.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict units sold per store-product in the coming week. The label is the realised sales figure, known at the end of that week.
The decision is the order quantity; over-forecasting means write-offs and under-forecasting means empty shelves, at different prices per category.

Data

One example is one store-product-week: trailing sales aggregates, promotions, calendar features and last year's same-week sales. Three years of weekly history.
The random holdout held out a fifth of the store-product-weeks, scattered across the three years, so for almost every held-out week the model had seen the weeks before and after it for the same store-product.
Demand shifted noticeably in the last six months as a competitor opened stores; the random holdout diluted those months across the whole period.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

A time-based split orders rows by their snapshot moment and cuts once: everything before the cut trains, everything after validates. The validation set is then a sample of the future relative to training, which is what production is.
Two refinements matter. A *gap* between the last training snapshot and the first validation snapshot, at least as long as the label window, so that no training label was observed during a period a validation feature summarises. And *rolling* or *expanding* windows — several cuts at successive dates — so the estimate is an average over several futures rather than one, and its variance can be seen (Time-Series Validation).
The score from a time-based split is lower than from a random split on the same data whenever the data drifts, and the difference is not a defect of the split. It is the size of the temporal leak the random split was hiding, and it is how much the model will under-deliver in production.

The cut, the gap and the rolling window

The picture has three parts. A cut date separates training from validation. A gap the length of the label delay separates the two so no training label sits inside a validation feature window. And the cut is repeated at successive dates so the estimate is an average over several futures.

Test is the final period, after every validation period, and it is looked at once.

time ──────────────────────────────────────────────────────────────►

cut 1:  [ train ............ ]  gap [ validate ]
cut 2:  [ train ..................... ]  gap [ validate ]
cut 3:  [ train .............................. ]  gap [ validate ]
final:  [ train ....................................... ]  gap [  test  ]

gap  = label delay (30 d for a 30-day churn label; 0 for same-hour labels)
metric = mean and spread over cuts 1-3; test period touched once

A time split written out

The mechanism is small enough to write by hand, and worth writing by hand once, because every library helper hides the gap. The function takes snapshot dates, a cut date and a label delay, and returns row indices for training and validation.

Note what is not here: no shuffle, no random seed, and no reference to the row order in the file. The snapshot date is the only thing that decides which side a row lands on.

Temporal split with a label-delay gap

1from datetime import date, timedelta
2 
3def time_split(snapshot_dates, cut: date, label_delay: timedelta, val_len: timedelta):
4    """Rows whose snapshot is before (cut - label_delay) train.
5    Rows whose snapshot is in [cut, cut + val_len) validate.
6    Rows inside the gap are dropped: their labels were observed
7    during the validation period and would leak it into training."""
8    train, val = [], []
9    for i, d in enumerate(snapshot_dates):
10        if d < cut - label_delay:
11            train.append(i)
12        elif cut <= d < cut + val_len:
13            val.append(i)
14    return train, val
15 
16# rolling evaluation: several cuts, one metric each
17cuts = [date(2026, 1, 1), date(2026, 2, 1), date(2026, 3, 1)]
18scores = []
19for c in cuts:
20    tr, va = time_split(snapshots, c, timedelta(days=30), timedelta(days=30))
21    scores.append(evaluate(train_on(tr), va))
22# report mean(scores) and max(scores) - min(scores), not a single number

The rows inside the gap are the subtle part. A 30-day label observed on day 25 of the gap was observed during the validation period, and a training row carrying it has seen part of the future the validation set is meant to represent.

Where the time split still leaks

Ordering rows by time closes the biggest leak and leaves a smaller one open: any feature computed from a table that holds only current values describes the row's future no matter which side of the cut the row is on. The split cannot fix a feature; it can only stop the validation set from sharing the training set's period.

The leakage device shows the shape. It is the same feature a correct pipeline would use, computed from the wrong table.

leakagestore_formatTime-split rows, current-value feature

looks like A categorical feature — small, standard, flagship — joined from the stores dimension. Constant per store, so it seems harmless to join from the current table.

why it leaks Several stores were converted to the flagship format during the data's span, after their demand rose. Every historical row for those stores carries flagship, so the feature encodes "demand will rise here" for periods before the conversion happened.

offline

The time-split score is still flattered: training rows for soon-to-convert stores carry the future format, and validation rows for the same stores agree, so the model is rewarded for a pattern that does not exist at prediction time.

production

For a store about to convert, production sees the old format and the model under-forecasts the rise; for a store just converted, the model behaves as though the rise already happened.

fix Join the format valid as of the snapshot date from a slowly-changing dimension, so each row carries the format the store actually had that week (SCD Type 2 in Practice).

when this feature is fine The same feature is correct when it is joined point-in-time: at prediction time the service really does know the store's current format, and a row that carries the format valid at its own snapshot describes exactly that.

must stay trueThe validation period stands in for production

The drift between the training period and the validation period is of the same kind and size as the drift between the validation period and the period the model will serve.

holds when The rolling-cut scores are stable across successive periods and the most recent cut resembles the coming period in calendar, promotion mix and competitive situation.

breaks when A regime change — a competitor opening, a pricing overhaul, a pandemic — larger than any seen across rolling cuts occurs after the cut; the label delay grows so the gap is no longer long enough.

how you would know Spread across rolling cuts widening at retraining; first-period production error outside the range of the rolling scores; feature distributions in production outside the range seen across validation periods.

respond Shorten the training window to the post-change period if the change is permanent, and treat the pre-change score as no longer informative about the model.

How to build it

Most important first.

Cut on the snapshot date, never on a row index or a random draw, and make the cut date a recorded parameter of the dataset version.
Leave a gap equal to the label delay between the training cut and the validation start; for a 30-day churn label, the last 30 days before the validation period cannot contribute training labels.
Use several successive cuts — an expanding window over the last few periods — and report the mean and spread of the metric across them.
Keep the final test period after all validation periods, as the most recent fully-labelled data, and touch it once (Never Tune on the Test Set).
When a time split is combined with recurring entities, keep the time cut and accept that the same entity appears on both sides — production will also see old entities in new periods — but check separately how the model does on new entities (Group Split).

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

The metric on the most recent validation period and its spread across rolling cuts. This is the number that predicts production; a single random-holdout number is not comparable to it.
The gap between the random-split metric and the time-split metric on the same data — the size of the leak, and a useful thing to show anyone attached to the higher number.
The metric on the validation period broken down by how far into the period each row is; a model that decays within the validation window will decay in production on the same schedule (Performance Decay).

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Production scores a period after the training data ends, and the validation period stands in for it — the drift between training and validation resembles the drift between validation and production.
No feature in a validation row depends on events after the training cut that a training label also depends on; the gap is at least the label delay.
The distribution shift from one period to the next is of a size the model can tolerate; a regime change larger than any seen across rolling cuts is out of scope for the estimate.

How to verify — offline, online, and over time

Offline: compute the metric under several rolling cuts and under a random split; the random score should be at or above every rolling score, and the rolling spread bounds what production may do.
Online: compare the first production period's metric against the most recent rolling cut; they should be close, and a large gap means the validation period was not representative or something else leaks.
Over time: re-run the rolling evaluation at every retraining and track whether the newest period's score is drifting away from older periods' scores.

What can go wrong

Failure modes in production

The rows are ordered by time, but a feature is computed from a table that holds current values, so training rows still describe the future (Point-in-Time Correctness).
The gap is omitted; the last month of training labels overlaps the first month of validation features, and a small temporal leak survives an otherwise correct split.
The single cut lands on an anomalous period — a holiday, a stock-out, a competitor launch — and the estimate is a statement about that period alone.

What the recommended approach costs

A time-based split gives a lower number, which is a harder conversation, and a noisier one, because the validation period is a single stretch rather than a sample from everywhere.
The gap and the test period together mean the newest data cannot be trained on, so the deployed model is always trained on data that ends some weeks before it ships.
Rolling cuts multiply training cost by the number of cuts.

Misreads

"The time-split score is lower, so the model got worse." The model is the same. The random split was measuring interpolation; the time split measures forecasting, and forecasting is harder. The lower number is the true one.
"We ordered by time, so there is no temporal leakage." The split is only half of it. Features computed from current-value tables, and a missing gap, leak the future into training rows regardless of how the rows are ordered.
"One recent holdout period is enough." One period is one draw. A holiday quarter or a stock-out month gives a number that says more about that month than about the model.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALWhenever the prediction is made at a moment and evaluated after it, a validation set from after the training set is the only one that measures the production task; this holds for any model family.
TASK-SPECIFICFor forecasting and any entity-over-time task the time split is mandatory; for a static classification task on independent items with no drift — a fixed image collection, a one-off survey — time may not be a meaningful axis and a random split is correct.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringsnapshot-tables scd-type-2 event-time late-arriving-data