SIMULATED

Leakage Simulator

Inject one leak into a churn-prediction pipeline and watch the validation AUC climb above the AUC on a later month of new users — the only number here that resembles what production would see. The model is really trained; the gap is really computed.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

Leakage is not a list of forbidden columns. It is information that exists at training time and does not exist at prediction time, reaching the model by any route — a feature computed over the wrong window, a column written by the same process as the label, the same person on both sides of a split, a normaliser that saw the held-out rows. Every route produces the same symptom: an offline number that is better than the truth, and no error anywhere. The four leaks below are injected the way they happen in real pipelines, and the honest baseline is on the same page so you can see what agreement looks like.

The leak

Each control is a change a reasonable engineer might make on a Tuesday afternoon.

Future timestamp

Add days_since_last_login, computed from the whole event log — including events after the snapshot date.

What it injects
A feature that knows the future. Users who churned stopped logging in, so "days since last login" measured over the following month is large precisely for the users the label says churned.
Why it leaks
At training time the feature is computed from data that did not exist yet at the snapshot. At serving time only the past is available, the feature is computed correctly, and the correlation the model leaned on is gone.
Temporal Leakage →

What the numbers did

Logistic regression trained on 560 rows, validated on 240 rows from the same table, then scored on 300 rows of new users from a later month.

validation AUC
1.000
honest future-month AUC
0.677
gap
+0.323
what offline said

Validation AUC 1.000 — better than the honest baseline’s 0.731. The validation set came from the same table as training, so it carries the same leak and cannot see it.

what production would do

Future-month AUC 0.677 on people the model has never seen, with every feature computed from the past only. The offline number flattered the model by more than five points of AUC, and nobody would have known until the labels arrived.

Where the model put its weight

A leaked feature does not hide. It becomes the strongest coefficient, because it is the answer wearing a feature's name.

logins_30d+0.239
support_tickets_30d+0.487
plan_price-0.062
tenure_months-0.069
usage_trend-0.095
days_since_last_login ◄+4.880
Weights on standardised features. Positive pushes toward churn (swap), negative toward staying (sorted). The injected feature is marked ◄.

The reading

  • — Validation AUC 1.000 against future-month AUC 0.677. The strongest weight is on days_since_last_login (4.88), a feature computed from events after the snapshot date.
  • — In train and validation, days_since_last_login was measured over the following month too, so churners — who stopped logging in — got large values. The label was baked into the feature.
  • — In the future month the feature is computed the only way it can be at serving time, from the past, and its relationship to churn is the mild one it always really had.
  • — Every number here comes from a synthetic subscription dataset drawn from a seeded generator and a logistic regression actually trained on it; the shapes are real, the magnitudes are illustrative.

How to read this page honestly

What the model is, and what it deliberately refuses to be.

  • SIMULATEDA synthetic subscription dataset — 200 users × 4 monthly snapshots, plus a later month of 300 *different* users — drawn from a seeded generator. A logistic regression is really trained on it by batch gradient descent, and both AUCs are really computed from ranks. Nothing on this page is a stored number; the shapes are real and the magnitudes are illustrative of this data only.
  • SIMULATEDThe baseline has to be honest or the lab is worthless: the *no leak* case splits by user, standardises with training statistics only, and must show validation ≈ future. If it ever showed a gap, the page would be demonstrating a broken split rather than leakage.
  • CONTESTEDThe global-normalisation gap is small, and the page says so instead of inflating it. Standardising is an affine map and a converged linear model ranks identically under any affine map, so the only route for that leak here is the ±3σ cap on z-scores. That is a fact about this data and this model; a tree or a neural network on data with heavier tails can pay far more. The reason to refuse the leak is not that it is always large but that its size is unknowable in advance — and a lab that made it look large would be teaching the wrong reason.

Take it further

Threshold Explorer →

Once the score is honest, it still has to become a decision.

Drift Explorer →

The other way the future stops resembling validation — slowly, after launch.

Data Leakage →

Every route by which the answer reaches the model.

The Leakage Audit →

How to find these in a pipeline you did not write.