Entity Leakage
The same user, patient or device appears on both sides of the split. The model memorises the entity, the evaluation rewards it, and production is full of strangers.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Every feature is honest and every window is correct. Why does a random row split still overstate how the model will do on people it has never seen?
A hospital analytics team built a model to flag patients at risk of readmission within 30 days of discharge. It validated strongly on a held-out set of admissions. Deployed on a new ward, and then across a partner hospital, its flags were far weaker, and the difference was largest for exactly the patients the hospital had never seen before.
Each admission is an example; shuffle and split the examples. More rows per patient is more data, and the model should learn from all of it.
A patient with ten admissions and nine readmissions has roughly eight of those rows in training. The model learns that this patient — through the id column, or through the near-unique combination of demographics and diagnoses — is a readmitter. Validation contains the other two rows and scores them correctly. It has measured recall of a person, not prediction of an outcome.
- A patient with ten admissions and nine readmissions has roughly eight of those rows in training. The model learns that this patient — through the id column, or through the near-unique combination of demographics and diagnoses — is a readmitter. Validation contains the other two rows and scores them correctly. It has measured recall of a person, not prediction of an outcome.
- In production on a new ward or hospital, the frequent patients are strangers. The features that identified them are now unseen categories or novel combinations, the memorised signal is gone, and the model falls back to whatever it learned about patients in general, which was little because the memorisation was cheaper.
- The simulator's
entity-overlapinjection is this exactly: four snapshots per user scattered across train and validation, plus cohort columns that make the memorisation easy. Validation rises; the future month of new users does not move.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict, at discharge, whether a patient will be readmitted within 30 days. The label is the presence of a subsequent admission within the window.
- The model is meant to work for any discharged patient, most of whom have one or two admissions in the data, and some of whom have dozens.
- One example is one admission. A patient with ten admissions contributes ten rows, which share demographics, chronic diagnoses, and a patient identifier that was one-hot encoded into the feature space by a generic pipeline.
- The split was a random shuffle of admissions. Frequent patients — the chronically ill, who are also the most often readmitted — appear on both sides in nearly every case.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- When the same entity contributes rows to training and validation, the rows are not independent. Anything stable about the entity — its id, its rare attribute combination, its baseline rate — is a shortcut that predicts its validation rows from its training rows without learning anything that transfers to a new entity.
- The model does not need an explicit id column. Enough high-cardinality or near-unique features — a postcode plus a birth year plus a rare diagnosis — identify the entity, and a flexible model will use them (Overfitting to entities rather than to rows).
- Whether this is a leak depends on the question. If production will score the same entities repeatedly, and the training data will always include their history, entity overlap in validation is a fair simulation. If production will meet new entities — a new ward, a new customer, a new device batch — it is leakage, because validation answered a different question from the one production asks (What Is One Example?).
- Near-duplicates are the same problem in disguise: the same document with two ids, the same transaction ingested twice, two images from the same session. Deduplication before the split is part of the split.
Memorising who, not learning what
The patient id was one-hot encoded because the pipeline encoded every categorical column, and even without it the combination of birth year, postcode and three chronic diagnoses identifies most patients. A tree ensemble finds those combinations easily. For a frequent patient with rows on both sides of the split, "this is the patient who always comes back" is a better predictor of their validation rows than any clinical feature.
Nothing about this is visible in feature importance or in the metric. The model is genuinely good at the validation set. The validation set just asked a question — will this known person be readmitted — that a new ward and a partner hospital never ask.
looks like A generic categorical encoding step applied to every string column, plus ordinary demographic and diagnosis features.
why it leaks The same patient's admissions are on both sides of a random row split. The id, and the near-unique combination of stable attributes, lets the model predict validation rows from the same patient's training rows — carrying the label across the split through identity rather than through anything that generalises.
fix Split by patient so each is wholly in one fold; drop or safely encode identifying features; validate on patients absent from training if that is what production will see.
1import numpy as np2 3def group_split(df, group_col, val_frac=0.2, seed=0):4 # every row of an entity lands on one side; the unit of the split is the entity5 groups = df[group_col].unique()6 rng = np.random.default_rng(seed)7 rng.shuffle(groups)8 n_val = int(len(groups) * val_frac)9 val_groups = set(groups[:n_val])10 is_val = df[group_col].isin(val_groups)11 return df[~is_val], df[is_val]12 13train, val = group_split(admissions, "patient_id")14assert not set(train.patient_id) & set(val.patient_id)The assertion is the test worth keeping. If identity is unresolved — the same person under two ids — it passes and the leak survives; the split is only as clean as the entity key.
Which question does production ask?
Entity overlap is not always a leak. A bank retraining its churn model every month on its own customers will, in production, score customers whose past rows were in training. For that deployment, a time-based split on the same customers is the faithful simulation, and a group split answers a question — how does the model do on strangers — that production rarely asks.
The mistake is not choosing a row split; it is choosing any split without deciding what production will see. The decision below is the one to make before the split, and to record with the evaluation so the number is read against the right question (Choosing a Split Strategy).
Strong ranking quality on a random split of admissions, strongest on frequent patients.
At the partner hospital, where every patient is new to the model, flags are far weaker; on the home wards, quality on first-admission patients is well below the validation number while repeat patients look fine.
- 1Frequent patients had rows on both sides of the row split, and the id plus near-unique attribute combinations let the model memorise them; validation rewarded recall of known people.
- 2The partner hospital and first-admission patients are strangers, so the memorised signal contributes nothing and the model falls back on clinical features it barely learned.
- 3A modest genuine difference in the partner hospital's population — case mix, coding practice — accounts for some of the gap, but that would show as feature drift rather than as a seen-versus-unseen split.
Will the entities the model scores in production have been present in its training data?
when The deployment reaches populations the training data did not contain, or entities are short-lived and mostly seen once.
cost Group split by entity; fewer effective examples and a wider confidence interval; drop identifying features.
when The model is retrained regularly on the same population and scores entities whose history is in training.
cost Time-based split on the same entities with a gap; per-entity history features are legitimate; must still guard against temporal leakage inside each entity's rows.
when A mixed population — most predictions are for known entities, a meaningful fraction for new ones.
cost Two holdouts and two numbers; production monitoring stratified by seen-in-training so the weaker population is not hidden in the average.
Near-duplicates are entities without a key
The same problem appears wherever rows are not independent and no id says so: a document scraped twice from mirrors, an image and its resized copy, a transaction ingested by two pipelines, sensor readings a second apart. A random split puts one copy on each side and the model is graded on recall of the training set.
Deduplication before the split is therefore part of the split. Where an exact key exists, use it; where it does not, cluster by similarity and treat the cluster as the entity. The data engineering side of this — Deduplication: Bounded Memory Against an Unbounded Stream, Duplicate Rows — is where the depth lives; this domain's job is to insist it happens before, not after, the split.
No entity, and no near-duplicate of a row, appears on both sides of the split unless production will score seen entities at the same rate.
holds when A group split by a resolved entity key, after deduplication, matched to a stated deployment question.
breaks when Identity resolution degrades and one entity acquires several keys; a new ingestion source introduces duplicates; the deployment expands to a population the split never simulated.
respond Resolve identity or deduplicate, re-split, re-evaluate on the population production actually reaches, and report the number for that population.
How to build it
Most important first.
- Split by entity: assign each patient wholly to one side (Group Split). The unit of the split is the unit production will meet fresh, which is often not the row.
- Decide explicitly what production will see. If the deployment is to new entities, validate on new entities; if it is to the same entities over time, validate by time on the same entities. If both, validate both and report both numbers.
- Deduplicate and cluster near-duplicates before splitting, using the entity key where one exists and a similarity threshold where one does not.
- Be suspicious of high-cardinality features that identify entities — raw ids, exact coordinates, free-text names — and either drop them or encode them in a way that cannot memorise (Categorical Encoding, Target Encoding).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The metric on a validation set of entities absent from training, against the metric on a random row split. The gap is the memorisation the row split was rewarding.
- Production quality stratified by whether the entity was in the training set. A model that is strong on known entities and weak on new ones is the signature.
- The random-split number is not a measurement of generalisation to new entities.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The proportion of production predictions made for entities the model has seen in training is the same as the proportion in the validation set — either both near zero or both matching production's mix.
- Entity identity is resolved well enough that a group split by key actually separates entities, and near-duplicates without a shared key have been clustered before splitting.
- No feature acts as a de facto entity identifier unless production will encounter the same entities at the same rate.
- Offline: compare the random-split and group-split metrics; then compute the metric on validation rows whose entity appears in training against rows whose entity does not.
- Offline: for each high-cardinality feature, check how many rows share each value; a feature with nearly one row per value is an id in disguise.
- Online: log whether each scored entity was present in the training set and report production quality split by that flag once labels arrive.
What can go wrong
- The group split is by patient id, but the same patient has multiple ids across systems, so the split leaks through unresolved identity — a data engineering deduplication problem surfacing as a modelling one.
- The group split is correct and the metric drops sharply, and the team concludes the model is useless — when in fact production really will score known patients most of the time and the row split was closer to the right question.
- A group split removes entity overlap but not temporal overlap: a patient wholly in validation still has admissions in validation that precede admissions in training, so the model has seen the future of the validation period (Temporal Leakage).
- A group split has fewer effective examples and a higher-variance estimate; with a few hundred entities the confidence interval on the metric widens considerably (Metric Uncertainty).
- Dropping identifying features loses genuine per-entity history that is legitimate when production scores known entities; the right encoding is more work than the drop.
- Deduplication and identity resolution are ongoing data engineering costs, and the split is only as clean as they are.
- "Random split is fine because the rows are shuffled." Shuffling scatters an entity's rows across the split; it is the cause of the problem, not a protection against it.
- "We removed the id column, so the model cannot memorise patients." Enough near-unique features identify a patient without an id. The test is a group split, not a column drop.
- "Group split gave a lower number, so the model is worse than we thought." It gave an honest number for new entities. If production scores known entities, the row split was answering the closer question; decide which question production asks.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALNon-independent rows from the same entity inflate any evaluation that splits by row, for any model family — including for images and documents, where the entity is a session, a source or a near-duplicate.
- CONTESTEDThe strongest opposing view is that in many deployments — a bank scoring its own customers monthly, a hospital scoring its own patients — the model will score known entities with their history available, so a row or time split on the same entities is the honest simulation and a group split is needlessly pessimistic. That is correct for those deployments; the failure is applying the row split by default without having asked which deployment this is.
Where the depth lives
This domain teaches the model and hands the rest off by name.