Selection Bias
The dataset only contains the cases that reached the step where the label was recorded. Approved loans have repayment labels; declined ones do not. The model learns about the selected, and is deployed on everyone.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Which process decided that these rows have labels and those do not, and is the model going to be used on the rows that were never labelled?
A lender wants a model to decide which applicants to approve. They have five years of loans and whether each was repaid. The obvious training set is every loan ever issued.
Train on issued loans. They are the only rows with labels, and there are hundreds of thousands of them. A model that predicts default among issued loans will predict default among applicants.
Among approved applicants, thin credit history is only weakly related to default, because the underwriters approved thin-file applicants only when something else looked strong. The model learns "thin file is fine" and approves thin-file applicants the underwriters would have declined.
- Among approved applicants, thin credit history is only weakly related to default, because the underwriters approved thin-file applicants only when something else looked strong. The model learns "thin file is fine" and approves thin-file applicants the underwriters would have declined.
- Validation on held-out issued loans looks excellent, because held-out issued loans are the same selected population. The first cohort approved by the new model has a default rate the offline evaluation never hinted at, and it is visible only twelve months later.
- The model is then retrained on its own approvals, which are a new selected population, and the bias compounds (Feedback Loops).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an applicant would default if approved. The label — default or repaid — exists only for applicants who were approved by the previous policy.
- The decision is approve or decline for every applicant, including the ones the old policy would have declined, which is the whole point of building a new model.
- One example is one issued loan with the application features as recorded at application time and the repayment outcome twelve months later.
- Applicants who were declined have application features and no outcome. They are a third of all applications and are systematically different — lower income, thinner credit history — from the approved.
- The old policy was partly a manual underwriter decision, so the selection mechanism is not fully recorded.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A label is recorded only when some process lets the case reach the point where the outcome is observed. That process — an approval, a purchase, a funnel step, a doctor ordering a test — selects on features, and often on the very features the model uses.
- Within the selected population, the relationship between features and outcome is distorted, because selection removed exactly the feature combinations the old process considered risky. The model learns the conditional distribution given selection and is applied to the unconditional population.
- This is invisible to any offline evaluation on labelled data, because every labelled row went through the same selection. The metric is a true statement about the selected population and says nothing about the rest.
Where the labels came from
Every labelled loan passed through an approval step. That step is a filter applied on the application features, and it removed the applicants the underwriters thought would default. The dataset is what is left, and inside it the underwriters' judgement is already baked in.
The diagram makes the two populations explicit. The model is trained on the right-hand branch and deployed at the decision node.
Excellent offline, expensive online
The held-out set is drawn from issued loans, so it shares the selection. The model ranks the approved population well, and the number reports that honestly. The gap appears twelve months later, in the default rate of the applicants the old policy would have declined.
Closing the gap has a price the offline evaluation does not: a random exploration slice, which means issuing loans the policy expects to lose, and waiting a year to learn from them.
Strong ranking of default among held-out issued loans; a clear improvement over the previous scorecard on the same held-out set.
Default rate among model-approved, policy-declined applicants several times the portfolio average, visible only as the twelve-month labels arrive.
- 1The training population was selected by the old policy on the same features, so the learned feature-outcome relationship does not hold outside the approved region.
- 2The held-out set shared the selection and could not measure the region where the new model changes decisions.
- 3A fraction of the gap is ordinary drift in the applicant pool, but drift would show up in feature distributions immediately rather than only among the newly-approved segment.
The same shape, everywhere
Selection bias is not a lending problem. Every time a label exists only because something upstream let the case through, the training set is the selected population. The table gives the shape for several domains, with the fix that is actually available in each.
The common thread in the response column: the only clean fix is to observe outcomes for a random slice of the unselected, and that always costs something the business would rather not pay.
The cases the model decides in production fall in regions of feature space where the training data has enough labelled examples to have learned the outcome relationship.
holds when An exploration slice supplies labels across the full applicant distribution, or an out-of-support check routes unsupported cases elsewhere.
breaks when The exploration slice is cut; a marketing campaign brings in a new applicant segment; the model is reused for a product with a different applicant pool.
respond Restore exploration or widen the human-review routing before retraining; retraining on selected outcomes reproduces the bias.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Loan default model trained on issued loans | Validation strong; defaults among newly-approved thin-file applicants a year later | Approval selected on the same features; no labels for the declined | Random exploration slice of approvals; restrict decisions to supported region |
| Hiring model trained on past hires' performance | Recommends candidates like past hires; no signal on rejected profiles | Performance observed only for those hired by the old process | Blind evaluation of a random sample of rejected candidates where legally possible; otherwise decision support only |
| Disease model trained on patients who were tested | High offline accuracy; misses cases in patients doctors did not test | Test ordered only when doctor suspected disease | Screening cohort with universal testing; calibrate on it |
| Click model trained on impressions the old ranker showed | Items never shown get no clicks and stay unshown | Previous model selected what was labelled — see Feedback Loops | Logged randomised exploration traffic; inverse-propensity weighting |
How to build it
Most important first.
- Name the selection process explicitly: which step must a case pass to get a label, who decides, on what features. If the model will be used before that step, the training population is the wrong one.
- Where possible, label a random slice of the unselected — approve a small random fraction of applicants the policy would decline, so the model gets unbiased outcomes on the region it will be applied to. This costs real money and is the only clean fix.
- Where that is impossible, treat it as a known limitation: restrict the model's decision to the region of feature space where training data exists, and route the rest to the old process or to a human (Human Oversight).
- Model the selection where the data allows — reweight by the inverse of the estimated approval probability — and state clearly that this assumes the selection depended only on recorded features.
- Compare the scored population's feature distribution against the labelled population's on every rollout, so a model applied far outside its training region is visible (Evaluation Slices).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The share of production decisions that fall in a feature region with little or no labelled support. This number bounds how much the offline metric can be trusted; it is not on any dashboard by default.
- The default rate of the first cohort approved by the new model that the old policy would have declined, once labels mature. That is the number the business will judge the model by.
- Validation AUC on held-out approved loans measures ranking quality among the approved. It is not wrong; it is about a different population.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The population the model scores in production has the same feature-to-outcome relationship as the population that received labels, or the difference is bounded by a deliberate exploration slice or a support check.
- The selection mechanism that produced labels is known well enough to name which feature regions have no labelled support.
- The share of production traffic outside the labelled region is monitored and stays small, or those cases are routed elsewhere.
- Offline: estimate the old approval policy from the application data and plot the model's predicted default rate against approval probability; a model that looks great only where approval was likely has learned the selected region.
- Online: for the first cohorts, hold out a random exploration slice and compare the model's predicted default rate on it against the realised rate when labels arrive.
- Over time: monitor the proportion of scored applicants outside the training support and the default rate of the model-only approvals against the model's own predictions.
What can go wrong
- The random-approval slice is set up, then cut in a cost review because "those loans lose money", and the model quietly returns to learning only from the selected.
- Reweighting is applied, but the underwriters used a feature that was never logged, so the estimated approval probability is wrong exactly where it matters.
- The out-of-support router sends thin-file applicants to humans, the humans learn to rubber-stamp the model's score, and the selection mechanism is now the model.
- A random exploration slice means deliberately approving loans the policy expects to lose money on; the cost is real and must be budgeted as the price of unbiased labels.
- Restricting the model to the supported region means the model does not decide the cases the business most wanted automated.
- Inverse-probability reweighting corrects for selection only on recorded features and adds variance, sometimes more than it removes bias.
- "We have hundreds of thousands of labelled loans, so the data is not the problem." Volume within the selected population does nothing for the unselected. The missing rows are the problem, and no number of present rows replaces them.
- "The model beat the old policy on the validation set." It beat the old policy on loans the old policy approved. On those, the old policy is by construction already good; the comparison the business wants is on the rest.
- "We can fix it by retraining once we have outcomes from the new model." The new model's approvals are also a selected population. Without an exploration slice, retraining replaces one selection with another.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAny dataset whose labels exist only for cases that passed a gate — approvals, purchases, hires, diagnoses, funnel steps — has this structure, whatever the model family.
- DOMAIN-SPECIFICLending, hiring and medicine have severe versions because the gate was a human decision on the same features; an ad click model has a milder version because the gate is an impression the previous model chose, which is at least logged and modelable.
Where the depth lives
This domain teaches the model and hands the rest off by name.