Causality vs Prediction
A model that predicts Y from X has learned that X and Y move together in data generated by an old policy. Acting on X to change Y is a different question, and usually needs an experiment rather than a model.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model says customers who receive a discount rarely churn. Should we give everyone a discount?
A subscription business trained a churn model. The strongest feature is whether the customer received a retention discount in the previous quarter; discounted customers almost never churn. Finance has been asked to fund discounts for every at-risk customer on the strength of that finding.
The model has found that discounts prevent churn. Feature importance confirms it. Give discounts to everyone the model flags, and expect churn among flagged customers to fall by the amount the coefficient implies.
Discounted customers rarely churn because the retention team gave discounts to customers it had already persuaded to stay. The discount is a marker of a successful retention call, not a cause of retention. Giving it to customers who never called changes nothing, and costs the discount.
- Discounted customers rarely churn because the retention team gave discounts to customers it had already persuaded to stay. The discount is a marker of a successful retention call, not a cause of retention. Giving it to customers who never called changes nothing, and costs the discount.
- Once the new policy is in place the feature's meaning changes: "received a discount" now means "was flagged by the model", and the next retrain learns that flagged customers churn at the base rate, which looks like the model getting worse (Feedback Loops).
- The offline metric was excellent and remains excellent. Prediction was never the problem; the mistake was reading a prediction as an intervention effect.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a subscriber cancels within thirty days. The label is the cancellation event.
- The decision the business actually wants to make is whether to intervene, which is a question about the effect of the intervention, not about who is likely to churn.
- One example is one subscriber-month with usage, tenure, support contacts, and whether a discount was granted. Discounts were granted by a retention team, by their own judgement, mostly to customers who called to cancel and were persuaded to stay.
- The training set is therefore data from a world where the discount was given to a specific kind of customer chosen by a specific process — the old policy (Selection Bias).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A supervised model estimates P(Y | X) as it appears in the training data. That distribution is generated by a process that includes whatever policy chose the actions in the data. The model learns correlations that the old policy induced, including ones that would vanish under a different policy.
- A confounder is a variable that affects both X and Y — here, the customer's intention to stay affects both whether they get a discount and whether they churn. Conditioning on observed features helps only if the confounder is among them; intention is not.
- The intervention question is P(Y | do(X)): what happens to churn if we set the discount, rather than observe it. Answering that from observational data requires assumptions about the causal structure that the data cannot verify, which is why the reliable answer is a randomised experiment (A/B Testing Models).
What the model learned, and from whom
The training data was generated by a world in which the retention team decided who got a discount. Their decision depended on a signal the model never sees — a conversation in which the customer agreed to stay. The discount marks that conversation. The model learns the mark.
This is the intervention problem: the model learned from data where the action was chosen by an old policy, and the model's deployment replaces that policy. The correlation was a fact about the old policy. It is not a fact about the discount.
Prediction was never the question
The churn model can be excellent at ranking who will cancel and useless for deciding what to do about it. The offline/online gap here is not a metric that lied; it is a metric that answered a different question from the one the business acted on.
Closing the gap means running the experiment. It costs a control group, calendar time, and a retention team willing to leave some flagged customers alone.
Churn model validation ranking is strong; the discount flag is the top feature and its partial dependence shows near-zero churn when set.
Discounts extended to every flagged customer. Churn among flagged customers barely moves; the discount budget is consumed; the next retrain shows the discount feature's effect collapsing.
- 1The discount flag was a marker of a successful retention conversation, not a cause of retention; setting it for customers who never had the conversation sets the marker without the thing it marked.
- 2The retrain sees discount = 1 for customers chosen by the model rather than by the team, so the feature now means "flagged", and its correlation with churn returns to the base rate.
- 3Some customers who would have stayed anyway received a discount they did not need, which lowered revenue without changing churn.
Where you need an experiment instead of a model
The rule is simple to state and hard to follow: if the decision changes the value of a feature the model learned from, the model's estimate of that feature's effect does not survive the decision. The reliable route to the effect is randomisation; observational alternatives exist and carry assumptions.
What must remain true after deployment is that the feature-outcome relationships the model exploits are not the ones the deployment itself alters. That assumption is checked by looking at what the model changes.
The relationships between features and outcome that the model relies on are not altered by the actions taken on the model's predictions.
holds when The model informs a report or a ranking that does not alter the features of the entities it ranks — or the action features are excluded and the intervention effect is estimated separately.
breaks when The model's flag triggers an action that is also a feature, or that changes the outcome for exactly the population the model flagged (Feedback Loops).
respond Remove the action feature from the predictive model, estimate the intervention effect from the holdout, and make the policy decision on the effect rather than on the coefficient.
1def treatment_effect(flagged, rng, treat_fraction=0.8):2 # flagged: customers the churn model scored above threshold3 treated, control = [], []4 for c in flagged:5 (treated if rng.random() < treat_fraction else control).append(c)6 offer_discount(treated) # the intervention under test7 # ... wait 30 days for the outcome ...8 churn_t = mean(c.churned for c in treated)9 churn_c = mean(c.churned for c in control)10 # the number the funding decision depends on11 return churn_c - churn_t, len(control)The control group is flagged and not treated. That is the population the coefficient claimed to know about and never observed, because under the old policy every flagged-and-persuaded customer got the discount.
How to build it
Most important first.
- Separate the two questions in the design document. "Who will churn?" is prediction and a model answers it. "Does the intervention reduce churn, for whom, and by how much?" is causal and needs an experiment or an explicit causal model with stated assumptions.
- Run the experiment: randomise the intervention within the flagged population, hold out a control group that is flagged but not treated, and measure the difference. This is the only way to learn the effect under the new policy.
- When an experiment is impossible, use an observational design — matching, instrumental variables, difference-in-differences — and state its assumptions as assumptions, not findings.
- Never let an action feature that the model's own deployment will change stay in the feature set without a plan for what the retrain will learn (Prediction vs Decision).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The treatment effect: churn among treated flagged customers minus churn among untreated flagged customers, from a randomised holdout. This is the number the funding decision depends on.
- The churn model's validation metric answers a different question and does not inform the discount decision at all.
- Uplift by segment, if the experiment is large enough — the intervention may help one segment and annoy another.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The correlation the model uses between an action feature and the outcome was induced by a policy that is still in force; the moment the model changes who gets the action, the feature means something else.
- The experiment's randomisation is honoured downstream — no one treats the control group out of goodwill.
- The effect measured in the experiment transfers to the population the policy will be applied to, which is only guaranteed for the population that was randomised.
- Offline: audit the feature set for actions chosen by a human or a prior system, and mark each as a policy-induced correlation that must not be read causally.
- Online: a randomised holdout inside the flagged population, with the treatment effect and its interval reported before the policy is scaled.
- Over time: keep a small permanent holdout so the effect can be re-estimated as the population and the intervention change (Exploration vs Exploitation).
What can go wrong
- The holdout is too small to detect the effect, the experiment is read as "no effect", and a working intervention is cancelled.
- The retention team, seeing flagged customers in the control group, treats them anyway because that is their job; the control is contaminated and the experiment measures nothing.
- The experiment shows an effect on average that is driven entirely by one segment, and the policy is applied to everyone.
- An experiment withholds a possibly-beneficial intervention from a control group for the duration, and someone must accept that cost.
- Causal designs on observational data are slower to build, rely on assumptions that cannot be tested, and are argued about by specialists.
- Removing action features from the churn model can lower its predictive quality; that quality was partly the confounder's.
- "The feature importance shows discounts reduce churn." It shows the model relies on the discount flag. The flag is a record of the retention team's judgement about who would stay.
- "We have millions of rows, so the effect is reliable." Volume reduces the variance of the estimate of P(Y | X); it does nothing about confounding, which is bias and does not shrink with data.
- "We can just add more features to control for it." Only if the confounder is among them. The intention to stay is not a column, and no amount of feature engineering makes it one.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat P(Y | X) estimated from data under one policy does not give P(Y | do(X)) under another follows from what a supervised model estimates, whatever the model family or the domain.
- CONTESTEDPractitioners of observational causal inference hold that experiments are often infeasible or unethical and that a well-specified causal model with matching or instruments recovers the effect reliably enough to act on, and that engineers who insist on randomisation leave value on the table. The counter-position is that every observational method rests on an untestable assumption about the causal graph, and that in product settings where randomisation is cheap the experiment is almost always available and almost always more honest. Both agree the churn model's coefficient is neither.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Experimentation and statistics — power calculations, sequential testing and observational causal designs are their own discipline; this lesson says when you need them, not how to run them.