Attribution Is Not Causality
Every importance and attribution method describes how a model's output depends on its inputs. None describes what would happen if you changed the world. "X predicts Y" and "X causes Y" are different claims, and the business hears the second.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model says support tickets are the strongest predictor of churn and that discounts predict retention. Can we act on those as causes — and if not, what would it take to know?
An executive read the churn model's attribution report. Two proposals followed: close the self-service ticket queue "because tickets drive churn", and cut the retention discount "because the model says discounted customers churn less, so we are wasting money on people who would stay anyway." The ML team is asked to confirm the numbers.
The attribution method says which features push each prediction up or down, and it is additive and local, so it must be telling us what drives the outcome per customer. Tickets push churn up; discounts push it down. Reduce tickets, reconsider discounts.
The ticket queue is closed. Customers with problems still have problems and still churn — now without warning, because the feature that used to flag them is gone. Churn does not fall; the model's recall collapses.
- The ticket queue is closed. Customers with problems still have problems and still churn — now without warning, because the feature that used to flag them is gone. Churn does not fall; the model's recall collapses.
- The retention discount is cut.
discount_activewas assigned to people who had already decided to leave and were talked out of it; the model learned that discount recipients stay because the discount is what made them stay. Remove it and those customers leave. - Both readings were consistent with the attribution. The model was not wrong about prediction. It was asked a question it does not answer.
- The team's "explanations" were approximate: a local additive attribution on a non-additive model is a linear story about a curved surface, and the story for an individual customer changes with the baseline it is computed against.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a subscriber will cancel within 60 days. The label is the cancellation event.
- The model was built to rank accounts for outreach. It is now being read as a study of what causes churn, which is a different target that no one designed for.
- One example is one subscriber-month with usage aggregates, support-ticket counts, plan and tenure, and
discount_active— whether a retention discount is currently applied. discount_activeis not a property of the customer; it is the output of last year's retention process, which offered discounts to customers who called to cancel and were persuaded to stay.- Support tickets are filed by customers who are having problems. Customers with problems churn. The tickets are a symptom the data recorded, not the disease.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A predictive model estimates the conditional distribution of Y given X *as the data was generated*. If tickets and churn share a cause (a bad experience), tickets predict churn perfectly well without causing it — a confounder produces correlation in both directions.
- An intervention changes how X is assigned. Setting
discount_activeby policy is a different data-generating process from last year's, where the discount was assigned *because* the customer was about to leave. The model has no way to know which process produced its inputs; its weights encode the old one. - Local additive attributions (the SHAP family, at concept level) distribute a prediction's deviation from a baseline across features so the shares sum to the deviation. That is a description of the fitted function around one input. It inherits every distortion of the model — a wrong model produces confident, additive, wrong attributions — and it still answers "what did the model use", now per row (Explainability).
Three ways a feature predicts without causing
A confounder: a bad product experience causes both tickets and churn, so tickets predict churn while doing nothing to it. A symptom: the ticket is a recording of a problem that already exists. A policy output: the discount was assigned by a process that looked at the customer's intent to leave, so the feature carries the process's knowledge, not the discount's effect.
The model cannot distinguish any of these from a genuine lever, because all of them produce the same thing in the training table — a column that co-varies with the label. Prediction uses co-variation; intervention needs the direction of the arrow, and the arrow is not in the table.
The intervention changes the feature
Last year the discount went to customers who called to cancel. discount_active therefore means "was about to leave and was persuaded". A model trained on that table learns that meaning. Now change the policy — give the discount to everyone, or to no one — and the same column means something else. The model's weights are about the old meaning.
This is the practical form of the distinction. A prediction is a statement conditional on the data-generating process; an intervention replaces the process. Any feature that is itself a decision the business makes is the sharpest case, because the business is proposing to change the very mechanism that gave the feature its predictive power.
looks like A clean boolean column, well populated, strongly associated with retention, and a discount is a plausible retention lever.
why it leaks The discount was assigned by a process that observed the customer's intent to cancel. The column encodes that observation. The model learns "discount recipients stay" because the process only discounted people it could persuade — the answer reached the model through the policy.
fix Keep the feature for prediction if the policy is stable, and answer the policy question with a randomised experiment; if the policy is about to change, the feature's meaning changes and the model needs retraining under the new process.
Local attributions are approximate descriptions of the model
Shapley-style attribution splits a prediction's difference from a baseline across the features so the shares add up. On a linear model this recovers the coefficients times the feature deviations exactly; on a tree ensemble or a network it is a linear summary of a non-linear function around one input, exact in its accounting and approximate in its meaning. Change the baseline — the "average customer" — and the shares change.
It is also expensive at scale and inherits everything wrong with the model: an attribution of a leaked or skewed model is a faithful account of the leak. That is valuable for debugging — a customer whose prediction is dominated by one odd feature is a bug report — and it is exactly why it is not a study of customers.
Each feature the model relies on is still produced by the same process that produced it in training, so the feature still means what the weights assume it means.
holds when No policy that assigns a feature value — discounts, outreach, limits, queue routing — has changed since the training window, and features that are policy outputs are documented as such in the artifact.
breaks when The business acts on the attribution report by changing the policy behind a feature; the feature keeps its name and column and loses its meaning.
respond Retrain under the new process once enough outcomes exist; until then, treat the model's predictions for the affected segment as unreliable and say so.
"Yes, tickets are the strongest driver and discounts reduce churn by a large margin per the attribution." Two policy changes follow; churn does not fall and recall collapses.
"The model relies on tickets and discounts to *predict* churn. Whether closing the queue or cutting the discount would *change* churn is a different question the model cannot answer. We can randomise the discount for a subset next quarter and measure it; for tickets, the honest read is that they flag customers with problems, and the lever is the problem."
The first answer is true about the model and false about the world; the second separates the two claims and proposes the only measurement that answers the second. The cost is a quarter and a control group; the alternative was two wrong decisions.
How to build it
Most important first.
- Label every attribution as a description of the model. On the report, in the dashboard title, in the meeting: "features the model relies on", never "drivers".
- When the question is causal, answer it with an intervention: an experiment that assigns the discount by randomisation to a subset, or a policy change rolled out to a random region, with the outcome measured against control (A/B Testing Models).
- Where an experiment is impossible, use a causal design and state its assumptions out loud — which confounders are measured, why the unmeasured ones are believed small — and treat the result as conditional on those assumptions (Causality vs Prediction).
- Know which features are post-treatment or policy outputs.
discount_activeis the result of a decision the business makes; a model that reads it will learn the policy, not the customer (Feedback Loops).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For a causal claim: the difference in outcome between treated and control under random assignment, with its interval. That is the only number that maps to "should we do X".
- For the model: recall at the outreach budget, which is what it was built for and what the attribution report distracts from.
- Do not measure "impact of tickets on churn" by attribution magnitude. It measures the model's use of a symptom.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The process that assigns each feature value — especially policy outputs like discounts and outreach flags — is the same in production as in the training data; a policy change changes what those features mean.
- Everyone acting on the attribution report understands it as a description of the model, and the labelling that says so survives the report being forwarded.
- Causal claims the business is acting on came from an intervention or a stated causal design, and the population the intervention covered is the population the action is applied to.
- Offline: for each top feature in the attribution, write down the mechanism that would make it predictive without being causal — a confounder, a symptom, a policy output — before anyone presents the chart.
- Online: when a business action is taken on a feature "because the model says so", measure the outcome against a held-back control; if the outcome does not move, the reading was correlational.
- Over time: watch the model's recall after any policy change that touches a feature it uses; a drop after the ticket queue closes is the confounder revealing itself.
What can go wrong
- The experiment is run and the discount has a real effect; the finding is then generalised to all customers, though the experiment only randomised among those who called to cancel. The effect on people who never called is still unmeasured.
- The ticket queue stays open but the outreach team starts calling every high-ticket customer; the calls themselves change behaviour, and next year's training data has the model's own action baked into the label (Feedback Loops).
- The attribution method is upgraded to a more faithful one, the charts change, and the business reads the change as a new finding about customers rather than a new approximation of the same model.
- Experiments cost the treatment on the control group — giving no discount to people who would have stayed with one — and take as long as the outcome window to read.
- Refusing to present attribution as causal frustrates a business that wants levers; the alternative is to present it and let the levers be pulled on symptoms.
- Local attributions are genuinely useful for debugging a single prediction and for regulatory explanation; reserving them for that use gives up their appeal as strategy.
- "Feature importance proves causality." It proves the fitted function depends on the feature. A symptom, a confounded proxy and a policy output all have high importance and none is a lever.
- "The attribution is local and additive, so it is the model's reasoning for this customer." It is a linear decomposition around one point of a function that is not linear, against a baseline that was chosen. Change the baseline and the shares move.
- "Discounted customers churn less, so the discount works." Or the discount was given to the customers who were about to leave and it worked on them; or it was given to loyal customers as a reward and they would have stayed anyway. The data cannot tell those apart; an experiment can.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat conditional dependence in observational data does not identify the effect of an intervention is a property of the data-generating process, not of any model family or attribution method; it holds for a linear coefficient, a split gain and a Shapley value alike.
- DOMAIN-SPECIFICIn domains where the features are physical measurements no policy assigns — sensor readings, weather — the confounding is usually milder and attributions are closer to mechanism; in domains where the features are outputs of the business's own decisions — discounts, outreach, credit limits — nearly every strong feature is a policy in disguise and the gap is at its widest.
- CONTESTEDA serious position holds that local additive attributions are the best available explanation of a complex model and that dismissing them as "not causal" throws away a tool that catches real bugs, satisfies regulators and builds trust with users. That is right about their use for debugging and explanation of the model; the disagreement is only about whether they may be read as statements about the world, and the position defended here is that they may not, however faithful they are to the model.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — measuring a policy change against a held-back control is the same discipline as a controlled rollout with a baseline cohort, and the statistics of reading that comparison honestly are a topic this domain assumes rather than answers.