Explainability
An explanation describes the model, not the world, and describes a wrong model just as fluently. Know whether you need a global picture, a local reason or a counterfactual, and whether an interpretable model would make the question go away.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A regulator or a customer wants to know why the model decided what it did. What can an explanation method actually tell them, and what does it only appear to tell them?
An insurer's claims model flags claims for investigation. A customer whose claim was delayed has asked, under a right-to-explanation rule, why. The team has a feature-attribution plot for the claim and is not sure whether sending it counts as an answer.
Compute feature attributions for the flagged claim, send the top five features with their contribution weights, and call that the reason. The method is standard and the plot looks authoritative.
The attribution says which inputs moved this model's output. If the model learned that claims filed on a Monday are suspicious because of a data artefact, the explanation faithfully reports "filed on Monday" as the reason, and the customer is told a falsehood with great confidence.
- The attribution says which inputs moved this model's output. If the model learned that claims filed on a Monday are suspicious because of a data artefact, the explanation faithfully reports "filed on Monday" as the reason, and the customer is told a falsehood with great confidence.
- Two attribution methods give different top features for the same claim. Both are correct about their own approximations; neither is the model's reason, because the model does not have reasons, it has weights.
- The explanation is unstable: a claim one cent different in amount gets a different top feature. A customer who compares notes with a neighbour sees the inconsistency.
- The offline evaluation said nothing about any of this. Explanations are not evaluated; they are generated.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a claim will be found fraudulent on investigation. The label is the investigator's conclusion, produced only for claims that were investigated.
- The decision is fast-track or investigate. An explanation is owed for the decision, which is the threshold applied to the score, not for the score alone.
- One example is one claim with claimant history, claim details, policy attributes and free-text description features derived from an embedding.
- The model is a gradient-boosted ensemble over a few hundred features. Local attributions are computed by a post-hoc method that perturbs inputs and fits a local approximation.
- Investigator conclusions are the label, and investigators saw the model's flag before concluding (Feedback Loops).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Global interpretation asks what the model does on the whole population: which features carry weight, what the partial dependence on each looks like. Local explanation asks why this prediction: which of this input's features moved the output from a baseline. Counterfactual explanation asks what would have to change for the decision to flip.
- Post-hoc attribution methods explain a black box by approximating it — a local linear fit over perturbed inputs, or an allocation of the output difference across features under a game-theoretic rule. The approximation has assumptions (feature independence, a chosen baseline) that the real data usually violates, so the numbers are an estimate of a property of the model, not a fact about the claim.
- An interpretable model — a linear model, a small tree (Decision Trees), a scorecard — is its own explanation: the reason is the model. It may be less accurate, and that trade is the actual decision.
Three different questions
Global, local and counterfactual explanations answer different questions for different people, and a team that generates one kind because a library provides it will send the wrong one. The regulator wants to know what the model does in general; the customer wants to know what would have changed their outcome; the engineer wants to know why this one prediction is strange.
Each has a mechanism and a failure. The comparison below is the reframe: the explanation method is chosen by the recipient's question, not by the model.
| Kind | Question | Method | What it cannot say |
|---|---|---|---|
| Global | What does the model rely on overall? | Permutation importance, partial dependence, the structure of an interpretable model | Anything about one individual decision; whether the reliance is causal |
| Local | Why this output for this input? | Local surrogate fit, additive attribution against a baseline | Whether the model is right; anything stable across methods and baselines |
| Counterfactual | What would have to change to flip the decision? | Search for a nearby input across the boundary | Whether the change is possible or honest for the person to make |
The explanation is of the model
An attribution method takes the model as ground truth and allocates its output across inputs. If the model learned a spurious relationship — claims filed on Mondays are flagged because the fraud team's batch import used to land on Mondays — the attribution reports "Monday" with the same confidence it reports a real signal. The method did its job; the model was wrong; the customer receives a fluent falsehood.
This is what "explanations may be approximate" means in practice. They approximate the model, and the model approximates the world, and the second approximation is the one the explanation cannot see.
Top contributing features: claim filed on a Monday (+0.21), description length (+0.14), policy tenure under a year (+0.09). Sent to the customer as "the reasons your claim was reviewed".
Your claim was selected for review. Claims with a policy tenure over one year and a description matching the itemised receipt are fast-tracked; the review is the model's decision, not a finding, and the reviewer's decision is independent of it.
The first hands a person a model artefact — Monday — as if it were a fact about their claim, and the number is unstable under a different baseline. The second states something actionable and true about the decision boundary and does not imply the model has found anything.
Interpretable by construction
The other route is to not need a post-hoc method. A logistic regression on twenty features is its own explanation: the coefficient is the reliance, the linear form is the whole story. A shallow tree is a list of rules. Where a reason is owed to a person, the trade is often worth making, and it should be made as a trade — with the quality difference measured, not assumed.
The assumption an explanation pipeline depends on is that the thing being explained is the thing that decided. A version skew between the deployed model and the one the explainer loads makes every explanation wrong in a way nobody checks.
The explanation for a decision is computed against the exact model version, feature set and preprocessing that produced the decision.
holds when The prediction log records the model version and feature vector, and the explainer is invoked on that record rather than on a fresh feature fetch.
breaks when The explainer fetches features at explanation time, after the feature values have moved, or loads the current model while the decision was made by the previous one.
respond Explain from the log, never from a live fetch; treat a mismatch as a serving bug, not an explainability bug.
| Option | Quality | Interpretability | Cost | Operational | Note |
|---|---|---|---|---|---|
| Logistic regression on curated features | The coefficients are the explanation; quality depends on feature engineering. | ||||
| Shallow tree or rule list | Rules a person can read; unstable structure across retrains. | ||||
| Boosted ensemble with post-hoc attribution | Best quality; explanations are approximations that must be validated and logged. | ||||
| Two-stage: ensemble ranks, interpretable model decides | Keeps the reason on the decision; two models to maintain. |
caveat The interpretability scores assume the recipient can read the model, which a customer cannot for any of them; and quality here is offline quality, which says nothing about how the explanation changes behaviour once people learn what the model looks for.
How to build it
Most important first.
- Decide who needs the explanation and what they will do with it. A regulator auditing the model needs the global picture and the training data; a customer needs a counterfactual they can act on; an engineer debugging needs local attributions with their instability visible.
- Where a reason is owed to a person, prefer a model whose structure is the reason, or a two-stage design where an interpretable model makes the decision and the complex model only ranks (The Linear Baseline).
- When using post-hoc methods, report them as approximations: show the stability of the attribution under small perturbations, and never present a feature's contribution as the cause of the outcome (Attribution Is Not Causality).
- Validate explanations against the model, not against intuition: if an explanation says a feature matters, removing it should change the prediction.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Attribution stability: how much the top-k features change under small input perturbations or a different baseline. Unstable attributions cannot be sent to anyone.
- Faithfulness: the change in the prediction when the highest-attributed features are ablated. An attribution that does not predict ablation effect is decorative.
- The validation metric says nothing about explanation quality and should not be quoted in its place.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The attribution method's approximation is adequate on the region of input space where explanations are being generated; a method tuned on typical claims can be badly wrong on the unusual ones that get flagged.
- The features in the explanation mean to the recipient what they mean in the pipeline; an engineered ratio with an internal name is not an explanation to a customer.
- The model being explained is the model that made the decision — same version, same features, same preprocessing (Model Lineage).
- Offline: for a sample of decisions, compute attributions with two methods and two baselines and report agreement; ablate the top features and confirm the prediction moves.
- Online: log the explanation alongside the prediction and the model version so the explanation given to a customer can be reproduced later (Prediction Logging).
- Over time: re-run the stability and faithfulness checks on every promoted model; a retrain can change the attribution landscape without changing the metric.
What can go wrong
- The explanation is used to fix the model — "the model should not use that feature" — by removing the feature, and the model recovers the same signal through a correlated one. The explanation changes; the behaviour does not.
- Counterfactuals recommend an action the customer cannot take ("have a longer policy tenure") or one that would be gaming ("describe the claim in fewer words").
- A global feature-importance chart is presented to a regulator as a statement about what causes fraud. It is a statement about what this model, trained on this data, found convenient.
- An interpretable model can cost quality, and the team has to own the quality it gave up for the reason it can give.
- Attribution methods on a large ensemble are expensive per prediction and are usually computed only for the decisions that are challenged.
- Explanation logging stores more personal data per prediction (ML Privacy).
- "The attribution shows the model is using the right features, so it is correct." It shows which features moved the output. A model can use plausible features to reach wrong conclusions, and an attribution cannot tell the difference.
- "Feature importance proves causality." It proves that the model relies on the feature. The feature may be a proxy for the cause, a consequence of the outcome, or an artefact of data collection.
- "Explainability is a checkbox: generate a plot per prediction." An unstable, unfaithful plot sent to a customer is a liability, not compliance.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- MODEL-SPECIFICA linear model or a shallow tree is its own explanation and needs no post-hoc method; a boosted ensemble or a network needs one, and the method's assumptions — independence, a baseline — are where the approximation error lives.
- CONTESTEDA serious position holds that post-hoc explanation methods are so unstable and so easily gamed that they should not be used for anything consequential, and that where a reason is owed the model must be interpretable by construction, whatever the accuracy cost. The counter-position is that interpretable models are also misread — a linear coefficient is not a cause either — and that a well-validated attribution with its instability reported is more honest than a scorecard whose simplicity hides the same proxies. Both agree that an attribution sent without a stability check is malpractice.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Law and policy — what counts as a sufficient explanation under a given regulation is decided outside engineering; this lesson gives what an explanation method can and cannot support.