RepresentationGENERALDATA-SPECIFICCONTESTED

Permutation Importance

Shuffle one column, re-score the model on held-out data, and the drop is what the deployed model depends on. Done on training data it measures memorisation; done one correlated feature at a time it splits the credit and hides the group.

Target & data What to measure What must stay true

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

How do we measure what the deployed model actually depends on, in a way that does not reward leaked or high-cardinality features, and what does the method get wrong on correlated columns?

The problem

A pricing team is about to decommission a third-party data source to save its licence fee. It feeds four features into a demand-forecast model. The split-gain chart shows those four near the bottom, and finance wants the licence cancelled this quarter. The team lead wants a number that says what would actually happen to forecast error if the feed disappeared.

The obvious approach

Read the split-gain chart: the four feed features are near the bottom, so the feed contributes little. Cancel the licence, retrain without the columns, and expect the forecast to hold.

Why it breaks

Split gain was computed on training data with the internal price features present. The feed features rank low because the ensemble met the internal price features first; that says nothing about whether the model — or the next model — needs the feed.

How it breaks — usually after the offline metric looked fine

Split gain was computed on training data with the internal price features present. The feed features rank low because the ensemble met the internal price features first; that says nothing about whether the model — or the next model — needs the feed.
The team computes permutation importance on the training set and the four features look strong, because the trees partly memorised training rows through them. The number describes memorisation, not dependence that generalises.
Computed correctly on the held-out period, one feature at a time, each of the four shows a small drop — and dropping all four together shows a large one. The signal is shared among them and among the internal price features; per-column shuffling splits the credit until every column looks dispensable.
Shuffling a competitor-price column independently of own-price creates rows where the competitor undercuts by an amount that never occurs; the model is being scored on inputs from outside the data distribution, and its errors there are not evidence about the world.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict daily unit demand per product for the next 14 days. The label is realised sales, observed as each day closes.
The decision at hand is not about a prediction but about the model's inputs: whether removing a source materially degrades the forecast the buying team plans against.

Data

One example is one product-day, with lagged sales, calendar features, price and promotion state, and four features from the third-party feed — competitor price index and category-level demand signals.
The third-party features are correlated with the model's own price and promotion features; competitors move price in response to the same events.
A held-out period of the most recent eight weeks exists, split by time, on which the model was validated (Time-Based Split).

How it actually works

Precisely enough to predict its behaviour — not a framework API.

Permutation importance holds the model fixed and destroys the relationship between one feature and the label by shuffling that column across rows. The metric drop between the intact and shuffled data is how much the model's performance depended on that feature carrying information.
It is model-agnostic — any predictor with a score can be probed — and, on held-out data, it measures dependence that generalises rather than dependence that memorised. It does not reward split-point count, and a leaked feature shows up as a huge drop, which is the right signal for the audit.
When features are correlated, shuffling one leaves its siblings intact, so the model recovers most of the signal from them and the drop is small for each individually. The information is not gone; it is shared. Group permutation — shuffling the correlated set together — measures what the group carries.
Shuffling also breaks the joint distribution. A shuffled column takes values that are individually plausible but jointly impossible next to the unshuffled ones, and the model is being evaluated on inputs it never saw (Robustness Testing cares about the same off-distribution region).

Shuffle one column, re-score, read the drop

The model is fixed. Take the held-out data, score it, then shuffle a single column so its values are randomly reassigned across rows and score again. The column's marginal distribution is unchanged — same values, same mean — but its relationship with the label and with the other columns is destroyed. The drop in the metric is how much the model's held-out performance was carried by that relationship.

Because the model is not retrained, this probes the deployed artifact rather than what a retrain would do. Because the data is held out, a feature that the model used to memorise training rows produces no drop — the memorised rows are not there. That is the property split gain lacks.

Permutation importance on the holdout, repeated, with groups

1def permutation_drop(model, X_val, y_val, columns, score, rng, repeats=10):
2    base = score(model.predict(X_val), y_val)
3    drops = []
4    for _ in range(repeats):
5        Xp = X_val.copy()
6        # shuffle the whole group together: one permutation applied to every column in it
7        perm = rng.permutation(len(Xp))
8        for c in columns:
9            Xp[:, c] = X_val[perm, c]
10        drops.append(base - score(model.predict(Xp), y_val))
11    return mean(drops), std(drops)
12 
13# per-feature: columns=[c] for each c   -> credit is split among correlated siblings
14# per-source:  columns=feed_columns      -> what disappears with the licence
15# NOT on X_train: the drop there measures memorisation through the column

The one permutation applied to every column in the group is what keeps the group internally consistent while breaking its link to everything else. Shuffling each column of the group separately would also shuffle them against each other.

Correlated features share the credit

When the competitor-price index and the model's own price feature move together, shuffling one leaves the model most of the signal through the other. Each individual drop is small; the group drop is large. Reading the per-feature chart, every column in the group looks safe to remove — and removing them all removes the signal.

This is the mirror of the split-gain artefact. Split gain gives all the credit to one sibling; per-feature permutation gives too little to each. Neither is lying; both are answering "this column, with the others held as they are". The question the business asked — "this source, gone" — is a group question and needs a group answer.

Method	Data it is computed on	Correlated siblings	Rewards leaks?	Cost
Coefficient magnitude	Training fit	Split arbitrarily; unstable	Yes, and by scale	Free
Split gain	Training loss	Credit to the first sibling split on	Yes, strongly	Free
Permutation, per feature, held-out	Validation / holdout	Credit divided; each looks small	Shows as a huge drop (good for the audit)	features × repeats × scoring
Permutation, by group, held-out	Validation / holdout	Measures the group as a unit	Shows as a huge drop	groups × repeats × scoring
Retrain without the group	Full training + holdout	Answers "can the next model cope"	Removes the leak and shows the true level	One training run per group

A probe of the model, not a property of the source

The group drop says what this artifact loses if the feed vanishes. The licence decision also needs what the next artifact could do without it, which is the retrain-without ablation on the same split — a cheaper source may substitute, or an internal feature may cover most of the gap. Both numbers, with intervals, are the honest input to finance.

Whatever the decision, the dependence is a property of the model that was probed. The next retrain can shift weight elsewhere, and a source that was load-bearing becomes decorative or the reverse. Recomputing at every retrain and diffing the result is how the number stays true.

Licence cancelled on the per-feature chart

offline evaluation said

Per-feature permutation drops for the four feed columns each small on the recent eight-week holdout; split-gain chart agrees they are near the bottom. Model retrained without them; validation error unchanged within noise on the same eight weeks.

production did

First promotion season after the cancellation: forecast error on promoted products roughly doubles against the prior year; the buying team over-orders on the model's numbers.

What explains the gap — most likely first

1The four columns shared one signal — competitor response to promotions — and per-feature shuffling split it into four small drops; as a group it was load-bearing.
2The eight-week holdout contained no promotion season, so neither the permutation test nor the retrain-without ablation was evaluated in the regime where the feed mattered.
3A genuine change in competitor behaviour that year, which would have hurt the old model too — checkable by scoring the old artifact on the new season if the feed data can be recovered.

what it costs to close or detect The honest version costs a holdout that spans a full seasonal cycle — which for a fourteen-day forecast means holding out a year of the most recent data — plus a group permutation and a retrain-without ablation per candidate source. That is a week of work against a licence fee, and the finance deadline was this quarter.

must stay trueThe holdout contains the regime where the feature matters

The held-out period used for permutation includes the conditions — seasons, promotions, market moves — under which the probed features carry their signal.

holds when The holdout spans at least one full cycle of the business, or is stratified to include the regimes the buying team plans for; the same holdout is used for the retrain-without ablation.

breaks when The most recent eight weeks were quiet and the feed only matters in promotion season; the drop is small because nothing that needed the feed occurred during evaluation.

how you would know Compute the drop per regime slice of the holdout (promotion weeks vs not) and compare; a group whose drop is concentrated in one slice is load-bearing there whatever the average says (Evaluation Slices).

respond Extend or re-stratify the holdout before deciding; if that is impossible, state the regime the number covers and do not let it stand for the others.

How to build it

Most important first.

Compute it on held-out data — the validation fold, or a time-based holdout for temporal data. Never on the training set.
Repeat the shuffle several times per feature and report the mean drop with its spread; a single shuffle on a small holdout is noise.
Cluster correlated features and permute the clusters. Report the cluster's drop as the answer to "what happens if this source disappears", which is the question the business asked.
For a decision like cancelling a source, also run the direct experiment: retrain without those columns on the same split and compare the held-out metric with uncertainty (Metric Uncertainty). Permutation importance is a probe of the current model; retraining answers what the next model can do without the feed.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

The held-out metric drop when the whole third-party group is permuted, in the units the buying team plans in — forecast error in units of stock, not a relative importance score.
The held-out metric of a model retrained without the group, against the current model, with intervals. That is the number for the licence decision.
Do not measure per-column drops on correlated features and sum them, or read the smallest as "safe to remove". The sum under-counts and the smallest is an artefact of sharing.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

The held-out data used for the permutation test is drawn from the same distribution — and, for temporal data, the same regime — that the model serves in, including the seasons where the probed features carry their signal.
The correlation structure among features is the same at serving time as in the holdout; a group that shared a signal in the holdout still shares it, so the group-level drop is still the right unit.
The importance was computed against the artifact that is serving, and is recomputed when the artifact changes, because dependence is a property of a model, not of the data.

How to verify — offline, online, and over time

Offline: permutation drops on the validation fold, repeated with several shuffle seeds, individually and by correlated group; then the direct ablation — retrain without the group — with intervals from resampling or repeated folds.
Online: for a feature the model depends on heavily, a serving-time monitor on that feature's distribution and null rate, because a dependence that large is a single point of failure (Serving Fallbacks).
Over time: recompute at every retrain and diff the group-level drops; a group that became dispensable or became load-bearing is a change in what the model is, and should be part of the promotion review.

What can go wrong

Failure modes in production

The holdout is small; the per-feature drop is inside the noise of a single shuffle and the ranking reorders on every run. A conclusion is drawn from one of those runs.
The holdout is not representative of the period where the feed matters — it contains no promotion season — and the feed looks dispensable because nothing that needs it happened during the evaluation window.
The group permutation says the feed matters; the licence is kept; nobody re-checks after the next retrain, which now leans on a new internal feature and no longer needs the feed. The importance was a property of one model.

What the recommended approach costs

Cost scales with features × repeats × the price of scoring the holdout; on a large model with hundreds of features it is hours of inference, which is why people fall back to split gain.
Group permutation needs someone to decide the groups, which is a judgement the split-gain chart never asked for.
The off-distribution problem has no clean fix; conditional permutation methods keep the joint distribution plausible at the cost of complexity and their own assumptions.

Misreads

"Each of the four feed features has a small drop, so the feed is dispensable." Shuffle them together. Four small individual drops and one large group drop means the signal is shared, not absent.
"Permutation importance is model-agnostic, so it tells us what matters in the data." It tells you what this model depends on. A different model on the same data can depend on a different subset of the same shared signal.
"We computed it on training data because that is where we had labels." On training data it measures how much the model memorised through that feature. Use the holdout, or wait for the labels.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALHolding a model fixed and destroying one feature's information measures the model's dependence on it whatever the model family or task; what changes across tasks is the metric being dropped, and across data the size of the correlated-feature artefact.
DATA-SPECIFICOn tables with many correlated columns per-feature permutation is close to useless and group permutation is required; on a small set of near-independent features the per-feature drops are a direct and stable answer and the caveats here mostly disappear.
CONTESTEDA serious position holds that permutation importance should be avoided on correlated features entirely, because shuffling produces impossible input combinations and the metric drop measures the model's behaviour on data that cannot occur — conditional or refit-based methods are the honest alternative. That criticism is correct about the mechanism; the reply is that group permutation over correlated clusters, on held-out data, with the direct retrain-without ablation next to it, answers the operational question well enough for most decisions at a fraction of the cost.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Observability & Performancecost-per-request

Domains that do not exist yet

— Testing & Reliability Engineering — a permutation test is an ablation, the same idea as a fault-injection test that removes a dependency and measures degradation; deciding which dependencies deserve a standing ablation in the promotion pipeline is a reliability-engineering question this domain assumes rather than answers.