Model Invariant Tests
A probability is in [0, 1]. No output is NaN. A higher income does not lower a credit score. A change to an irrelevant field does not change the prediction. Invariants are the tests a model must pass regardless of its metric, and the ones a metric cannot express.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What must be true of a model's outputs on inputs chosen to probe it — not on a held-out set — and how do you test behaviour that an aggregate metric would never reveal?
A lending team's new credit model has a better validation metric than the old one. An analyst, playing with the demo, raised an applicant's income and watched the approval probability fall. The team says it is a rare interaction and the metric is what matters. Compliance disagrees.
The validation metric is the test of the model. If the new model's metric is better, it is better; individual odd predictions are noise in a large population.
The metric is an average over the validation distribution, where high-income high-amount applicants are rare. The model can be badly wrong on that region and barely move the aggregate. The region is where the largest loans are (Evaluation Slices).
- The metric is an average over the validation distribution, where high-income high-amount applicants are rare. The model can be badly wrong on that region and barely move the aggregate. The region is where the largest loans are (Evaluation Slices).
- The income inversion is not noise. It is a learned interaction the model applies consistently: in a region with few training examples the boosted trees found a split that reads high income as a risk signal. It will apply it to every such applicant, and each of them can see it.
- On a candidate with a rarely-seen category, a downstream feature divides by a zero count and the model returns NaN; the serving path casts NaN to zero, which is the lowest possible risk. It approves. No metric on the validation set contains a NaN because the validation set is drawn from the training distribution.
- Two identical applications with different application ids get different scores, because the id was accidentally included as a numeric feature. The metric is unaffected; the model is nondeterministic in a way the product cannot explain (Entity Leakage is the adjacent failure).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The model predicts the probability of default; the decision is approve, refer or decline at thresholds (Thresholding). This lesson's target is a set of properties the output must satisfy on *any* input, stated as tests, that the validation metric is silent about.
- The properties come from the domain (a higher income should not raise default risk, all else equal), from the mathematics (a probability is bounded), and from the product (a different application id must not change the score).
- Applicant features — income, debt, tenure, requested amount, a few dozen more — with a default label. The training data is skewed: high-income applicants with high requested amounts are rare, and the model has learned an odd interaction there.
- The invariant tests use synthetic inputs constructed to probe: a base applicant with one feature swept across its range; pairs that differ only in an irrelevant field; inputs at and beyond the training range; inputs with every feature missing.
- A small directional set: cases where the domain says which way the score must move — more debt, higher risk; longer tenure, lower risk — checked as inequalities, not values.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An invariant is a property of the model as a function that holds for every input, or for every input in a stated region, independent of the label. Range invariants: a probability output is in [0, 1]; a regression output is within physical bounds; no output is NaN or infinite. Monotonic invariants: for a named feature, the output is non-decreasing (or non-increasing) as the feature increases with everything else held fixed. Invariance to irrelevant change: perturbing a field that should not matter — an id, a formatting difference, an unused column — leaves the output unchanged. Directional expectations: on a constructed input, changing a feature in a stated direction moves the output in a stated direction, as an inequality.
- These are properties of the function, so they are tested by constructing inputs, not by sampling a dataset. A sweep holds a base input fixed and varies one feature across its range; a pair test constructs two inputs differing in one field. The tests are cheap — a few thousand forward passes — and run on every candidate.
- Monotonicity can be *enforced* rather than only tested: gradient-boosted tree libraries accept monotonic constraints per feature, and linear models are monotone by construction in each feature's coefficient sign. Enforcing it restricts the function class and may cost a little metric; testing it detects a violation after the fact (XGBoost and LightGBM as Implementations for where the constraint is applied).
- The invariants a metric cannot express are exactly the ones the product and the regulator ask about: whether the score behaves sensibly for one person, not whether it ranks a population well. A metric is a statement about a distribution; an invariant is a statement about a function (Explainability is the adjacent question of why the function did what it did).
Properties of a function, not of a dataset
A metric is computed on a sample and says how the model does on average over that sample's distribution. An invariant is a statement about the model as a function: for every input in a region, the output satisfies a property. Testing one means constructing inputs, not drawing them, because the inputs that violate an invariant are usually the ones a sample rarely contains.
The four families — range, monotonic, invariance, directional — are each a few lines of code around a forward pass. The sweep in the code below holds one applicant fixed and walks income across its range; if the score ever rises as income does, the model has a behaviour that no metric would reveal and that every high-income applicant will meet.
1import numpy as np2 3def assert_monotone(model, base, feature, grid, direction="decreasing"):4 xs = [dict(base, **{feature: v}) for v in grid]5 p = np.array([model.predict_proba(x) for x in xs])6 assert np.all(np.isfinite(p)) and np.all((0 <= p) & (p <= 1))7 diffs = np.diff(p)8 bad = diffs > 1e-9 if direction == "decreasing" else diffs < -1e-99 if bad.any():10 i = int(np.argmax(bad))11 raise AssertionError(12 f"{feature}: score moved the wrong way between {grid[i]} and {grid[i+1]} "13 f"({p[i]:.4f} -> {p[i+1]:.4f}) for base {base}")14 15def assert_irrelevant(model, base, field, alternatives):16 p0 = model.predict_proba(base)17 for v in alternatives:18 assert model.predict_proba(dict(base, **{field: v})) == p0, field19 20# default risk must not rise with income, holding everything else fixed21for base in probe_bases(): # typical AND rare-region applicants22 assert_monotone(model, base, "income", np.linspace(0, 500_000, 200))23 assert_irrelevant(model, base, "application_id", ["A1", "Z9", "0000"])The base inputs are the whole test. A sweep from a typical applicant passes on a model that only fails for the rare high-income, high-amount corner; probe_bases() has to include the corners, and the way to know which corners is to look at where the training data is thin.
Enforce, test, or both
Some model families can be told the invariant: gradient-boosted trees accept a per-feature monotonic constraint and grow only splits that respect it; a linear model is monotone in each feature by its coefficient sign; a network can be built with monotone layers for chosen inputs. Enforcement guarantees the property everywhere, at the cost of restricting the function and possibly a little metric.
The test still belongs in the promotion path. The constraint is configuration that can be dropped, the feature can be transformed upstream in a way that flips its sign, the next model family may not support the constraint at all — and the test is the written specification of what the product requires, which the constraint is only one way to satisfy.
| Option | Quality | Interpretability | Operational | Cost | Note |
|---|---|---|---|---|---|
| Unconstrained model, invariant test only | Best metric; the test catches violations after training; a failure means a retrain with a fix or a product decision. | ||||
| Monotonic constraint in a tree ensemble, plus the test | The property holds everywhere; a small metric cost where the data disagreed with the domain; the test guards the configuration. | ||||
| Linear model in the constrained features | Monotone by construction and fully explainable; usually a larger metric cost; the right answer when the regulator wants the coefficient. |
caveat The scores cannot say how much metric the constraint costs on this data — sometimes none, sometimes a lot — nor whether the domain's monotonicity claim is actually true; both are measured, per problem, by training the constrained and unconstrained models and comparing them on the slices where they differ.
The serving path can break an invariant the model satisfies
A model that never outputs NaN can still approve a NaN applicant if the serving handler casts it to zero. A model that is invariant to the application id can become sensitive to it if the feature service starts passing the id through. The invariants are properties of the deployed function — model plus preprocessing plus handler — and the tests have to run through all of it.
The assumption that must hold, then, is not only that the model satisfies its invariants but that nothing between the request and the decision undoes them. The probe fixtures are the same; the entry point is the serving endpoint.
Every invariant the candidate passed in the promotion path also holds for the deployed function — preprocessing, model and handler — on the same probe inputs.
holds when The probe set runs through the serving endpoint before deploy; the handler refuses non-finite outputs rather than defaulting them; the feature list at serving equals the one the pair tests were written against.
breaks when A handler adds a cast or clip; a feature service adds a field; preprocessing is updated on one side; the model family changes and a constraint is lost.
respond Roll back the deploy, not the model, if the model passed in promotion; fix the handler or the feature list; then redeploy through the same probes.
How to build it
Most important first.
- Write down the invariants before training, from the domain: which features must be monotone and in which direction, which fields must be irrelevant, what the output range is, what a missing-everything input should produce. This is a specification, and it belongs with the problem formulation (Problem Formulation).
- Enforce monotonicity in the model where the family supports it and the domain demands it; test it regardless, because the constraint can be misconfigured and a future model family may not support it.
- Build the probe sets — sweeps, pairs, edge inputs, directional cases — as versioned fixtures and run them against every candidate in the promotion path, before any metric comparison.
- Treat NaN, infinity and out-of-range outputs as hard failures, and make the serving path refuse rather than cast: a NaN score must not become an approval by way of a default.
- Report invariant violations by region — where in feature space the monotonicity failed — so the fix can be a constraint, more data in that region, or a decision to exclude it, rather than a retrain and hope.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Violation count per invariant per candidate, with the region of feature space where it occurred. Zero is the bar for range and NaN invariants; for monotonicity, zero on the sweep grid.
- The metric cost of enforcing a constraint, measured against the unconstrained model — the number that tells the product owner what the sensible behaviour costs.
- Pair-test maximum output difference for irrelevant fields, which should be exactly zero for deterministic models.
- The validation metric is not an invariant test and cannot substitute for one. A better metric with a monotonicity violation is a model that is better on average and wrong for a named person.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The stated invariants remain the domain's actual requirements — the feature that must be monotone today is not later legitimately non-monotone because the product changed.
- The probe fixtures cover the regions of feature space where violations are likely: the rare, high-value corners with few training examples, not only the typical base case.
- The serving path preserves the invariants the model satisfies — it does not cast NaN to zero, clip a probability differently, or add a feature the pair tests did not know about.
- Offline: run the probe sets against every candidate in the promotion pipeline, before the metric comparison, and block on any range, NaN or invariance violation; report monotonicity violations by region.
- Before deploy: run the same probes through the serving path end to end, so a cast or a clip in the handler is caught (Serving Contract Tests).
- Over time: sample production requests into the probe set — especially those from rare regions — so the fixtures follow the population the model actually sees.
What can go wrong
- The sweep grid is coarse and the violation lives between grid points; or the base input is typical and the violation lives in the rare region the analyst found by hand.
- Monotonic constraints are enforced in training and the test is dropped as redundant; a later change of model family loses the constraint and the test with it.
- An invariance test on an irrelevant field passes because the field is not in the feature list today; a feature-store change adds it tomorrow, and the test fixture does not include the new field.
- The NaN check is done on the validation set — where there are no NaNs — rather than on constructed edge inputs, and passes forever.
- Enforcing monotonicity restricts the model and can cost metric where the data genuinely has a non-monotone relationship the domain did not anticipate.
- Probe fixtures are a specification the team must maintain as features and product rules change; an out-of-date invariant is a test that blocks a correct model.
- A blocking invariant test in the promotion path will occasionally stop a model that is better on the metric and wrong on one corner, and the argument about which matters more is a product decision, not a technical one.
- "The metric improved, so the inversion is a rare edge case." The metric is an average over a distribution in which the edge case is rare. The inversion is a consistent behaviour of the function that every applicant in that region experiences.
- "We enforce monotonic constraints, so we do not need the test." The constraint is configuration on one model family. The test is the specification, and it survives a change of family, a misconfiguration and a preprocessing step that flips a feature's sign.
- "NaN never appears in validation, so the model does not produce NaN." Validation is drawn from the training distribution. NaN comes from inputs outside it, and only a constructed input finds those.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALRange, NaN and invariance-to-irrelevant-change hold for every model that outputs a probability or a bounded quantity; monotonic and directional invariants exist wherever the domain has a known sign, which is most tabular problems and few perceptual ones.
- DOMAIN-SPECIFICWhich features must be monotone is a domain fact — income and default risk in lending, dose and response in medicine — and in a domain like recommendation there may be no feature with a required direction, leaving only the range and invariance tests.
- CONTESTEDWhether monotonic constraints should be enforced in the model or only tested is disputed. The strongest case for enforcement is that a constraint guarantees the property everywhere, including between grid points, and is what a regulator can be shown; the strongest case against is that real relationships are sometimes non-monotone in a way the domain expert did not anticipate, and a constraint bakes an assumption into the function where a test would merely have flagged it for a human to consider.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — invariant tests are property-based tests over constructed inputs, and the discipline of choosing generators that reach the corners is a testing skill this lesson assumes.