ML Testing

A dedicated stack: data and feature tests, training smoke tests, model invariants, serving contracts, robustness — because a green unit test suite says nothing about a model.

The ML Testing Stack

Data tests, feature tests, training tests, model tests, serving tests, integration tests and drift tests — seven layers because a model can fail at every one of them while every unit test stays green.

Q · The unit tests pass and the model is wrong in production — which tests were missing, and where in the pipeline does each kind of test belong?
Data & Feature Tests
▶ lab

Schema, nulls, ranges, cardinality, target prevalence and distribution against the training reference — and the one test that catches most leakage: no feature timestamp may exceed its prediction time.

Q · What should be asserted about the training and serving data on every refresh, and which single assertion catches the failures that make offline metrics lie?
Training Smoke Tests
▶ lab

On a tiny dataset, in CI, in minutes: the pipeline runs end to end, the loss goes down, an artifact appears, and the model can memorise a handful of examples. A pipeline that cannot overfit ten rows is broken, whatever the full run reports.

Q · How do you test a training pipeline on every commit without running the training — and what does "the model can overfit ten examples" prove?
Model Invariant Tests
▶ lab

A probability is in [0, 1]. No output is NaN. A higher income does not lower a credit score. A change to an irrelevant field does not change the prediction. Invariants are the tests a model must pass regardless of its metric, and the ones a metric cannot express.

Q · What must be true of a model's outputs on inputs chosen to probe it — not on a held-out set — and how do you test behaviour that an aggregate metric would never reveal?
Serving Contract Tests
▶ lab

Request schema, preprocessing equivalence, model version, output schema, latency budget and fallback behaviour — the contract between the artifact and the request path, tested on every deploy, with a replay against the training path as the test that catches skew.

Q · What must be true of the path from a request to a prediction for the deployed artifact to be the model that was promoted — and how do you test it before traffic arrives?
Robustness Testing

Missing features, extreme values, noise, rare segments and corrupted inputs — the test is not whether the model stays accurate under damage but whether it does what the design says it should: degrade gracefully, refuse, or fall back.

Q · When the inputs are damaged — missing, extreme, noisy, corrupted, from a segment the model barely saw — what should the model do, what does it actually do, and how do you test the gap?
Model Regression Tests

A challenger with a better aggregate metric can still lose the cases the champion passes. A regression test holds the challenger to the champion's slices and to a golden set of known hard examples — and the golden set is a leakage risk the moment anyone trains on it.

Q · The challenger beats the champion on the validation metric — how do you check it has not quietly become worse on the slices and the specific cases the product depends on?