Serving Contract Tests
Request schema, preprocessing equivalence, model version, output schema, latency budget and fallback behaviour — the contract between the artifact and the request path, tested on every deploy, with a replay against the training path as the test that catches skew.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What must be true of the path from a request to a prediction for the deployed artifact to be the model that was promoted — and how do you test it before traffic arrives?
A retailer's recommendation service was redeployed with a new feature-service version on a Friday. The model was unchanged. By Monday click-through had dropped and nobody could say why: the health check was green, the latency was normal, and the responses were well-formed.
Deploy the model server, hit the health endpoint, send a sample request and check for a 200 with a well-formed body. The model is unchanged, so the model is fine.
The new feature-service version changed the default for a missing last_purchase_days from 999 to 0. Zero means "bought today" to the model. Every user with no purchase history now looks like an active buyer, and the ranking is confidently wrong for the whole cold-start segment (Cold Start).
- The new feature-service version changed the default for a missing
last_purchase_daysfrom 999 to 0. Zero means "bought today" to the model. Every user with no purchase history now looks like an active buyer, and the ranking is confidently wrong for the whole cold-start segment (Cold Start). - The response is well-formed and fast. Nothing in the health check, the schema check or the latency monitor can see that the number inside the response is computed from a feature vector the model never trained on.
- The model server loaded the right artifact, but the handler reads the model version from a config file that was not updated, so every logged prediction is attributed to the previous version. The regression is invisible in the per-version dashboard (Prediction Logging).
- On the first spike after the deploy the model server timed out and the handler's fallback — return the popularity list — kicked in for a fifth of requests, which nobody noticed because the fallback returns the same shape (Serving Fallbacks).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model scores (user, item) pairs for a ranking (Designing a Recommendation System). This lesson's target is the serving contract: the set of promises the request path makes to the artifact and the artifact makes to the caller, each stated as a test.
- The contract has two sides. The caller's side is schema and latency. The model's side is that the feature vector it receives is the one its weights were trained for (Train / Serve Skew).
- Requests arrive as JSON with a user id, a list of candidate item ids and a context. The feature service enriches them into a vector per pair; the model server scores; the handler sorts and returns.
- The training path computes the same features from the warehouse. The two implementations were written separately and reconciled by a document.
- A replay sample: a few thousand production requests from last week with their logged serving feature vectors, kept as a fixture.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A serving contract test is a test of the deployed path with the real artifact behind it, run before traffic. Its components: request schema — the handler rejects malformed input and accepts the documented shape; preprocessing equivalence — for a replay sample, the feature vector the serving path produces matches the training path's to a tolerance; model version — the artifact loaded is the one promoted, and the version the handler logs is the one loaded; output schema — types, ranges and shape of the response, including that a probability is a probability; latency budget — the endpoint meets its budget at the expected batch shape on the deployed hardware (Latency Breakdown); fallback behaviour — when the model server is unavailable or slow, the handler does what the design says and *marks* the response as a fallback.
- Preprocessing equivalence is the test that catches skew, and it is the one most contract suites lack. It replays logged requests through the serving path and through the training feature path and diffs the vectors. A default that changed, a window that differs, a null policy that diverged all show as mismatches on the replay — before any prediction is made (Preprocessing Lives in the Artifact is the design that makes this test trivial by bundling the preprocessing with the weights).
- The model-version test closes the loop with the registry: the artifact's recorded hash must equal the hash of what the server loaded, and the version the handler stamps on each logged prediction must equal that (Artifact Integrity). Without it, dashboards attribute behaviour to the wrong model.
- These are backend contract tests in shape — an endpoint, a fixture, an assertion — with two ML-specific additions: the equivalence replay against a second implementation, and the requirement that the fixture reach the rare paths (unknown categories, missing fields, out-of-range values) where skew hides.
Six promises, one of them about the number
Five of the six contract components are ordinary backend contract tests: the request schema, the output schema, the latency budget, the version, the fallback. They are necessary, they are cheap, and they are what most teams already have. All five pass on a service returning the wrong prediction for every cold-start user.
The sixth — preprocessing equivalence on a replay — is the one that looks at the number. It compares the feature vector the serving path builds for a logged request against the vector the training path builds for the same request, and any mismatch names the feature and the request shape before a single prediction is served.
- 1Request schema
Malformed requests rejected with the documented error; the documented shape accepted, including optional and missing fields.
fails by Tests only the happy shape; a missing optional field is accepted with an undocumented default.
- 2Preprocessing equivalence
Replay a stratified fixture through the serving feature path and the training feature path; diff every feature; zero mismatches.
fails by A uniform fixture that never reaches the rare branch; a tolerance wide enough to hide a unit change.
- 3Model version
Hash of the loaded artifact equals the registry's promoted hash; the handler's logged version is read from the artifact.
fails by Comparing a config value to the registry rather than hashing what is loaded.
- 4Output schema
Types, shape and ranges of the response; probabilities in [0, 1]; no non-finite values; the invariant probes through the endpoint (Model Invariant Tests).
fails by Shape only; a NaN cast to zero passes.
- 5Latency budget
p99 at the deployed batch shape on the deployed hardware within budget.
fails by Measured on a warm single request; the production batch shape is never tested.
- 6Fallback
With the model server failed or slowed past its timeout, the handler returns the designed fallback, marked as such.
fails by The fallback is returned unmarked and is indistinguishable from a prediction.
Only the second step can fail on the Friday deploy in the problem. The other five pass — which is why a team with only the other five had a good weekend and a bad Monday.
The replay is the skew test
The serving path and the training path both claim to compute last_purchase_days. The replay takes logged requests and asks both to compute it, then compares. A default that changed from 999 to 0 for users with no purchase is a mismatch on every such request, and the diff names the feature and the request shape. The failure is caught in the deploy pipeline, on a fixture, with the diverging feature identified.
The fixture is the whole test. A uniform sample of last week's requests is mostly active users with purchase histories, and the cold-start branch is not in it. Stratify by the rare values of every categorical and by every missing-field pattern, or the test is green because it never asked.
The model artifact was unchanged; its most recent offline evaluation on warehouse features was the same as at promotion. The deploy pipeline's health, schema and latency checks passed.
Click-through fell over the weekend; the drop was concentrated among users with no purchase history, who were being ranked as active buyers. Nobody could attribute it because the logged model version had not changed.
- 1The feature-service deploy changed the default for a missing
last_purchase_daysfrom 999 to 0, so the cold-start segment received a feature vector from outside the training distribution. - 2The contract suite had no preprocessing-equivalence step, so the only test that could have named the feature did not exist; the tests that ran could not see inside the response.
- 3The handler read the version from a config rather than the artifact, so the per-version dashboard showed one unchanged model and hid the step change.
What the contract must keep promising
The contract is a set of assumptions the artifact makes about its surroundings, and each has a test. The one that decays fastest is equivalence, because both implementations change: a feature service ships weekly, a warehouse job monthly, and each legitimate change fails the diff until the other side catches up. The gate is right to fail; the organisational question is who owns getting it green again.
The others decay more slowly and more quietly. A handler refactor that reintroduces a config-based version; a latency budget that was measured at batch size one and is now served at sixty-four; a fallback path that a retry policy change now takes far more often. Each is caught by re-running the suite on every deploy — and by charting the fallback rate, which is the only one of the six that has a production signal of its own.
The deployed request path produces, for every request shape, the feature vector the training path would have produced, and delivers it to the promoted artifact within the latency budget, or falls back visibly.
holds when Preprocessing is bundled with the artifact, or the stratified replay diff is zero on every deploy; the loaded hash matches the registry; the budget is tested at the production batch shape; fallbacks are marked and counted.
breaks when Either feature implementation changes independently; a handler reads version from config; the batch shape changes; a retry or timeout change makes the fallback the common path.
respond For a mismatch, fix the diverging side before the deploy proceeds — a retrain is the wrong tool. For a version mismatch, refuse to start.
1// The handler stamps every prediction with the version of what it actually loaded.2const artifact = await loadArtifact(process.env.MODEL_PATH!)3const loadedHash = sha256(artifact.bytes)4const promoted = await registry.promotedHash('recommender')5if (loadedHash !== promoted) {6 // refuse to serve a model the registry did not promote7 throw new Error(`loaded ${loadedHash} != promoted ${promoted}`)8}9 10app.post('/score', async (req, res) => {11 const features = await featureService.vector(req.body) // the path under test12 const scores = artifact.predict(features)13 if (!scores.every(Number.isFinite)) {14 return res.status(503).json({ fallback: 'popularity', reason: 'non-finite score' })15 }16 log.prediction({ version: artifact.version, hash: loadedHash, request: req.id })17 res.json({ version: artifact.version, scores })18})Two decisions carry the lesson: the startup refuses to serve an artifact the registry did not promote, and the non-finite branch returns a marked fallback rather than a cast. Both are the difference between an incident that can be attributed and one that cannot.
How to build it
Most important first.
- Bundle preprocessing with the artifact where possible, so the serving path and the training path execute one definition and the equivalence test degenerates to a version check (Preprocessing Lives in the Artifact).
- Where two implementations remain, keep a versioned replay fixture stratified by the rare values of every categorical and by missing-field patterns, and diff the vectors in the deploy pipeline with a mismatch budget of zero per feature.
- Assert the model version end to end: registry hash, loaded hash, logged version. Make the handler read the version from the artifact, not from a config.
- Test fallback explicitly by failing the model server in the test environment and asserting both the response and the fallback marker, then count fallback responses in production as a first-class metric.
- Run the latency test at the deployed batch shape on the deployed hardware, against the budget, and fail the deploy on a miss — a budget is a contract, not an aspiration (Throughput vs Latency).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-feature mismatch rate on the replay, which must be zero; any nonzero rate names the diverging feature before the first prediction.
- Loaded-artifact hash equals registry hash equals logged version, on every deploy, as a pass/fail.
- p99 latency at the deployed batch shape against the budget, in the deploy pipeline, and the fallback rate in production after it.
- A 200 response with a well-formed body is not a contract pass. It says the handler works; it says nothing about the number inside.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The serving path produces, for every request shape in the replay fixture including the rare ones, the same feature vector as the training path, to a tolerance that would not hide a default or unit change.
- The artifact the server has loaded is the promoted one, and every logged prediction carries that artifact's version.
- The fallback path is taken only when designed, is marked when taken, and its rate is visible; the latency budget holds at the production batch shape.
- On every deploy: the full contract suite against the deployed endpoint with the real artifact — schema, replay equivalence, version, output, latency, fallback — as a required gate.
- After deploy: the fallback rate and the per-feature serving distribution against training for the first day (Data & Feature Tests on the serving side).
- Over time: refresh the replay fixture from production monthly, re-stratified, so new rare paths enter the test.
What can go wrong
- The replay fixture is a uniform sample of last week and never contains a user with no purchase history; the default change passes the equivalence test because the branch is not exercised.
- The equivalence tolerance is set to absorb floating-point noise and also absorbs a unit change on a feature with small values.
- The version test compares the config's version to the registry and passes, because the config was updated and the artifact was not — the test must hash what is loaded.
- The fallback test passes, and in production the fallback rate is never charted, so a fifth of requests silently receive the popularity list for a month.
- A replay fixture with logged serving vectors is storage, a retention question and a privacy review; it is also the only way to test equivalence on real request shapes.
- A zero-mismatch budget fails the deploy whenever either implementation legitimately changes, which is correct and which trains people to raise the budget.
- Bundling preprocessing with the artifact simplifies the contract and constrains the serving stack to whatever can execute the bundle.
- "The model was unchanged, so the deploy could not have broken the model." The deployed function is the model plus the feature path plus the handler. A feature-service default change is a model change from the weights' point of view.
- "Health check green, latency normal, schema valid — the service is fine." All three are true of a service returning confidently wrong numbers. None of them looks inside the response.
- "We log the model version, so we can attribute regressions." Only if the logged version is read from the loaded artifact. A version from a config file attributes behaviour to whatever the config says.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALEvery deployed model has a request path whose feature vector must match training, and the replay equivalence test applies whatever the model or the serving stack; the specific components vary with whether preprocessing is bundled or separate.
- SCALE-SPECIFICA small team serving one model from a single process that also computes features in training has no second implementation and the equivalence test is a version check; the full contract suite earns its place once feature computation, model serving and request handling are separate services owned by different people.
- CONTESTEDWhether the deploy should be blocked on replay equivalence or the equivalence merely monitored is disputed. The strongest case for monitoring instead is that legitimate feature changes are frequent, that a blocking zero-mismatch gate turns into a rubber stamp within months, and that a canary with feature-distribution monitoring catches the same failures on a fraction of traffic; the strongest case for blocking is that the canary catches it after users have seen it and cannot name the feature, while the replay can.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a serving contract test is a consumer-driven contract test between two services, and the practice of keeping the fixture representative and the gate honest as both sides change is the testing discipline this lesson assumes.