Shadow Deployment
The candidate scores production inputs but controls nothing. It catches skew, latency and crashes before a user sees them — and it cannot measure business impact, because it never makes a decision.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What can a model running in shadow tell you that offline evaluation cannot, and what can it never tell you?
We promoted a new pricing model last year and it took the site down for an hour — a feature it needed was not in the serving cache. The offline numbers had been fine. Leadership now wants every candidate "tested against production" before it touches a user, without a repeat of last year.
Run the candidate in shadow for a week and compare its predictions against outcomes. If it predicts sale prices more accurately than the champion did on the same requests, it is better, and it has also proven it can run in production. Promote.
The outcomes were produced under the champion's suggestions. A seller who saw the champion's price and listed at it sold at roughly that price; the candidate's different suggestion was never shown, so its "accuracy" is measured against a world it did not influence. On a target the decision moves, shadow accuracy is not the accuracy it would have in control.
- The outcomes were produced under the champion's suggestions. A seller who saw the champion's price and listed at it sold at roughly that price; the candidate's different suggestion was never shown, so its "accuracy" is measured against a world it did not influence. On a target the decision moves, shadow accuracy is not the accuracy it would have in control.
- The candidate's new feature is served from a table that is refreshed nightly, while training computed it hourly. Shadow catches this — the candidate's feature distribution differs from training — but only if someone compares the shadow feature log to the training set, which the "accuracy" comparison does not do.
- The candidate's p99 latency is twice the champion's. In shadow nobody waits for it, so nobody notices, until it is on the request path.
- The shadow fork doubles feature-store load. The cache tier degrades, and it is the champion — the one users see — that starts timing out.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the price a listing will sell at, used to suggest a price to sellers. The label is the eventual sale price, known days to weeks later; a suggestion changes what the seller asks and therefore what the listing sells for.
- The decision — the suggested price — is made by the champion. The shadow candidate computes a price nobody sees.
- Each production request carries the seller's listing and the serving-time features fetched for it. In shadow, the same request is forked to the candidate, which fetches its own features — including any new ones — and logs a prediction.
- The shadow log holds, per request: champion score, candidate score, candidate latency, candidate errors, and the feature vector each saw. Outcomes arrive later and attach to the request, but they were produced under the champion's suggestion.
- The candidate depends on two features the champion does not use, and this is the first time they are fetched at request time.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Shadowing forks the production request to the candidate, which computes features and a prediction as it would in control, and logs them. The candidate's output is discarded; the champion's decision is returned. The candidate therefore experiences real inputs, real feature-fetch behaviour and real load, and has no effect on the world.
- That gives shadow two things offline evaluation cannot: the serving-time feature vector (so skew is directly observable, feature by feature, against training) and the serving-time behaviour (latency, errors, memory, dependency load). Anything that depends on the inputs and the code path is testable in shadow (Train / Serve Skew, Serving Contract Tests).
- It withholds one thing: the outcome of the candidate's decision. Outcomes in the shadow log were produced under the champion. Where the prediction does not move the outcome — an anomaly score that nobody acts on, a forecast of the weather — shadow accuracy is real. Where it does — a price, a recommendation, a fraud block — shadow can only say how the candidate would have scored a world it did not touch.
What the fork can see
The candidate in shadow lives the same life as the champion up to the moment of decision: the same request, its own feature fetches at request time, its own preprocessing, its own inference, all under real load. Everything about that path is observable — and every one of last year's failures lived in it.
The diagram marks where shadow stops. The candidate's prediction is logged and discarded; the champion's goes to the user; the outcome that comes back was produced by the champion's suggestion. The candidate is measured on the path, never on the world.
The skew check that needs no outcomes
The most valuable thing in the shadow log is not the candidate's score. It is the feature vector the candidate computed at request time, which can be compared to the distribution it was trained on the same day, before a single outcome exists. Last year's missing cache entry would have shown up as a feature that was null in every shadow request and rarely null in training.
The check is per feature, against the candidate's own training set — not against the champion's features, which may legitimately differ. A candidate with a new feature has never had that feature observed at serving time before; shadow is the first and only opportunity before users.
The candidate's shadow request path — feature sources, cache state, preprocessing version, timeouts — is the path it will have when it controls decisions.
holds when The shadow deployment uses the candidate's own serving image and feature configuration, fetches its features rather than reusing the champion's, and is measured against the production latency budget.
breaks when The shadow reuses the champion's feature vector for speed (so the candidate's new features are never fetched), runs with a longer timeout, or benefits from a cache the champion warmed that will be cold for its own features in control.
respond Fix the serving path before any canary. A skew found in shadow is the cheapest skew you will ever find; the same skew found in a canary has already priced listings wrong.
1async function scoreWithShadow(req: Request): Promise<Decision> {2 const decision = await champion.score(req) // the only thing the user waits for3 4 if (sampler.take(req)) {5 // fire-and-forget: a slow or failing candidate never delays the response6 void withTimeout(candidate.score(req), SERVING_BUDGET_MS)7 .then((c) => shadowLog.write({8 requestId: req.id,9 championScore: decision.score,10 candidateScore: c.score,11 candidateFeatures: c.featureRefs, // ids/hashes, compared to training later12 candidateLatencyMs: c.latencyMs,13 fellBackToDefaults: c.usedDefaults,14 }))15 .catch((err) => shadowLog.write({ requestId: req.id, candidateError: String(err) }))16 }17 return decision18}The timeout is the production budget, not a generous one: a candidate that would time out in control should time out in shadow and be counted. The sampler is what keeps a doubled feature-fetch load from degrading the champion.
The counterfactual shadow cannot see
When a sale price arrives for a shadowed listing, it is tempting to score both models against it. But the seller listed at the champion's suggested price, and buyers responded to that listing. Had the candidate's price been shown, the listing, the buyer response and the sale would all have been different. The candidate is being scored against an outcome it would have changed.
This is the difference between a prediction and an intervention, and it is why the offline/online gap for a decision-shaped outcome cannot be closed by any amount of shadowing. The only measurement of business impact is to let the candidate decide for some users and compare what happens (Offline vs Online Evaluation, A/B Testing Models).
In shadow, the candidate's suggested prices were closer to the eventual sale prices than the champion's, on the same listings.
On a canary where the candidate controlled the suggestion, listings sold slower and the median sale price fell; the champion's slice was unchanged.
- 1The candidate's prices were closer to sale prices that had been anchored by the champion's suggestion; shown to sellers, they anchored a different, lower asking price, and buyers took it.
- 2A price suggestion is an intervention; "accuracy against the sale price" in shadow rewards the model for predicting the champion's influence, not the market.
- 3Some genuine market softening may have coincided with the canary, but the champion slice would have shown it too and did not.
How to build it
Most important first.
- Define what shadow is for before running it: skew detection, latency and error budget, dependency load, crash-freedom. Write those as pass conditions. Accuracy against champion-produced outcomes is a secondary signal, valid only where the decision does not move the outcome.
- Compare the candidate's logged serving features against its training distribution per feature, on the first day. This is the check that would have caught last year's missing feature, and it needs no outcomes.
- Measure the candidate's latency and errors as if it were on the request path — same timeout, same budget — and fail the shadow if the budget is breached (Latency Breakdown).
- Fork asynchronously and rate-limit the shadow so its feature-fetch load cannot degrade the champion; sample if necessary. Then move to a canary, where the candidate's decisions are real and business impact can be measured (Canary Rollout).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-feature distribution distance between the candidate's shadow feature vectors and its training set. This is what shadow is for; a gap here is skew and blocks promotion.
- Candidate p99 latency and error rate under production load, against the serving budget. The second thing shadow is for.
- Agreement between candidate and champion scores — useful as a sanity check (a candidate that disagrees on everything is probably broken), not as quality. Outcome-based accuracy in shadow is a real measurement only where the decision does not shape the outcome.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The shadow candidate's request path is the one it will have in control — same feature sources, same timeouts, same preprocessing version — so that what shadow observes is what production will do.
- The shadow fork is isolated from the champion's latency and dependencies, so that testing the candidate cannot degrade the model users rely on.
- Where shadow accuracy is reported, the outcome is one the prediction does not move; where it is, the report says so.
- Offline: before shadowing, the candidate passes the contract tests — schema, feature ranges, preprocessing version pinned to training.
- Online, in shadow: per-feature skew against training on day one; latency and error budget over a representative week including peak; dependency load on the feature store within limits.
- Over time: when the candidate is promoted via canary, compare the canary's real outcomes against the shadow's outcome-based estimate. The gap is the size of the decision's effect on the outcome, and it calibrates how much to trust shadow accuracy next time.
What can go wrong
- The shadow log records the candidate's features but nobody compares them to training; the shadow "passes" because the candidate did not crash, and the skew ships.
- Shadow is run on a sample chosen for cheapness — daytime traffic, one region — and the crash is in the request shape the sample excluded.
- The candidate reads the same feature cache the champion warmed, so its feature fetches are fast in shadow and slow in control, where its new features are cold.
- Shadowing doubles inference and feature-fetch cost for the shadowed traffic, and the second copy has no user-facing value.
- It finds skew, latency and crashes, and the team that ran it will want it to have found quality too. It did not, and the temptation to promote on shadow "accuracy" is the main risk of running one.
- A shadow of a model whose decisions shape outcomes measures a counterfactual nobody can observe, and there is no fix for that within shadow — only a canary.
- "The shadow model was more accurate on production outcomes, so it is better." The outcomes were produced under the champion's decisions. On a price, a ranking or a block, the candidate's accuracy in shadow is against a world it did not make.
- "It ran in shadow for a week without errors, so it is safe." It ran without errors on the traffic it saw with a warm cache and no latency budget. Safe means skew-checked, budget-checked and load-tested, each explicitly.
- "Shadow is a slower canary." A canary makes decisions and measures their outcomes. Shadow makes none. They answer different questions and both are usually needed.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a candidate scoring real inputs without acting on them exposes skew and serving behaviour but not decision outcomes follows from what shadowing is, for any model whose prediction influences the outcome it is later scored against.
- TASK-SPECIFICFor a model whose output does not change the outcome — a demand forecast used for planning, a quality score nobody acts on per item — shadow accuracy against arriving outcomes is a real quality measurement; for pricing, ranking, fraud blocking or any intervention, it is not.
- SCALE-SPECIFICAt low request volume the doubled inference and feature-fetch cost is negligible and shadow can take all traffic; at high volume the fork must be sampled and rate-limited or it degrades the champion, and the sample must be stratified to include the rare request shapes.
Where the depth lives
This domain teaches the model and hands the rest off by name.