Canary Rollout
Give the candidate 1% of decisions, then 5%, 25%, 100%, watching quality, latency, cost and errors at each step — with the honesty that a 30-day label makes a 30-day canary.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How do you roll a model out so that a mistake hurts a few users instead of all of them, and what do you watch at each step when the real outcome will not arrive for a month?
Our recommendations model went to 100% in one deploy and a bad artifact showed everyone the same ten items for six hours. We want a staged rollout, but the metric we care about — 30-day retention — will not tell us anything for a month, and we cannot sit at 1% for a month per release.
Put the candidate on 1% of traffic for an hour, check the error rate and latency dashboards, then go to 100%. If retention looks bad in a month, roll back. A canary is a deploy-safety mechanism; the model's quality was already validated offline.
The hour at 1% catches crashes and latency, and nothing else. The bad artifact that showed everyone ten items had normal error rates and fast latency; it was wrong, not broken.
- The hour at 1% catches crashes and latency, and nothing else. The bad artifact that showed everyone ten items had normal error rates and fast latency; it was wrong, not broken.
- By the time 30-day retention reports the candidate's slice is worse, it has been at 100% for four weeks, the previous artifact is cold, and the users who churned are gone.
- Assignment is per request rather than per user. A user refreshes and sees a different model's feed; the feed changes under them; the session-length proxy is polluted in both arms.
- The candidate is cheaper per prediction on paper and more expensive in practice because its features are cold in the cache for the 1% slice. Cost per prediction at 1% is not cost at 100%.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Rank items for a user's home feed to maximise 30-day retention. The label is whether the user is still active a month later; the short-term proxies are session length and click-through, which the model can move without moving retention.
- The decision is the ranked feed itself, so the candidate at 1% controls the feed for 1% of users — real decisions with real, delayed outcomes.
- Every request carries a user id, which is what the canary assigns on: a user is in the canary or not, stably, for the whole rollout. Per-request random assignment would show the same user two models on two refreshes.
- The canary log records, per request: assignment arm, model version, prediction, decision, latency, errors, and — later — the retention outcome joined by user.
- The leading indicators available within hours are session length, click-through, feed diversity and the fraction of feeds that fell back to a default ranking.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A canary is a controlled comparison run in stages. At each stage, a stable fraction of the decision-making population is assigned to the candidate; the rest stays on the champion. Because assignment is random and stable, the two arms are comparable populations experiencing two policies, and any difference in their measured outcomes is attributable to the model — subject to sample size.
- The stages exist to bound harm: at 1% a wrong model hurts 1% of users while the leading indicators are checked; at 5% and 25% the sample is large enough for smaller effects and rarer slices to show; at 100% the champion is retired but kept warm. Each promotion step is a decision that the previous step's signals justify the next exposure.
- The signals available at each step depend on their delay. Errors, latency and cost are immediate. Prediction distribution and fallback rate are immediate and model-specific. Proxy outcomes — clicks, session length — arrive within a session. The true outcome arrives after the label delay, and for a 30-day label, no stage shorter than 30 days can measure it (Ground-Truth Delay).
Four steps, four signal groups
Each step of the ladder is an exposure and a set of pass conditions. The exposure bounds how many users a wrong model touches; the pass conditions are what must be true of the candidate arm, relative to the champion arm, before the next exposure. The signals split by delay: some are available in minutes, some in a session, and the one the rollout is for arrives after the label delay.
The pipeline below is a rollout for a model with a 30-day outcome. Notice that no step waits for retention; every step is gated on what is measurable within the step, and the holdback after 100% is where retention is measured.
- 11% — the smoke step
Stable user-hash assignment. Watch errors, p99 latency, cost per prediction, fallback rate and prediction distribution against the champion arm for a few hours.
fails by Only errors and latency are watched; a wrong-but-healthy model passes. Or the assignment is per request and the proxy is polluted.
- 25% — the proxy step
Hold for a day or more. Session length, click-through and feed diversity per arm, with intervals. Stop on a collapse in diversity or a fallback spike; do not accelerate on a proxy improvement.
fails by A click-through gain is read as success and the rollout accelerates toward a model that learned clickbait.
- 325% — the slice step
Enough sample for rare slices — new users, low-activity users, each region — to be compared per arm. Cost at scale and cache behaviour become visible.
fails by Slices are not checked and the regression lives in the 3% of users who are new.
- 4100% with holdback
Champion retired from the main path, kept warm; a small holdback stays on it. Retention at 30 days is compared between the candidate arm and the holdback when it arrives.
fails by No holdback, so the delayed outcome has no comparison; or the champion is cold and rollback is a redeploy.
The DevOps canary practice supplies the mechanics — traffic splitting, automated analysis, promotion gates. What the model adds is the prediction-distribution and fallback signals in step one, the proxy honesty in step two, and the holdback in step four.
Stable assignment
A canary compares two populations, and the comparison is only valid if each user is in one arm for the duration. Per-request assignment gives every user a mixture of both models, the arms become indistinguishable, and any per-user proxy — session length, retention — is measured on a population that was in both arms.
The key is a hash of the user id and the rollout id. The rollout id matters: without it, every canary lands on the same users, who have by now seen every candidate first, and whose behaviour is not representative of anyone else.
1function arm(userId: string, rollout: { id: string; candidatePct: number }): 'candidate' | 'champion' {2 // hash the user together with the rollout id so this rollout's 1% is not last rollout's 1%3 const bucket = hash32(`${rollout.id}:${userId}`) % 10_0004 return bucket < rollout.candidatePct * 100 ? 'candidate' : 'champion'5}6 7// promoting from 1% to 5% keeps every user who was already in the candidate arm:8// buckets 0..99 stay in, buckets 100..499 join — nobody moves back.The last comment is the property that makes stepping up safe: increasing the percentage adds users to the candidate arm without moving any user out of it. A user who was in the canary at 1% is still in it at 25%, so their outcome is attributable to a single model.
The delayed label and the honest proxy
Retention at 30 days is the goal, and a canary gated on it would be a 30-day step. The way out is not to pretend a proxy is the goal; it is to use the proxy for what it can do — detect harm quickly — and to keep a holdback so the goal can still be measured when the label arrives.
The assumption underneath is that the proxy is at least harmless-when-flat: a candidate that leaves session length and diversity where the champion had them has not broken anything obvious. Whether it improved retention is a separate question the holdback answers a month later.
A collapse in the leading indicators — session length, click-through, feed diversity, fallback rate — reliably signals a harmful candidate, even though an improvement in them does not signal a good one.
holds when The proxies are measured per arm with stable assignment; the pass conditions are written as "no worse than the champion arm" with an interval; diversity and fallback rate are included so a degenerate feed (ten items for everyone) is caught by its shape, not by clicks.
breaks when The candidate finds a way to raise clicks while lowering retention; the proxies are compared without intervals so noise is read as signal; the degenerate-feed failure is one the proxies do not cover.
respond Roll back to the warm champion — a routing change, not a redeploy — and add the missing shape signal to the step-one pass conditions for the next rollout.
How to build it
Most important first.
- Assign by a stable key — user, account, session — hashed with the rollout id, so a user's arm does not change on refresh and does not correlate with a previous rollout's assignment.
- At every step, watch four groups of signals with pass conditions written before the rollout: errors and latency against the serving budget; cost per prediction against the champion; prediction distribution and fallback rate against the champion's; and the leading indicators against the champion's arm (Model Monitoring).
- Treat the leading indicators honestly. Clicks and session length are proxies that the model can move without moving retention; use them to detect harm (a collapse in diversity, a spike in fallbacks) and to stop, not to declare victory (Business Metrics vs Model Metrics).
- Keep a holdback. Leave a small fraction of users on the champion after 100% so the delayed outcome can still be compared when it arrives, and keep the champion artifact warm so rollback is a routing change (Rollback & Fallback). Cross-check the rollout mechanics with the DevOps canary practice — the model adds signals; it does not change the shape of a progressive delivery.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- At each step: candidate-arm versus champion-arm on errors, p99 latency, cost per prediction, fallback rate, prediction distribution and feed diversity. These are the stop signals, and they are immediate.
- The leading proxy — session length, click-through — as a difference between arms with an interval. It detects harm within hours; it does not measure the goal.
- Retention at 30 days, candidate arm versus holdback, is the number the rollout was for. It arrives after the rollout ends, and the holdback is what makes it measurable at all.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Assignment is random with respect to the outcome and stable per user, so the two arms differ only in the model.
- The leading indicators are, for this model, harmless-when-flat: a candidate that keeps them flat has not been shown to be good, but a candidate that collapses them has been shown to be bad.
- The champion artifact and its feature path remain deployable and warm for the length of the label delay after 100%, so the delayed outcome can still trigger a rollback.
- Offline: before 1%, the candidate has passed shadow — skew, latency budget, load — so the canary is testing decisions, not the serving path.
- Online: at each step, the four signal groups against their pass conditions, per arm, with the step held long enough for the proxies to have an interval that excludes harm.
- Over time: the 30-day retention comparison between the candidate arm and the holdback, recorded against the rollout so that the next rollout's proxy thresholds can be calibrated against what the proxy actually predicted.
What can go wrong
- The rollout id is not in the hash, so the 1% slice is the same 1% of users that every previous canary landed on; they have seen every candidate first and their behaviour is not representative.
- The leading indicator improves — clicks are up — and the rollout accelerates. Retention at 30 days is down: the candidate learned clickbait (Feedback Loops).
- The canary passes at 25% and fails at 100% because the feature store's cache was sized for the champion's access pattern; at full traffic the candidate's features evict the champion's and both degrade.
- A staged rollout delays full deployment by the sum of its stages, and a stage that is long enough for proxies to be conclusive is a slower release cadence.
- A holdback keeps some users on the old model deliberately, for the length of the label delay — a cost paid to make the true outcome measurable.
- Watching four signal groups per step per arm is a dashboard and an alerting policy per model, and each pass condition is a threshold someone has to choose and defend.
- "The canary passed — errors and latency were fine at 1%." Errors and latency say the service is up. A model that is wrong is up. The canary has to watch the prediction distribution and the leading indicators too, or it is a deploy check.
- "Clicks are up in the canary, so retention will be up." Clicks are a proxy that the model can move without moving retention. Up is a reason not to stop; it is not a reason to declare success before the label arrives.
- "We can't wait 30 days per stage, so the delayed metric is impractical." You cannot wait 30 days per stage. You can keep a holdback and measure it at 30 days after the rollout finishes, and roll back then if needed.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALStaged exposure with stable assignment and per-step pass conditions applies to any model that makes decisions; the specific signals and their delays differ by task, and the label delay sets how much of the rollout can be outcome-driven.
- DOMAIN-SPECIFICIn ads or search the true outcome — a click, a purchase in-session — arrives within the canary and the rollout can be outcome-gated; for retention, credit or medical outcomes the canary is proxy-gated and the holdback carries the real comparison weeks later.
- SIMPLIFIEDThe 1% → 5% → 25% → 100% ladder is illustrative; real ladders depend on traffic volume, the minimum detectable effect on the proxies and the harm a wrong model does per user, and any step percentages quoted here are for the shape of the argument.
Where the depth lives
This domain teaches the model and hands the rest off by name.