Canary: One Percent, Then Five, Then Watch
Exposing a small share of real traffic to the new version and widening only while health holds — the strategy that bounds width rather than duration.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do I let real production traffic find the bug without letting it find every user?
Some failures only appear under real traffic: the request mix you cannot synthesise, the data shapes you did not anticipate, the concurrency you cannot reproduce. You have to run the new code in production to find them, and you would prefer not to do that to everybody at once.
Deploy to one instance first, look at the dashboard, and if nothing looks wrong, deploy the rest.
One instance out of many may receive a share of traffic too small for any signal to be distinguishable from ordinary variation, so "nothing looks wrong" means "nothing is measurable".
- One instance out of many may receive a share of traffic too small for any signal to be distinguishable from ordinary variation, so "nothing looks wrong" means "nothing is measurable".
- The dashboard aggregates all instances. The canary's errors are averaged into the fleet, diluted by the instances running the old code, and disappear (Canary Analysis: Compared Against What?).
- A newly started instance has cold caches and empty pools, so it looks worse than the fleet for reasons unrelated to the change — and teams learn to discount exactly the signal they set the canary up to read.
- "Looks fine" is not a defined stopping condition, so canaries get promoted because the deploy is taking a while and the person watching has another meeting.
- If the change is a schema, config or shared-state change, the canary is exposing one percent of traffic to new code and one hundred percent of data to the new behaviour.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A canary is a small, deliberately chosen share of *real* production traffic sent to the new version, with a defined observation window and a defined decision at the end of it.
- The standard ramp is 1% → 5% → 25% → 50% → 100%, with health observed at each step before the next one begins. The multiplication matters: each step is large enough to produce a readable signal and small enough that the previous step would have caught a gross failure.
- What the small percentage buys is arithmetic. If the candidate fails every request it serves, at 1% you have broken one percent of requests rather than all of them. The bug is fully present; the exposure is not.
- Traffic can be split per request, per session, or per tenant, and the choice decides who experiences the failure and whether they experience it consistently.
- The step size is bounded below by signal, not by caution. Below the traffic volume at which a difference is readable, a smaller canary is not a safer canary — it is a blinder one (Canary Analysis: Compared Against What?).
The ramp, and what each step is for
The steps are not arbitrary caution. Each one has a job: the first proves the candidate can serve at all, the middle ones make the signal readable, and the last ones expose load-dependent behaviour that a small slice cannot produce.
- 11%
Proves the candidate serves real requests at all: it starts, connects, parses real payloads and returns real responses.
fails by Too little traffic for anything subtle to be visible — and on a low-traffic service, too little for anything at all.
evidence Non-zero successful requests on the candidate, and no crash loop.
- 25%
Produces enough volume for an error-rate comparison against the concurrent baseline to mean something.
fails by Rare endpoints still unexercised; the slice may miss the expensive paths entirely.
evidence Candidate error rate indistinguishable from baseline over a full observation window.
- 325%
Exercises the long tail of request shapes, and starts to load shared dependencies measurably.
fails by Downstream effects appear here that were invisible below — connection budgets, cache miss ratios (The Connection Budget).
evidence Latency distribution and downstream load compared, not just the service's own errors.
- 450%
Reveals anything that only appears when the candidate carries serious load: memory growth, pool exhaustion, throttling.
fails by At half traffic, a failure is no longer a small blast radius — this step is where you should already be confident.
evidence Saturation signals stable; no upward drift in memory or queue depth over the window.
- 5100%
Completes the rollout; the old version is retired and stops being an instant rollback target.
fails by The mixed window closes, and with it the option to reverse by routing weight rather than by redeploying.
evidence Full-traffic behaviour matches the 50% step, held for longer than one cache lifetime (A Successful Deploy Is Not Evidence of a Healthy System).
The ramp multiplies rather than adds because readability grows with volume. Going 1% → 2% → 3% costs the same wall-clock time and never reaches a step where load-dependent failures can appear.
How you slice the traffic decides who gets hurt
One percent of requests, one percent of sessions and one percent of tenants are three different one percents, and they fail differently. Per-request splitting spreads a small amount of pain over many users; per-session concentrates all of it on few. Neither is right in general.
Which slice of traffic goes to the candidate?
when The service is stateless and each request stands alone. Gives the most representative traffic mix for the smallest weight.
cost A single user can hit both versions in one workflow, so any inconsistency between the versions becomes visible to them (Version Coexistence: N and N+1, in Both Directions).
when The user experience spans requests, and inconsistency mid-flow would be worse than the bug you are hunting.
cost A much smaller number of users absorb all of the failure, and the same users may be chosen every time unless you rotate.
when A business-to-business product, where blast radius is naturally measured in accounts and where some accounts must never be a canary.
cost Tenants differ enormously in traffic and usage patterns, so the slice is representative of that tenant and not of your traffic.
when You want real usage with a cohort that will tell you rather than churn. A reasonable first step before any external percentage.
cost Internal usage is not representative — it is a friendlier traffic mix and a more forgiving audience, so passing means less than it seems.
when The change interacts with infrastructure — networking, storage, capacity — where a code-level slice would not isolate anything (Reducing Blast Radius).
cost The smallest slice is one zone's worth of traffic, which is a much larger step than one percent.
Canaries that prove nothing
The failure mode of a canary is rarely that it fails to stop a bad release loudly. It is that it passes, tells you nothing, and converts an unverified change into a verified-feeling one.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Weight set to 1%, no version label on metrics | Dashboard looks normal throughout the ramp | The candidate's errors are averaged into 99% of healthy traffic and vanish below the noise | Emit and query by version label; compare candidate to baseline as two series, never as one (Canary Analysis: Compared Against What?) |
| Canary instance just started | Latency far above the fleet in the first minutes | Cold caches, unestablished pools, runtime warm-up — nothing to do with the change | Exclude a defined warm-up interval from comparison, and know how long that interval actually is |
| Sticky routing by client address | Canary serves a tiny, unrepresentative set of users | The slice was chosen by a key that correlates with usage patterns | Split on a key uncorrelated with behaviour, and verify the endpoint mix on the candidate matches the baseline |
| Change is a database migration | Canary is green; every user is affected anyway | The schema is not traffic-routed — one percent of requests does not mean one percent of the data | Treat migration and code as separately staged changes (A Migration and a Deploy Are One Event) |
| Observation window shorter than the failure | Canary passes, failure appears at full traffic hours later | Memory growth, connection leaks and cache-driven failures need time, not traffic share | Hold the highest pre-full step across at least one full cycle of the slowest relevant process (A Successful Deploy Is Not Evidence of a Healthy System) |
| Nobody defined the abort condition | Ramp continues through an ambiguous signal | Promotion by absence of an obvious disaster rather than by a criterion | Write the abort condition before the ramp; make abort the default when the window is unclear |
How to do it properly
Most important first.
- Ramp in steps with an observation window at each: 1%, 5%, 25%, 50%, 100%. Do not skip from a small percentage straight to everything, which is the step where most canaries fail to catch anything.
- Make the window long enough to include the behaviour you care about — a cache cycle, a scheduled job, a background flush. A two-minute window catches crashes and nothing slower.
- Compare the candidate against a concurrent baseline of the old version, never against the fleet aggregate and never against yesterday (Canary Analysis: Compared Against What?).
- Exclude the warm-up period from the comparison, and know how long it is, or the first minutes of every canary will look like a regression.
- Define the abort condition before starting, and make aborting the default action when the window is ambiguous.
- For anything that touches shared state, canary the code and treat the state change as a separate, unprotected deploy — because that is what it is (A Migration and a Deploy Are One Event).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by the traffic weight at the current step and by the abort condition that ends it — genuinely one percent of requests while the step holds. The containment applies only to the code path: shared state, schema and external side effects escape it entirely.
What can go wrong
- A canary carrying no traffic: routing weight applied to a version that has no healthy instances, or a sticky-session scheme that keeps sending the same handful of users to it.
- Traffic that is not representative — health-check requests, internal callers, or a slice that happens to exclude the expensive endpoints.
- Cold-start effects read as regressions, which trains the team to ignore the first minutes and then to ignore the whole window.
- A canary held at 1% for hours, so a mixed fleet becomes the steady state and the compatibility obligations of a rolling deploy apply indefinitely (Version Coexistence: N and N+1, in Both Directions).
- The same users always chosen as the canary cohort — which is fine for consistency and unfair if they are always the ones who see the breakage.
- Promotion by timeout: nothing failed loudly, so the ramp continued, though nobody could say what would have stopped it.
- "The canary passed, so the release is safe." The canary showed that the change is not catastrophically wrong under the traffic mix and duration you sampled. Slow failures, rare paths and load-dependent effects are all still ahead of you.
- "Smaller is safer." Smaller is narrower and blinder. Past the point where the signal is unreadable, a smaller canary only means you find out later, with more users affected.
- "Canary protects against bad migrations." It protects the code path. The schema is not routed, and one percent of traffic on new code does not mean one percent of rows changed shape.
- "We deployed to one instance, so we did a canary." Only if that instance receives representative traffic and is observed separately. Otherwise it is a rolling deploy with a pause in it.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The candidate's error rate, latency distribution and saturation are readable *separately* from the baseline's, by version label (Deploys on the Same Timeline as the Symptom).
- The canary actually served a meaningful number of real requests across the endpoints that matter — verified, not assumed.
- At least one business-level signal moved as expected or did not move at all: a successful checkout, a completed job, a delivered message (Canary Analysis: Compared Against What?).
- Set the current step to zero percent. Only the users on the current step were exposed, and the reversal is a routing change rather than a redeploy.
- That is the fastest and narrowest reversal in the module — provided the change was confined to the code path.
- It does not undo what the candidate wrote. One percent of users' data written in a new shape is still there afterwards, and the old version must be able to read it (Version Coexistence: N and N+1, in Both Directions).
- If the candidate emitted external side effects — webhooks, emails, charges — those one percent are as irreversible as they would be at one hundred (Roll Forward: When Going Back Is the Harder Option).
- Automate the ramp, the observation window, and the abort. A rollout that pauses on a failing comparison and reverses itself is one of the highest-value automations in this domain.
- Automate the comparison against the concurrent baseline, so the decision is made against a defined query rather than against whoever is looking at the dashboard.
- Keep the choice to override an abort human, and require a reason. Automatic abort with human override is the right asymmetry; the reverse is not.
- Rollouts become long. A five-step ramp with real observation windows turns minutes into a substantial part of a working day, and long rollouts collide with other changes (Change Correlation).
- It requires traffic-splitting infrastructure and per-version telemetry, both of which are systems you now operate.
- It requires traffic. Below a certain request rate, a one percent slice cannot produce a readable signal in any reasonable window, and the whole apparatus is ceremony (Canary Analysis: Compared Against What?).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- SCALE-SPECIFICCanary needs enough traffic that a small slice is statistically legible within the observation window. On a service handling a few requests a minute, one percent is not a sample, and the honest strategies are feature flags for specific users or a small blue/green cutover with a fast reversal.
- PLATFORM-SPECIFICWeighted traffic splitting comes free on a serverless platform with version aliases and on most managed app platforms. On Kubernetes it requires an ingress controller, a service mesh or a rollout controller. On a plain VM autoscaling group, exposure is quantised by instance count — with ten instances your smallest step is ten percent, no matter what your plan says.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.