The question this answers
How do I expose a new version to real production traffic while limiting the damage a bad one can do — and what has to exist before that is more than theatre?
Some classes of defect only appear under real traffic, real data and real concurrency. The release process must expose the new version to those conditions while capping the number of users affected if it is wrong.
A bounded exposure: a controlled fraction of real traffic on the new version, with a metrics-based decision to widen or to shift back, and a blast radius equal to that fraction rather than to the whole fleet.
The gradient, and what makes each step a decision
The mechanism is a weighted route. The gateway sends 5% of requests to the new version and 95% to the current one. Both are real production traffic against the real database. You wait — long enough to accumulate a meaningful sample — and compare error rate, latency percentiles and saturation between the two groups. If the new version looks the same or better, raise the weight: 5%, 25%, 50%, 100%. If it looks worse, set the weight to zero and the incident is over, having affected 5% of requests for ten minutes.
The word doing the work is "compare". A canary is not a threshold check; it is a comparison between two populations receiving the same traffic mix at the same time, which controls for everything a static threshold cannot — time of day, a traffic spike, a slow dependency. A 2% error rate might be normal for your service; a canary at 2% while the baseline sits at 0.4% is a clear signal that a fixed threshold at 5% would have missed entirely.
That is also why canary is the strategy with a real prerequisite. Every metric involved must carry a version label, the routing layer must be able to weight by version, and someone has to have decided in advance what "worse" means numerically. Teams that adopt canary without that end up watching a dashboard for ten minutes and clicking continue, which is a slow rolling deploy with extra infrastructure. This is the cross-link that matters: build Infrastructure Observability first.
Sizing the canary so the gate means something
A canary is a statistical instrument, and an undersized one decides nothing. If a service handles 50 requests per second and the canary takes 5%, that is 2.5 requests per second — 150 requests in a minute. An error rate that rises from 0.4% to 1.5% is invisible in that sample. The gate passes, the weight goes to 100%, and the regression arrives at full traffic having been formally approved.
So the fraction and the observation window are a joint decision driven by traffic volume and by the size of the regression you want to catch. High-traffic services can use 1% and five minutes. Low-traffic services need a larger fraction, a longer window, or an honest admission that canary is not adding much and rolling with good alerting is the better fit.
Two refinements are worth knowing. Sticky routing — consistently sending the same user to the same version — avoids the confusing experience of a user seeing new behaviour and then old behaviour, at the cost of a less random sample. And a "dark canary" or shadow deployment sends a copy of real traffic to the new version without using its responses, which tests the code path with zero user impact and cannot test anything involving writes without a great deal of care.
- 1Deploy canary instances
A small number of instances on the promoted digest, registered with the gateway at weight 0.
Canary instances that differ from stable in anything but version — a different configuration invalidates the comparison.
- 2Weight 5%10 minutes — ILLUSTRATIVE
Real traffic begins. Metrics accumulate per version.
Too small a sample at low traffic: the gate passes on noise.
- 3Gate: compare
Error rate, p99 latency and saturation for canary versus stable over the same window.
Comparing against a static threshold instead of against stable, which misses regressions that stay inside the threshold.
- 4Weight 25%15 minutes — ILLUSTRATIVE
Wider exposure; slower-burning problems — memory growth, connection leaks — become visible.
Rushing past this step. Leaks need time, not traffic.
- 5Weight 50%
Half of traffic. Capacity effects and shared-dependency pressure now show.
Database connection count from two full-sized fleets, which is a canary-specific failure.
- 6Weight 100%
The canary becomes stable; old instances are drained and retired.
Retiring the old version before the observation window on the full weight has elapsed.
- 7Abort: weight 0
Traffic returns to stable within seconds. The canary instances stay for diagnosis.
Destroying the canary instances on abort, discarding the evidence that would explain what went wrong.
What a canary cannot catch
A canary is excellent at finding defects that show up in aggregate signals on the request path: an exception on a common code path, a query that got slower, a memory leak, a dependency that is now called twice per request. It is poor at almost everything else, and knowing the gap keeps the practice honest.
It cannot catch a problem that only appears at full traffic — a connection pool that is fine at 5% and exhausts a shared database at 100%, a cache hit rate that only degrades when the whole fleet is on the new version, a rate limit at a third-party API that a fraction of traffic never approaches. It cannot catch data corruption, because both versions write to the same database and the canary's writes are real: a version that writes bad rows has written 5% of them, permanently, whatever the gate decides afterwards. And it cannot catch anything whose effect is delayed past the observation window — a nightly job, a subscription renewal, a monthly report.
The data point deserves emphasis because it is the one people are surprised by. Canary limits how many *requests* are affected. It does not limit how much *state* is corrupted, and there is no weight setting that makes a write reversible.
| Failure class | Caught? | Why |
|---|---|---|
| Exception on a common path | Yes, quickly | Error rate diverges within the first window. |
| Latency regression | Yes | Percentile comparison against stable under identical traffic. |
| Memory or connection leak | Only with time at weight | Needs duration, not traffic share — do not rush the middle steps. |
| Capacity or pool exhaustion at full scale | No | The failure requires 100% of traffic to manifest. |
| Data corruption | No | The canary's writes are real writes. 5% of the damage is already permanent. |
| Rare-path defect | Unlikely | A path taken by 0.1% of requests, times a 5% canary, is not in the sample. |
| Delayed effects (nightly jobs, renewals) | No | Outside any plausible observation window. |
Key points
- A canary routes a small weighted fraction of real traffic to the new version and compares it against the stable version under identical conditions.
- Comparison against stable beats a static threshold, because it controls for time of day, traffic mix and dependency behaviour.
- The prerequisite is per-version metrics and a pre-agreed definition of "worse". Without those it is a slow rolling deploy.
- Size the fraction and window against traffic volume: a low-traffic canary decides nothing and approves the regression it was meant to catch.
- It bounds affected requests, not corrupted state — the canary's writes are real, and 5% of bad rows are permanent.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Canary instances are deployed on the promoted artifact and registered with the routing layer at weight zero.
- • The gateway, mesh or load balancer assigns a weighted share of requests to the canary target set.
- • Metrics, logs and traces carry a version label so the two populations can be compared over the same window.
- • An automated analysis or a human compares error rate, latency percentiles and saturation between canary and stable.
- • Weight is increased through a defined sequence, or set to zero to abort — both are routing changes taking effect in seconds.
- • You own the routing layer, permanently, whether or not a deploy is in progress.
- • You own the metric labelling discipline, without which nothing here works.
- • You own the analysis definition: which metrics, what threshold of difference, over what window, and whether it acts automatically.
- • You own the weight schedule and the patience to honour it — leaks need time at weight, not more traffic.
- • You own what happens to canary instances on abort: keep them, do not serve from them, and read their logs.
- • A canary too small to detect the regression, which then reaches 100% with formal approval.
- • An analysis comparing against a static threshold rather than against stable, missing regressions that stay inside the threshold.
- • Metrics without a version label, so the canary's errors are averaged into the aggregate and disappear.
- • A canary that differs from stable in configuration as well as version, so the comparison measures the wrong difference.
- • Data written by the canary in a format stable cannot read, discovered after the abort.
- • A routing layer misconfiguration sending 100% to the canary — the failure mode unique to the strategy that was supposed to limit exposure.
- • Automated rollback triggering on a coincidental dependency incident, which erodes trust in the gate and leads to people disabling it.
- • Statistical power scales with total traffic: high-volume services can canary at 1% for five minutes; low-volume services cannot canary meaningfully at all without long windows.
- • Metric cardinality scales with series count times versions in flight, and is the observability cost that surprises teams.
- • Shared dependency pressure scales with total instances, not with weight — at 50/50 you have two full fleets connected to one database.
- • Analysis complexity scales with the number of services doing this independently; a shared progressive-delivery controller becomes worth it around a dozen services.
- • The routing layer becomes security-relevant: whoever can set weights can direct production traffic to an arbitrary version.
- • A canary is the right way to roll out a risky security change, because it bounds the population affected if the change breaks authentication or authorisation.
- • It is the wrong way to roll out an urgent security *fix*: the fix is only in effect for the canary fraction while the vulnerable version keeps serving the rest.
- • Canary instances hold the same production credentials as stable, so an unverified version briefly has full workload identity.
- • The compute cost is small — a few extra instances during the rollout.
- • The permanent costs are the routing layer and the per-version metric cardinality, both paid whether or not you are deploying.
- • A long rollout schedule has an opportunity cost: an hour of gradual promotion is an hour during which two versions are live.
- • Against that: the cost avoided is error rate times traffic times time, and canary is the strategy that minimises the traffic term.
- • Error rate, latency percentiles and saturation per version, over the same window — this is the whole instrument.
- • Business metrics per version where they exist: conversion, checkout completion, signup rate. A technically healthy version that halves conversion is a failed release.
- • Sample size per canary step, so you know whether the gate had power or just passed.
- • Routing weights as an observable, since a misconfigured weight is a silent, total loss of the strategy.
- • The signal that lies: aggregate error rate. A 5% canary failing every request moves the aggregate by five percent, which is inside normal variation on most dashboards.
- • Rolling with good alerting and someone watching. For most services this catches the same defects a few minutes later and costs nothing extra.
- • Blue/green, when rollback speed matters more than limiting exposure, and you can afford double capacity instead of a routing layer.
- • Feature flags with percentage targeting, which gives the same gradual exposure at the behaviour level without a second version deployed — often the better tool when the risk is a behaviour change. See Deployment Is Not Release.
- • Shadow traffic, when you want to exercise the new code path with zero user impact — at the cost of being unable to test writes without significant care.
- • For a low-traffic service, no canary. The sample will not support a decision, and pretending otherwise adds infrastructure and false confidence.
- • Buys the smallest blast radius of the four strategies; costs a routing layer, metric cardinality and a slower rollout.
- • Buys evidence from real traffic; costs real consequences — the canary's writes and side effects are not simulated.
- • Automated analysis buys fast, consistent decisions; costs false positives that erode trust in the gate until someone turns it off.
What people believe, and what is true
A canary limits the damage of a bad release.
It limits the number of *requests* affected. Writes made by the canary are real and permanent, so it does not bound state damage at all.
If the canary's error rate is below the alert threshold, it is fine.
Compare against stable, not against a threshold. A canary at 2% against a 0.4% baseline is a clear regression that no reasonable static threshold catches.
Canary is the mature choice, so we should adopt it.
Without per-version metrics it is a slow rolling deploy with extra infrastructure and false confidence. The observability is the prerequisite, not the follow-up.