StrategiesPLATFORM-SPECIFICCLOUD-SPECIFIC

Blue/Green: Paying for the Fastest Rollback There Is

Two complete environments and a router between them: reversal in seconds, exposure of one hundred percent, and a database that is still shared.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

What exactly am I buying with a second environment, and what does it fail to protect me from?

The problem

Rolling back a rolling deploy takes as long as the deploy did. For changes where minutes of bad behaviour are unacceptable, you need reversal that is a routing decision rather than a redeployment.

What teams do first

Stand up the new version completely, test it, then point production at it. Because the old environment is still running, we can go back instantly — so this is the safest possible deploy.

How it breaks

The cutover is all-or-nothing. One second nobody is on v2, the next second everybody is. Blue/green reduces the *duration* of a bad state, never its *width* (Blast Radius: If This Is Wrong, How Much Does It Affect?).

How it breaks in production

The cutover is all-or-nothing. One second nobody is on v2, the next second everybody is. Blue/green reduces the *duration* of a bad state, never its *width* (Blast Radius: If This Is Wrong, How Much Does It Affect?).
The database is almost never duplicated, so everything the two environments share is exactly as shared as it was under a rolling deploy. A migration that v1 cannot tolerate breaks blue while green is being tested.
The reversal is only instant if the switch is instant. A DNS-based cutover is subject to resolver and client caching, so "point it back" can take as long as the record's time-to-live and longer for clients that ignore it (DNS in Production).
Data written by green during the exposure window stays after the switch back. You have reverted the code and not the consequences.
The idle environment rots. If green is only created at deploy time and blue is only kept for a day, the rollback target is an environment nobody has patched, scaled or exercised.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

Two environments run the same service at different versions. A router — a load balancer target group, a service selector, a DNS record, a weighted alias — decides which one receives production traffic.
The deploy is: build green, start it, verify it out of band, move the router, keep blue running as the rollback target, and eventually retire blue.
What is duplicated is the compute and the code. What is not duplicated is state: the database, the cache, the object store, the queue, the search index, external systems and anything a third party sees.
Reversal speed is therefore a property of the router, not of the strategy name. A target-group swap is effectively immediate for new connections; a DNS change is eventual and partially outside your control (DNS in Production).
In-flight requests on blue at the moment of the switch do not disappear — they must be drained, or the cutover drops them (Draining: Stopping Without Dropping).

What is duplicated, and what is emphatically not

The mental image of blue/green is two identical stacks. The real shape is two compute environments in front of one shared everything-else, and every hard problem in this lesson lives in the second column.

Two environments, one set of state

UserLLMAgentToolDataDecisionHumanGuardrail

The cutover and the reversal, with the parts that do not reverse

SIMULATEDAn Engineer Atlas reconstruction of the shape of a cutover, not a measurement from a real deploy. The ordering and the categories of what does and does not revert are the content; the minute marks are not.

The switch itself is the easy half. Read the last three rows carefully: the router went back, and the system did not return to its previous state.

Cutover to green, then back to blue

T-30mchangeGreen provisioned at full production size and started. Blue is serving everything.
T-10mactionGreen warmed: pools connected, caches primed, synthetic checks passing. This is evidence about green, not about the change.
T-2mchangeAdditive migration already applied — the schema now satisfies both blue and green (Expand, Migrate, Contract).
T+0changeRouter moved. Every request from this instant goes to green; blue keeps its in-flight work and drains.
T+30ssignalGreen is serving all traffic on caches that are warm for synthetic patterns and cold for real ones. Downstream read load rises.
T+4msignalError rate on a specific endpoint is elevated against the pre-cutover baseline (Canary Analysis: Compared Against What?).
T+5mactionDecision to revert. Router moved back to blue.
T+5m10srecoveryNew requests are on blue again. User-visible impact ends. This is the reversal people buy blue/green for.
T+6msignalRows green wrote in five minutes are still there, and blue is reading them. Any that use a shape only green understands are now blue's problem.
T+7msignalEmails green sent and webhooks it delivered during those five minutes cannot be recalled by any router (Roll Forward: When Going Back Is the Harder Option).

Durations are illustrative of ordering. The load-bearing rows are the last two: the router reverses traffic, and nothing reverses effects.

changesignalactionrecovery

Where the switch happens is where your rollback time comes from

PLATFORM-SPECIFICBehaviour of each layer varies with implementation: some load balancers finish in-flight requests on the removed target and some do not; DNS caching honours the record lifetime only as far as each resolver and client library chooses to. The general rule that survives every platform is that the further the switch is from your infrastructure, the less "instant rollback" means.

Every blue/green setup has a moment where one identifier stops meaning blue and starts meaning green. How quickly that propagates — and to whom — is the entire rollback guarantee, and it is a layer choice rather than a strategy choice.

Switch layer	How fast the change takes effect	Who can ignore it	What it costs you
Load balancer target group	New connections immediately; existing ones until they drain	Nothing — you own both ends	Both environments must be registered and health-checked continuously
Kubernetes Service selector	New connections immediately as endpoints are updated	Long-lived connections and clients that pin an address	Two full sets of pods running at once, and quota for both
DNS record	Eventually, bounded in theory by the record lifetime	Resolvers, client runtimes and libraries that cache beyond it	Rollback is eventual, which is the opposite of what you bought blue/green for (DNS in Production)
Serverless weighted alias	Per invocation, effectively immediately	Nothing, but in-flight invocations complete on the old version	Only available where the platform owns the version concept
Client-side configuration	When each client next fetches configuration	Any client that is offline, stale, or an old mobile build	Reversal time is set by your slowest client, which may be days

How to do it properly

Most important first.

Switch at a layer you control end-to-end. Prefer a load balancer or router change over DNS, precisely because DNS reversal is not under your control (DNS in Production).
Treat the shared database as the real constraint: the schema must satisfy both environments for the whole period blue remains your rollback target (Version Coexistence: N and N+1, in Both Directions).
Define, before the switch, how long blue stays alive and what would make you go back. "We will keep it around for a bit" is how the rollback target gets deleted an hour before you need it.
Exercise the reversal deliberately, outside an incident, so its duration is a measured number (Rollback: Only Useful If It Is Actually Safe).
Warm green before the switch — connection pools, caches, any lazily loaded state — or the cutover moves all traffic onto a cold system at once.
Combine with a canary if you can: route a small weight to green first, then complete the switch. Blue/green and canary are not exclusive (Progressive Delivery: Exposure as a Dial).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone

One testEveryone

What contains it

Not contained by width at all — the cutover exposes every user simultaneously. It is contained by *time*: the router can be moved back in seconds, so the exposure window is short even though it is total. That containment fails completely for anything green wrote.

What can go wrong

Failure modes, including of the mitigation

Green passes its out-of-band verification because it was tested with synthetic traffic, and fails on the traffic mix real users produce (Why Local Success Predicts So Little).
The cutover moves every user onto a cold environment simultaneously: caches empty, pools unestablished, downstream load spikes as every read misses (Operating a Cache).
Rollback restores the code and not the data. Rows, events and external side effects created by green persist, and blue may not understand all of them.
DNS-based reversal that is not a reversal: some resolvers and clients continue to send traffic to green for far longer than the record's stated lifetime.
The second environment is treated as a test environment and drifts from production configuration, so the thing you cut over to is not the thing you validated (Environment Drift).
Cost pressure kills blue immediately after the switch, deleting the rollback path at the exact moment it is most likely to be needed.

Misreads this invites

"Blue/green means zero-risk deploys." It means fast reversal. At cutover, one hundred percent of users move to code that no user has exercised.
"We have two environments, so we have two databases." Almost nobody duplicates production data, and the ones who do have a replication and reconciliation problem instead (Partial and Logical Data Recovery).
"Blue/green replaces canary." They control different variables — duration versus width. The strong setup uses both.
"The old environment is our rollback plan." Only while it is running, healthy, and compatible with the current schema. Otherwise it is an expensive idle server.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

Green served real production traffic successfully — a weighted slice before the full switch is far stronger evidence than any amount of testing against the idle environment.
Blue is confirmed still running, still healthy, and still compatible with the current schema. A rollback target that has not been health-checked is a hope.
The measured reversal time from a rehearsal, not the theoretical one.

How you get back

Point the router back at blue. Where the router is a load balancer or a service selector, this is the fastest reversal available in this module and it is why people pay for the second environment.
It does not undo writes. Anything green persisted — rows, queue messages, webhooks delivered, emails sent, charges made — survives the switch back (Roll Forward: When Going Back Is the Harder Option).
It does not undo schema changes. If green's migration removed something blue reads, switching the router back produces an outage rather than a recovery.
It is only as fast as the slowest client honouring the routing change, which is why DNS cutovers should be treated as eventual.

What to automate, and what stays human

Automate the whole sequence — provision, warm, verify, switch, hold, retire — so that the rollback target is retired by policy rather than by whoever remembers.
Automate the health gate on blue while it is the rollback target: an unhealthy rollback target should page someone before it is needed.
Keep the cutover decision human for significant changes. It is the moment exposure goes from zero to everyone, and that deserves a person.

What this costs

Roughly double the compute for the overlap period, and if you keep both environments permanently, permanently (Cost Drivers).
You buy reversal time and buy nothing at all in exposure. For a change whose failure is immediately visible, that is a great trade; for one that corrupts data quietly, it is close to worthless.
Two environments is two of everything to keep consistent — configuration, secrets, network policy, certificates — and every drift between them makes the rollback target less trustworthy.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

PLATFORM-SPECIFICHow the switch is made decides how fast reversal is. A load balancer target-group swap or a Kubernetes Service selector change affects new connections almost immediately; a DNS record change is eventual and partly controlled by resolvers and clients you do not own; a serverless weighted alias shifts per invocation. Same strategy, materially different rollback guarantees.
CLOUD-SPECIFICManaged platforms often ship a version of this as a product feature — a staging slot that is swapped into production — which handles warm-up and connection draining for you but hides which layer the switch happens at. Find out, because that layer is your rollback time.