CI/CD & Deployment

Blue/Green: Two Environments, One Switch

Blue is what is serving. Green is the new version, fully built and verified while nobody is looking. The switch is a load balancer change and rollback is the same change in reverse — which is why it is the fastest rollback there is, and why the database, which does not switch, is the part that ruins it.

The question this answers

Infrastructure question

How do I get a rollback measured in seconds, and what does the database do while the application environment is being swapped underneath it?

Application requirement

A bad release must stop serving within seconds, not within the time it takes to roll a fleet back — for a payment flow where every minute of elevated errors is measurable revenue.

What it provides

A complete, verified second environment and an atomic-looking traffic cutover, so the previous known-good environment stays intact and reachable for as long as you choose to keep it.

Application RequirementInfrastructure RequirementComputeNetworkStorageIdentityDeploymentScalingReliabilityObservabilitySecurityCostTrade-offs

The switch is the point; the second environment is the price

Provision a full copy of the application tier running the new version. Nothing routes to it. Run smoke tests against it through an internal address, check that it can reach its dependencies, warm its caches and connection pools. Then change one thing — the load balancer's target set, or a weighted route — and all traffic moves. If the new version misbehaves, change that one thing back. The old environment is still running, still warm, still holding its connections.

That is the entire value proposition, and it is a good one: rollback is a routing change rather than a deployment. Compare with rolling, where rolling back means another rollout with the same duration and the same mixed window. For a workload where minutes of errors are expensive, the difference is worth paying for.

What you pay is capacity. Two full application environments exist simultaneously, so for the overlap period you are running roughly double. On a modest fleet for twenty minutes that is negligible. On a very large fleet it is a real number and it is also a quota question — you need the headroom to exist at all, which is a common and unglamorous reason blue/green fails on its first attempt.

Mid-cutover. Note which components are duplicated and which are emphatically not.ILLUSTRATIVE
Clientspublic
Load balancerpublic— the switch lives here: one target-set change moves 100% of traffic
Blue: app v1.3 ×6private— currently serving; kept running for the rollback window
Green: app v1.4 ×6private— verified through an internal address before the switch
Postgres (shared)private
Cache (shared)private
Secret storeprivate— both environments assume the same workload identity
ClientsLoad balancercrosses boundary
Load balancerBlue: app v1.3 ×6· 100% → 0% at the switch
Load balancerGreen: app v1.4 ×6· 0% → 100% at the switch
Blue: app v1.3 ×6Postgres (shared)
Green: app v1.4 ×6Postgres (shared)· same rows, same time
Blue: app v1.3 ×6Cache (shared)
Green: app v1.4 ×6Cache (shared)

The database does not switch, and that is the whole difficulty

Duplicating the application tier is easy. Duplicating the data tier is not, because the data is the thing that must not fork. Two databases would mean writes landing in one and not the other, and a cutover that loses whatever arrived in the interval. So blue and green share one database, one cache, one queue set, one object store.

Which means the compatibility requirement from Rolling Deployment and the Compatibility It Demands does not go away. It changes shape. During the overlap both versions are connected to one database — briefly, if you switch promptly, and for as long as you keep blue warm. And after a rollback, the old version must cope with everything the new version wrote while it was live. If green ran for eleven minutes and wrote rows in a new format, blue is now reading them.

This is the honest limit of "instant rollback". Traffic reverts in seconds; data does not revert at all. Any release with a non-additive data change has a point of no return somewhere after the switch, and knowing where that point is — rather than assuming there is not one — is what separates a rehearsed blue/green practice from a demo.

ComponentDuplicated?Implication
Application instancesYesThe straightforward part, and the source of the capacity cost.
Load balancer target setSwitchedThe cutover mechanism. Fast, reversible, and the thing to rehearse.
DatabaseNo — sharedBoth versions run against one schema. Data written by green survives a rollback to blue.
CacheNo — sharedSerialisation changes must be compatible in both directions, or version the keys.
Queues and their consumersUsually sharedGreen may enqueue messages that blue consumes after a rollback.
Session storeSharedA session format change logs users out on the switch, or on the rollback.
DNS, when used as the switchn/aAvoid it. Client caches ignore TTL, so "instant rollback" becomes "instant for some clients". Switch at the load balancer.
What switches, what does not, and what that implies.

Running it without wasting the second environment

The operational details are where blue/green earns or loses its reputation. Verify green through an internal path before the switch, and make sure that path is not publicly reachable — an unverified environment on a public address is a real exposure, and it is easy to create by accident when you give green its own hostname for testing.

Warm green before switching. An environment that has never served a request has cold connection pools, cold JIT, cold caches and cold DNS entries. Switching 100% of production traffic onto it in one step produces a latency spike that looks like a bad release and is not — and this is the single most common reason a good blue/green deploy gets rolled back unnecessarily. Send synthetic traffic first, or shift weight in two or three steps rather than one.

Then decide how long blue stays. Tearing it down immediately after the switch discards exactly the property you paid for. Keeping it for an hour costs an hour of double capacity and buys an hour of one-command rollback. Most teams settle on "until the next deploy" for a low-frequency service, or a fixed window tied to how long a problem typically takes to become visible.

A blue/green cutover, with the step everyone skips marked.PROVIDER-NEUTRAL
  1. 1Provision green3–10 minutes — ILLUSTRATIVE

    A full environment on the promoted artifact digest, with production configuration and the production workload identity.

    Configuration drift between blue and green — green is built from the same code as blue only if both come from the same configuration.

  2. 2Verify privately

    Smoke tests and dependency checks through an internal address.

    Exposing green publicly to test it, which puts an unverified environment on the internet.

  3. 3Warm

    Synthetic traffic to establish connection pools, caches and any lazily-initialised state.

    The step that gets skipped. A cold environment taking 100% of traffic produces a latency spike that reads as a bad release.

  4. 4Switch

    Change the load balancer target set. All traffic moves to green.

    Using DNS instead: client caches ignore TTL, so the switch and the rollback are both partial.

  5. 5Observe

    Compare error rate, latency and saturation against blue's pre-switch baseline.

    Nobody watching. The rollback is instant and useless if the decision to use it takes forty minutes.

  6. 6Retire blue

    After the rollback window expires, tear blue down and reclaim the capacity.

    Tearing down immediately, which discards the property you paid double capacity for.

Key points

  • Blue/green gives the fastest rollback available: a routing change back to an environment that is still running and still warm.
  • It costs roughly double application capacity for the overlap, and needs the quota headroom to exist at all.
  • The database, cache and queues are shared. Both versions run against one schema, and data written by green survives a rollback.
  • Warm green before switching, or the resulting latency spike will be mistaken for a bad release.
  • Switch at the load balancer, never at DNS — client caches make a DNS switch partial in both directions.

The loop, answered

Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.

How it works
  • Green is provisioned as a complete application environment on the promoted artifact, with production configuration and identity.
  • It is verified through an internal address that is not publicly routable.
  • Synthetic traffic warms connection pools, caches and lazily-initialised state.
  • The load balancer's target set is changed — atomically for new connections, with existing connections to blue draining.
  • Blue keeps running for the rollback window; reverting is the same target-set change in reverse.
  • After the window, blue is destroyed and its capacity reclaimed.
What you still own
  • You own the rollback window length, which is a direct trade of capacity cost against rollback availability.
  • You own the verification suite that runs against green, since the switch is only as safe as what was checked before it.
  • You own the warming step and whatever generates synthetic traffic.
  • You own the data compatibility analysis: what green writes, and whether blue can read it after a revert.
  • You own quota headroom, which is what makes doubling possible at all and is a common first-attempt blocker.
How it fails
  • Cold green: connection pools and caches unestablished, so the first minutes after the switch show a latency spike that is misread as a bad release.
  • A DNS-based switch where client caches ignore TTL, leaving a long tail of traffic on the old environment in both directions.
  • Blue torn down immediately after the switch, so the rollback that was the entire point is unavailable when needed.
  • Green exposed on a public hostname for testing, putting an unverified environment on the internet.
  • A rollback that restores the code but not the data, leaving blue reading rows green wrote in a new format.
  • Quota exhaustion mid-provision: green is half-built, blue is still serving, and the deploy is stuck in an unplanned state.
  • Long-lived connections — WebSockets, streaming responses — that never move because the switch only affects new connections.
How it scales
  • The capacity cost scales linearly with fleet size, which makes blue/green cheap for small fleets and unattractive for very large ones.
  • Provisioning time scales with instance count and startup time, and it is what sets the floor on how long a release takes.
  • Shared database connection count roughly doubles during the overlap, which can be the binding constraint before compute is. See Managed Databases.
  • Long-lived connections extend the effective overlap indefinitely and need their own drain strategy.
Security
  • Green must be verified through a private path. A publicly reachable unverified environment is a genuine finding, unlike a public load balancer on 443, which is the design.
  • Both environments assume the same workload identity, so a compromise of green is a compromise of the same permissions blue has — an unverified environment briefly holds production credentials.
  • The fast rollback is a security capability: a version with a newly disclosed vulnerability can be removed from service in seconds.
  • The switch itself is a privileged operation. Whoever can change the load balancer target set can direct production traffic anywhere.
Cost shape
  • Roughly double application capacity for the overlap, shaped as a short spike rather than a sustained increase.
  • The rollback window is the cost dial: a longer window costs more double-capacity time and buys more rollback availability.
  • Doubling connections and cache load briefly can push shared components into a higher tier, which is the surprise line item.
  • Cheap on a small fleet and expensive on a large one — the arithmetic is unusually simple, which is why the decision is usually straightforward.
What to watch
  • Error rate and latency percentiles for green against blue's pre-switch baseline, not against a general threshold.
  • Connection counts and pool saturation on shared dependencies during the overlap.
  • Time from switch to rollback decision, which is the number that determines whether the fast rollback was worth paying for.
  • Drain progress on blue: connections still open after the switch, which reveals the long-lived-connection problem.
  • The signal that lies: green's health checks before the switch. They prove the process is up on an environment serving zero real traffic, which is not the same as prove it can serve.
Simpler alternatives
  • Rolling, when capacity doubling is not affordable and a slower rollback is acceptable. This is most services most of the time.
  • Canary, when you would rather limit the blast radius than optimise the rollback — it costs a routing layer instead of capacity.
  • Feature flags, when the risky part is a behaviour change rather than the deployment: flipping a flag is faster than any environment switch and does not need a second fleet. See Deployment Is Not Release.
  • For a service with two instances, blue/green means four instances for twenty minutes. That is cheap enough that it is often the right answer for small critical services, which is the inverse of the usual assumption.
What adopting this costs
  • Buys seconds-level rollback; costs roughly double capacity for the overlap and the quota to allow it.
  • Buys a fully verified environment before any traffic arrives; costs a warming step and the discipline to actually perform it.
  • Buys a clean cutover; costs nothing on the data tier, which does not switch — so the compatibility work is the same as rolling.

What people believe, and what is true

Claim

Blue/green gives instant rollback.

Reality

It gives instant *traffic* rollback. Data written by green stays written, so any non-additive data change has a point of no return somewhere after the switch.

Claim

Green is safe because it was tested.

Reality

It was tested with no load, cold pools and cold caches. Warm it before it takes 100% of production traffic, or the switch produces a latency spike you will misdiagnose.

Claim

You can do blue/green with DNS.

Reality

You can, and client caches will ignore your TTL in both directions. Switch at the load balancer where you control the target set.

Apply it