Immutable Infrastructure
Replacing servers instead of modifying them, so that what is running is a known function of an artifact rather than the sum of its history.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why replace a working machine to change one package, instead of just changing the package?
A long-lived machine accumulates changes: patches, hotfixes, manual edits, half-finished configuration runs. Two machines built from the same recipe months apart are not the same machine, and nobody can say how they differ.
Keep the fleet running and push changes to it — configuration management on a schedule, package upgrades in place, a script for the occasional fix. It is fast, and it preserves everything already on the machine.
Configuration converges only for the things the tool manages. Everything else — a file someone edited, a package installed by hand, a service left disabled — persists invisibly.
- Configuration converges only for the things the tool manages. Everything else — a file someone edited, a package installed by hand, a service left disabled — persists invisibly.
- A machine that failed its last configuration run is in a partial state, and the next run starts from that partial state rather than from a known one.
- Rebuilding a lost machine produces something subtly different from its siblings, and the difference surfaces as one host behaving oddly under load.
- "It works on the old instances but not the new ones" becomes a common incident shape, and the diff is unknowable because nobody recorded the mutations.
- Rollback means running the previous configuration forward, which is not the same as returning to the previous state.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Build an image or artifact once, from pinned inputs, and treat it as read-only. Every deploy launches new instances from that artifact and terminates the old ones (Build Once, Deploy Many).
- The running machine becomes a pure function of the artifact plus its injected configuration. That is what makes "which version is running" answerable and what makes rollback a relaunch (Tags Versus Digests).
- Nothing about this removes state. It relocates it: data goes into managed stores, volumes or object storage, and the compute layer becomes disposable (Pets and Cattle, Read Carefully is where that boundary gets drawn honestly).
- The replacement is a rollout, so it inherits every rollout concern: health checks, draining, version coexistence, and capacity for running both versions at once (Rolling: Two Versions, One Database, Draining: Stopping Without Dropping).
- Containers make this cheap enough to be the default, but the idea predates them and works with machine images, and does not apply automatically just because you are using containers — a container you exec into and modify is a mutable server with a smaller filesystem.
Mutable and immutable, on the same change
The difference shows up not in the happy path but in what you can say afterwards about what is running.
run upgrade across 40 hosts -> 37 succeed -> 2 fail on a lock, retried by hand -> 1 was down, upgraded next week what is running now? three different versions, and no record of which host has which
build image sha256:9f2c... -> launch new instances from it -> health check, shift traffic, drain old -> terminate old set what is running now? sha256:9f2c..., on every instance, or the rollout halted and you know where
The mutable path has no moment where the fleet is definitionally consistent, so "what is running" is a question you answer by surveying hosts. The immutable path makes it a property of the rollout: either the instance came from that digest or it is not in the fleet.
What has to move off the instance first
Adopting replacement is mostly an exercise in finding the durable things living on machines that nobody remembered were there. This is the list that generates the surprises.
| Lives on the instance | Why it is there | Where it goes |
|---|---|---|
| Application logs | Written to local disk by default | Shipped off-host as they are written (Using Observability, Not Building It) |
| Uploaded files | Simplest thing that worked at the start | Object storage, referenced by URL |
| Local cache | Warmed over hours of traffic | Shared cache, or accept the cold-start cost per replacement (Operating a Cache) |
| Session state | In-process, invisible until replacement | External store, or stateless tokens (Sticky Sessions in Backend covers the trap) |
| Cron jobs added by hand | Someone needed it once | Into the image, or into a scheduler (Cron Jobs in Production) |
| Credentials placed by a person | Set up on day one and never revisited | Workload identity or a secret manager (Workload Identity) |
The replacement is a rollout
Immutability makes the artifact trustworthy. It does not make the swap safe — that is a separate set of properties, and treating replacement as a background maintenance task is how a routine image update takes a service down.
- 1Build and pin
Produce one image from pinned inputs and record its digest.
fails by Floating base tags, so two builds of the same commit differ.
evidence Rebuilding the same commit produces the same digest.
- 2Launch new capacity
Start instances from the new digest alongside the old ones.
fails by No headroom for both versions, so the launch is refused or the service saturates (Capacity During Failover is the same arithmetic).
evidence New instances reach a healthy state before anything is removed.
- 3Verify health
Wait for real readiness — dependencies reachable, warm enough to serve — not just process start.
fails by A readiness check that only proves the port is open (Probes: Readiness, Liveness and Startup).
evidence Error rate and latency on the new set match the old set (Canary Analysis: Compared Against What?).
- 4Shift traffic
Move load to the new instances gradually.
fails by All at once, so a bad image takes everything with it.
evidence A halt condition exists and has been tested.
- 5Drain and terminate
Let in-flight work finish on the old set, then terminate it.
fails by Terminating before draining, dropping requests and in-flight jobs (Graceful Shutdown).
evidence No spike in client-side errors at the moment of termination.
How to do it properly
Most important first.
- Build the image in CI from pinned inputs, tag it by digest, and never modify a running instance (Reproducible Builds).
- Inject configuration at launch rather than baking environment-specific values in, so one artifact promotes across environments (Build-Time and Runtime Configuration, Promotion).
- Make replacement routine — daily or weekly, not only when something changes. An instance that has never been replaced is an untested recovery path.
- Move every piece of durable data off the instance deliberately, and write down where each one went. The pieces you forget are found during the first replacement.
- Disable interactive access in normal operation; when someone genuinely needs it, the session is logged and the instance is marked for replacement afterwards (Production Access).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Rolling replacement with health checks and a halt condition. A replacement that terminates the old set before the new set is healthy is contained by nothing.
What can go wrong
- A base image that is rebuilt from a floating tag, so "the same artifact" quietly changes between builds (Dependency Pinning).
- Configuration drift moving into the launch path: a startup script that pulls the latest package at boot reintroduces mutability at the worst layer.
- Data left on the instance — logs someone needs, a cache that takes an hour to warm, a local queue — discovered during the first replacement (Operating a Cache).
- Replacement without capacity headroom, so a rolling replacement browns out the service (Headroom).
- Image sprawl: hundreds of near-identical images, no retention policy, and a storage bill that grows monotonically (Artifact Retention).
- "Immutable means stateless." It means the compute is disposable. The state still exists and has been moved somewhere that is explicitly durable, which is a design decision, not a side effect.
- "We use containers, so we are immutable." Only if you never modify a running container and never rebuild from floating tags. Both are common.
- "Immutable infrastructure prevents configuration drift." It prevents drift on the instance. Drift in the infrastructure around it — security groups, DNS, load balancer settings — is a separate problem (Drift).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Every running instance can be traced to an image digest, and that digest to a commit (Build Provenance).
- Instance age is bounded and visible. If the oldest instance is four hundred days old, you do not have immutable infrastructure, whatever the pipeline says.
- A full fleet replacement has been performed recently without an incident.
- Rollback is launching the previous image digest and draining the new one — which is genuinely fast and genuinely reversible for the compute layer.
- It is not a rollback for anything the new version did to shared state. A schema change or a data migration is not undone by relaunching the old image (A Migration and a Deploy Are One Event).
- Automate: image build, replacement rollout, health verification, and termination of the old set.
- Keep human: the decision to disable replacement during an incident, and any interactive session on a live instance, which should be an explicit exception rather than a normal capability.
- A one-line fix becomes a full build and rollout. That is minutes rather than seconds, which matters when the fix is urgent — the mitigation is a fast pipeline, not an exception.
- Storage and registry cost grow with image count, and rebuild time is paid on every change however small (What Image Size Actually Costs).
- Warm-up state is lost on every replacement: caches, JIT warmth, connection pools. For some services this is a real latency cost at every deploy (Startup Time & Cold Start is Cloud's treatment of the same effect).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- PLATFORM-SPECIFICOn containers replacement is seconds and effectively free, so it is the default. On machine images it is minutes and involves an image bake, which makes replacement cadence a real cost decision. On bare metal it may be hours, and a mutable model with strict configuration management can be the honest choice.
- GENERALThe underlying property — what is running should be derivable from a recorded artifact rather than from history — holds everywhere, including for the platforms where full replacement is impractical.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — replacement cadence as a continuously exercised recovery path rather than a documented one.