The question this answers
Should I fix a running server or replace it, and when does the difference actually matter?
A fleet must be able to grow by adding machines that behave identically to the ones already serving, and must be rebuildable from source after a failure — without anyone having to remember what was done to the originals.
A machine whose entire content is derived from a versioned artifact, so that adding capacity, recovering from a loss and rolling back a change are all the same operation: launch this image, discard that one.
How a fleet stops being a fleet
Nobody decides to build a snowflake server. It happens one justified exception at a time. An incident at 02:00 needs a kernel parameter changed and there is no time for a pipeline. A debugging session leaves a package installed. A colleague raises a file-descriptor limit on the one host that kept hitting it. A certificate is renewed by hand on the machine that was missing the automation. Each change is small, correct, and undocumented, and the machines that were provisioned identically diverge.
This is configuration drift, and its cost is not aesthetic. Drift means the fleet is no longer one thing you can reason about: a bug reproduces on three of the eight machines, load is unbalanced because two of them have a different worker count, and a canary is not representative because the canary host has an accumulated fix that the others do not. Debugging changes from "why is the service failing" to "why is this instance failing", which is a much slower question.
The end state is the server nobody can reproduce — usually the oldest one, usually the most important one, and always the one nobody wants to reboot. It has a hand-edited config file that predates the current team, a package pinned to a version that is no longer in any repository, and a mount added interactively. It cannot be rebuilt because nobody knows what it contains, so it cannot be patched confidently, replaced safely, or moved. It has stopped being infrastructure and become an artefact.
- Drift is produced by correct, justified, individually small changes — which is why "be more careful" has never fixed it anywhere.
- The first symptom is usually a bug that reproduces on some instances and not others, or unexplained load imbalance.
- A canary on a drifted fleet tests the canary host, not the change.
- The machine nobody will reboot is the fleet's real single point of failure, and it is rarely on any architecture diagram.
- Drift also breaks infrastructure-as-code, which is the same problem one layer up — see Drift: When the File and Reality Disagree.
$ diff <(ssh web-01 'rpm -qa | sort') <(ssh web-03 'rpm -qa | sort') > openssl-3.0.7-18.el9 # web-03 only: patched during the March advisory < openssl-3.0.7-16.el9 # web-01 was in a maintenance window and got skipped > jq-1.6-16.el9 # web-03 only: installed during an incident, never removed < kernel-5.14.0-284.11.1.el9 > kernel-5.14.0-427.13.1.el9 # web-03 rebooted; web-01 has 417 days of uptime $ diff <(ssh web-01 'sysctl -a 2>/dev/null | sort') <(ssh web-03 'sysctl -a 2>/dev/null | sort') < net.core.somaxconn = 4096 # raised by hand on web-01 during a load event in 2024 > net.core.somaxconn = 128 # the distribution default, still, on web-03 $ for h in web-01 web-02 web-03; do > printf '%-8s workers=%s uptime=%s\n' $h \ > "$(ssh $h grep -c ^worker /etc/app/workers.conf)" "$(ssh $h uptime -p)" > done web-01 workers=12 uptime=up 1 year, 1 month, 22 days web-02 workers=8 uptime=up 3 days web-03 workers=8 uptime=up 3 days Three machines. One template. Three different machines.
Two ways to apply a change
The mutable model applies changes to running machines: a configuration-management tool or a person connects to each host and converges it toward a desired state. It is fast, it preserves local state, and it works — as long as every machine actually converges, every machine is reachable during the run, and nothing has been changed underneath the tool. The failure mode is partial application, and partial application is silent: the tool reports the hosts it reached.
The immutable model does not change running machines at all. A change to a package, a kernel parameter or a configuration file becomes a change to the image build; the build produces a new versioned image; and the fleet is rolled by launching machines from the new image and destroying the old ones. Nothing is ever patched in place, so nothing can partially converge. The comparison below is not about which is nicer to read — it is about which operations exist afterwards.
Notice what the immutable version gets for free rather than as extra features. Rollback is launching the previous image, which is an operation the team performs constantly rather than a special procedure. Scaling out produces a machine identical to the ones already serving, because it comes from the same artifact. Patching is a rebuild, which means it is tested by the same rollout that every other change goes through. And the boot path is exercised on every deploy, so the reboot that does not come back is discovered in a pipeline instead of at 02:00 — the failure in The VM Lifecycle becomes structurally impossible.
# apply the security patch
ansible web -m yum -a "name=openssl state=latest"
# ok: web-02, web-03, web-04, web-05, web-06, web-08
# unreachable: web-01 <- in a maintenance window, silently skipped
# skipped: web-07 <- pinned by hand months ago, nobody remembers why
# kernel patch needs a reboot; the fleet is now mixed
ansible web -m reboot --limit "web-02:web-03" # the two someone had time for
What exists afterwards:
fleet state = whatever each machine converged to
rollback = re-run the tool with an older version pin,
on the hosts that were reachable this time
new instance = launched from a base image + a converge run
that must succeed on first boot, over the network
reproducible? = only for the parts the tool manages;
everything ever done by hand is invisible to it# the change is a change to the image definition, reviewed like code
git commit -m "bump openssl, raise somaxconn to 4096"
# CI builds one artifact and tags it with the commit
packer build -var "git_sha=9f3c1a2" web.pkr.hcl
# -> ami-0b7e4 / image web-9f3c1a2 (identical bytes for every environment)
# roll the fleet: launch new, health-check, drain and destroy old
terraform apply -var "image=web-9f3c1a2"
What exists afterwards:
fleet state = every machine is exactly web-9f3c1a2
rollback = terraform apply -var "image=web-8d21f04"
(the same operation, run every day)
new instance = launch the image; no network converge step
reproducible? = yes, from the commit — including the boot path,
which every deploy re-testsBoth fleets end up patched, and only one of them can prove it. The mutable run reports the hosts it reached and says nothing about web-01 and web-07, so the fleet is now mixed and the report is technically accurate and practically false. The immutable version replaces machines with an artifact built from a reviewed commit, so "what is running" is a git SHA rather than an accumulation of maintenance history. The price is real and should be stated: an image build pipeline, a storage cost for images, a few extra minutes per change, and a hard rule that nothing valuable lives on the instance — see Persistent Data and Containers for the same rule stated for containers.
When the difference actually pays
On an ordinary Tuesday, immutable infrastructure is slower. Changing a kernel parameter means editing a definition, waiting for a build, and rolling a fleet — several minutes for something that took thirty seconds with an SSH session. If nothing ever goes wrong and the fleet never changes size, the mutable approach genuinely wins, and pretending otherwise is how this argument loses credibility.
It pays under exactly two conditions, and both are conditions of urgency. The first is scaling out: a burst arrives, ten new machines launch, and the question is whether they are the same as the ones already serving. Under the immutable model they are, by construction. Under the mutable model they are a base image plus a converge run that has to succeed over the network, right now, on ten machines at once, at the worst possible moment — which is also when the package repository is most likely to rate-limit you.
The second is recovery. A zone is lost, an instance is corrupted, a change has to be reversed. Recovery time is dominated by how confidently you can produce a correct machine, and confidence comes from having produced one recently. An immutable fleet builds and launches machines on every deploy, so the recovery path is the path it uses constantly. A mutable fleet's recovery path is a converge run last exercised at provisioning time, against a base image that has since changed, with hand-made adjustments nobody wrote down. This is the same argument as Restore Testing: an untested recovery path is not recovery, it is a hope with a runbook.
The cost panel below is the honest accounting. Immutable infrastructure moves cost from the incident to the pipeline: you pay continuously in build minutes and image storage so that you do not pay unpredictably in incident hours. That is a good trade for most fleets and a bad one for a single machine that changes twice a year.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Configuration drift is produced by individually justified changes, so process discipline alone has never solved it.
- The end state is a machine nobody can reproduce, nobody will reboot, and nobody has removed from the critical path.
- Immutable infrastructure replaces machines from a versioned image instead of changing them, so partial convergence cannot happen.
- Rollback, scale-out, patching and recovery all collapse into one operation: launch this image, discard that one.
- The payoff arrives specifically when you need to scale fast or recover fast — the two moments when confidence in a machine matters most.
- The price is a build pipeline, image storage, a few minutes per change, and a hard rule that nothing valuable lives on the instance.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A change is made to an image definition — packages, kernel parameters, configuration templates, agents — and reviewed as code.
- • A build pipeline produces one image artifact tagged with the commit, and promotes that same artifact through environments unchanged. See Build Once, Promote the Same Bytes.
- • Instances are launched from the image; environment-specific values come from configuration and secrets at boot, not from the image.
- • A rollout launches new instances, waits for health checks, shifts traffic and destroys the old ones — the standard replacement in Rolling Deployment and the Compatibility It Demands or Blue/Green: Two Environments, One Switch.
- • Rollback is a redeploy of the previous image tag, using the identical mechanism.
- • Nothing writes durable state to the instance: data goes to a database, files to object storage, logs to a shipper.
- • The image build pipeline itself — its identity, its base image source, its cadence, and what happens when the base image publisher ships a breaking change.
- • An image retention and cleanup policy, per region, or storage grows without limit.
- • A rebuild cadence independent of feature changes, so security patches land even in a week when nothing shipped.
- • A rule, enforced socially and technically, that shell access is for diagnosis and never for repair — a fix applied by hand is a fix that will be lost.
- • The base image supply chain: what you inherit, from whom, and how you would know it changed — see The Infrastructure Supply Chain.
- • A converge run reports success for the hosts it reached and silently leaves unreachable or pinned hosts on the old version.
- • A scale-out event launches instances that must converge over the network, and the package repository rate-limits during the burst — new capacity arrives broken.
- • A canary is tested on a host that has an accumulated hand-fix, so the canary passes and the fleet-wide rollout fails.
- • An immutable fleet writes something valuable to local disk, and every rollout quietly destroys it.
- • Image storage grows unbounded across regions because no retention policy was ever set.
- • The base image is rebuilt from a moving upstream tag, so two builds of the same commit produce different machines — reproducibility lost at the root.
- • Immutable scale-out is O(1) in risk: every new instance is the same artifact, whether you launch one or a hundred.
- • Mutable scale-out is O(n) in risk and depends on the network at the worst moment, because every new machine must converge.
- • Rollout time grows with fleet size in both models; only the immutable model makes it a routine, measured number.
- • Image build time becomes the bottleneck on change velocity for large images — the same pressure that makes Why Image Size Is an Infrastructure Problem an operational concern for containers.
- • Patching becomes a rebuild, which means it goes through the same review and rollout as every other change instead of being a separate privileged process.
- • Short instance lifetimes shorten credential lifetimes and discard anything an intruder installed on a machine that gets replaced anyway.
- • A drifted fleet cannot answer "are we patched?" honestly, which is a compliance problem before it is a security one.
- • Removing routine shell access removes a large class of privileged human action, and the audit trail becomes the commit history — see Audit Trails.
- • The base image is now part of your supply chain: pin it by digest, rebuild deliberately, and know who publishes it.
- • Immutable spends predictably: build minutes, image storage, and brief compute overlap during each rollout.
- • Mutable spends unpredictably: maintenance hours that scale with fleet size, plus incident time attributed to symptoms rather than to drift.
- • Image storage is the immutable model's recurring surprise, especially when images are copied to several regions and never expired.
- • The largest number in the comparison is unmeasurable in advance: the recovery you could not perform quickly because you could not produce a correct machine.
- • Image version per running instance, which should be one or two values across the whole fleet and is a direct drift measurement.
- • Instance age distribution — a long tail of old machines means replacement is not actually happening.
- • Time since last successful image build, which catches a pipeline that broke in a quiet week and left the fleet unpatched.
- • Configuration-management run results including unreachable and skipped hosts, if you are still mutable. The success count on its own is the lie.
- • The signal that lies: "the last configuration run succeeded". It describes the hosts the tool reached, and the hosts it did not reach are precisely the ones you needed to know about.
- • Stay mutable with disciplined configuration management. For a small, stable fleet that changes rarely, a well-run converge tool is genuinely adequate, and building an image pipeline for three machines is the cargo cult this domain argues against — see No Cargo-Cult Infrastructure.
- • Containers, which give you immutability by default without a VM image pipeline. If the workload fits in an image, this is usually the cheaper route to the same property — see What Is Inside a Container Image.
- • A managed platform, where the provider owns the image and the replacement entirely and the question stops existing for you.
- • A hybrid that is honest about itself: immutable for the stateless tier where replacement is cheap, carefully managed mutable machines for the few stateful hosts where it is not. The failure is drifting into this by accident rather than choosing it.
- • Immutable buys reproducibility, fast rollback and a continuously tested boot path, and charges a build pipeline plus minutes of latency on every change.
- • It buys the ability to prove what is running, and charges a hard constraint: no durable local state, ever, including the debugging conveniences people miss.
- • Mutable buys immediate change and preserved local state, and charges drift, unprovable patch status and a recovery path nobody has exercised.
- • Frequent replacement buys short-lived credentials and less accumulated risk, and charges continuous rollout activity that must itself be reliable.
What people believe, and what is true
Immutable infrastructure means nothing ever changes.
It means machines never change. Changes happen constantly — as new images and new instances. The rate of change is usually higher, not lower.
Configuration management gives me the same guarantees.
It converges the hosts it can reach toward the state it knows about. It cannot see what was done by hand, and it reports on the hosts it reached, not on the fleet.
You cannot debug an immutable server.
You can log in and read everything. The rule is that you do not repair it — you diagnose, fix the image definition, and replace. Anything you needed to keep should have been shipped off the box already.
This only applies to large fleets.
Fleet size decides whether it is worth the pipeline. The reproducibility problem starts at two machines, and the unreproducible server problem starts at one.
Go deeper
Overview
Either you change running machines or you replace them from a versioned image. The first drifts; the second is reproducible and costs a build pipeline.
Practical
Measure drift before arguing about it: diff package lists and key settings across the fleet, and look at the spread of instance ages. If the fleet is already inconsistent, that is your evidence. Then start with the stateless tier, where replacement is cheap and the win is immediate.
Advanced
Make the image the unit of promotion: build once, tag with the commit, promote the identical artifact through environments, and let configuration and secrets differ at boot. Pin the base image by digest and rebuild on a schedule independent of feature work, or security patches only land in weeks when something happened to ship. Set image retention per region on day one.
Internals
The property that does the work is that the artifact is content-addressed and the launch path is the same every time, so "what is running" is answerable from an identifier rather than reconstructed from history. That is also why the model breaks the moment durable state lives on the instance: replacement stops being idempotent, and every operation that depended on replacement being safe — rollback, scale-in, patching, recovery — silently becomes dangerous.