The question this answers
What states does a virtual machine pass through from request to destruction, and how would I notice when it fails in each of them?
A team must be able to create a machine that is correct on first boot, keep it correct while it runs, apply security patches without inventing an outage, and destroy it without losing data or leaving billable debris behind.
A named progression with a known failure at every step, so that "the deploy failed" becomes "the configure step could not reach the secret store" — and so that patching and termination are planned states rather than events that happen to you.
Seven states, and what breaks in each
The instance lifecycle in The Instance Lifecycle is about traffic: the gates where requests get lost. This one is about the machine: the states a virtual machine occupies from the API call that asks for it to the API call that destroys it. They overlap in the middle and differ at both ends, and the two states that get the least design attention — patch and terminate — are the two that cause the most avoidable incidents.
The reason is that both are rare, slow, and easy to defer. A machine that boots correctly today will boot correctly tomorrow, so nobody exercises the boot path. A machine that has run for four hundred days has not been rebooted, so nobody knows whether it comes back. And a machine that is terminated silently takes its local disk with it, which nobody notices until the day someone needed what was on it.
Read the progression below with one question in mind for each state: *if this failed right now, which graph would move?* For several of them the honest answer is "none of the ones on our dashboard", and that is the actionable output of this lesson.
- 1Provisionseconds
The control plane is asked for an instance: a size, an image, a zone, a subnet, a security group, an instance role, a root volume.
Insufficient capacity for that family in that zone, or an account quota — and both bite hardest during a scale-out, when everyone else is asking for the same thing. Notice it in the scaling group's activity log, not in application metrics.
- 2Boot20–90 s
The guest kernel starts from the image, mounts filesystems, brings up the network, and starts the service manager.
A volume that fails to mount drops the machine into an emergency shell with no network and no logs shipped. The instance is "running" as far as the provider is concerned and completely unreachable to you. The serial console is often the only way to see it.
- 3Configure10 s – several minutes
Startup scripts run inside the guest: fetch configuration and secrets, assume the instance identity, install or template files, register with service discovery, start the application.
The longest and most fragile state, because it reaches out to the network. A package repository that is unreachable, a secret the instance role may not read, a configuration store that rate-limits during a fleet-wide replacement — all produce a booted machine that never becomes useful.
- 4Rundays to years
The application serves. Health checks pass, metrics flow, logs ship, and the machine accumulates state, log files and manual changes.
The root filesystem fills with logs and the process can no longer write; the clock drifts and signature validation starts failing; someone SSHes in and fixes something, and the machine is now unique — see Mutable Servers and Immutable Images.
- 5Patchminutes, plus a reboot you must schedule
Security updates are applied — to the kernel, to system packages, to the runtime. Some take effect immediately; kernel updates do not take effect until a reboot.
The state most often skipped. Patching without rebooting leaves the machine reporting a patched package and running the old kernel. Patching with an automatic reboot at 03:00 across a whole fleet is an unplanned rolling outage. Notice the gap by tracking patch level and uptime together.
- 6Restart30–120 s
The guest reboots — deliberately for a kernel patch, or involuntarily after a host event.
The first honest test of whether the machine is reproducible. Services started by hand and never enabled at boot do not come back; a filesystem edited months ago and never written to the image is gone; a mount added interactively is absent. The failure is discovered at the worst time, because a reboot is usually part of fixing something else.
- 7Terminateseconds
The instance is deregistered, signalled, and destroyed. Compute billing stops. Local disk is gone.
Anything on local disk that was not shipped elsewhere is lost — buffered logs, uploaded files, queue state. Attached volumes, addresses and snapshots frequently survive and keep billing. Notice both with an orphaned-resource report, because nothing will ever alert you.
The reboot that did not come back
The most instructive VM failure is also the most ordinary. A kernel security advisory lands, an engineer patches and reboots the first instance in the fleet, and it does not return. The provider console reports the instance as running and both status checks as passing, because from the outside the virtual machine is doing exactly what a virtual machine does: it exists and it consumes CPU. There is no SSH, no metrics, no logs — because the log shipper is a service inside the guest, and the guest never got that far.
The serial console output below is what is actually happening. A data volume that was mounted by hand eighteen months earlier and never written into /etc/fstab correctly is now referenced by a unit file that depends on it. The mount fails, the dependency chain fails, and systemd drops to emergency mode waiting for a password on a console nobody was watching.
Everything about this is a lifecycle failure rather than a patching failure. The machine had been in run for over a year accumulating manual changes; restart was the state that revealed them; and nothing in the monitoring covered the gap between "the hypervisor says this instance is running" and "the application inside it is alive". The fix in the moment is the console. The fix that matters is to make restart a routine event — replace instances regularly, so the reproducibility of the machine is tested continuously rather than during an emergency.
- Provider status checks prove the virtual machine exists and its kernel responds. They say nothing about your application.
- A machine with a long uptime is an untested machine: every manual change since the last boot is an unexploded assumption.
- The serial console is the diagnostic of last resort for boot and configure failures, and most teams first learn it exists during the incident.
- Patch without reboot leaves a machine reporting patched packages and running the vulnerable kernel — track patch level and uptime together or the report is a lie.
- Rebooting an entire fleet on an automatic schedule is a rolling outage you did not plan. Stagger it, or replace instead of rebooting.
aws ec2 get-console-output --instance-id i-0a91... (equivalent exists on every provider)
[ OK ] Reached target Basic System.
[ OK ] Started Network Manager.
[ OK ] Started D-Bus System Message Bus.
[FAILED] Failed to mount /data.
See 'systemctl status data.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for app.service.
[DEPEND] Dependency failed for node-exporter.service.
[DEPEND] Dependency failed for log-shipper.service.
Give root password for maintenance
(or press Control-D to continue):
--- meanwhile, in the console the on-call engineer was looking at ---
instance state ............... running
system status check .......... ok <- the hypervisor can see the guest
instance status check ........ ok <- the guest kernel is responding
CPU utilisation .............. 2%
application health check ..... (no data — target was deregistered)
uptime before this reboot .... 417 days
The instance is running. The machine is not.What to watch, state by state
The value of naming the states is that it makes monitoring gaps obvious. For each one there is a signal that reveals the failure and, more usefully, a signal that stays green while it happens — and the second column is where post-incident reviews spend their time.
A pattern runs through the whole table: signals produced *outside* the guest tell you the machine exists, and signals produced *inside* the guest tell you it works — but the ones inside the guest are exactly the ones that disappear when boot or configure fails. That is why the load balancer's target health, which is produced outside the guest and depends on the application answering, is the single most valuable VM signal you have. It is the only common check that spans both sides of that line.
The second pattern is that patch and terminate produce no signal at all unless you build one. There is no metric for "this fleet has drifted three kernel versions behind", and there is no alert for "we have been paying for 340 GB of volumes belonging to instances that no longer exist". Both are reports you have to decide to run — see Drift: When the File and Reality Disagree and Idle Capacity: Headroom or Waste?.
| State | Symptom | Signal that reveals it | Signal that lies |
|---|---|---|---|
| Provision | Capacity is requested and never arrives; the fleet stays below its desired count | Scaling-group activity log, quota metrics | Instance health across the running fleet, which is perfect — those instances are fine |
| Boot | Instance exists, is unreachable, ships no logs | Serial console output; load balancer target health | Provider system and instance status checks, both reporting ok |
| Configure | Instance boots but never becomes ready; errors reference a secret or a package repo | Startup-script logs pulled from inside the guest; time-to-ready trending up | CPU utilisation, which is low and stable because nothing is running |
| Run | Scattered 500s, or writes failing while reads succeed | Filesystem free space; application error rate | Process liveness — the process is alive and simply cannot write |
| Patch | A patched package is installed and the vulnerable kernel is still running | Patch level joined against uptime, per instance | The patch tool's own success report |
| Restart | The machine does not come back; a service that never survived a boot is missing | Serial console; load balancer target health | Instance state "running", which is true and useless |
| Terminate | Data on local disk is gone; volumes and addresses keep billing | Orphaned-resource report; log-shipping continuity | The bill, which moves too slowly and too smoothly to notice |
Key points
- Seven states: provision, boot, configure, run, patch, restart, terminate — each with a characteristic failure.
- Configure is the most fragile state because it depends on the network: secrets, package repositories and configuration stores all fail there.
- Patch is the most skipped state, and patching without rebooting leaves a machine that reports as compliant and runs the old kernel.
- Restart is the first honest test of reproducibility, and a long uptime means that test has not been run in a long time.
- Terminate destroys local disk and frequently does not destroy the volumes, addresses and snapshots that keep billing.
- Provider status checks prove the virtual machine is running; only a check that reaches the application proves the machine works.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A provisioning request selects size, image, zone, subnet, security group, instance role and volumes; the control plane finds host capacity or fails.
- • The guest kernel boots from the image, mounts filesystems declared in the image, configures the virtual NIC and starts the service manager.
- • A startup script supplied as user data runs on first boot, fetching configuration and secrets using the instance identity from the metadata service.
- • While running, a health check outside the guest decides whether the instance receives traffic; agents inside the guest ship metrics and logs.
- • Patching updates packages in place; kernel changes require a reboot, which re-runs boot and re-tests every assumption made since the last one.
- • Termination deregisters the instance, signals the guest, destroys the guest and its local storage, and releases the host slot — attached resources follow their own deletion rules.
- • A tested startup path: build the image so that boot and configure do as little network work as possible, and fail loudly and visibly when they must.
- • A patch policy with an explicit reboot plan — staggered across the fleet, drained first, and scheduled rather than automatic at 03:00.
- • A maximum instance age, so restart is exercised routinely and reproducibility is proven continuously.
- • Filesystem free-space alerts on every mount, including the ones added after the image was built.
- • Continuous log shipping, since termination and boot failure are both states where a flush-at-shutdown design ships nothing.
- • An orphaned-resource report covering volumes, addresses and snapshots, because nothing else will ever surface them.
- • Provisioning fails for capacity or quota during the scale-out that needed it, and the fleet quietly stays below its desired count.
- • Boot drops to an emergency shell over a failed mount while the provider reports the instance as running with all checks passing.
- • Configure fails because the instance role cannot read a secret, producing a machine that looks healthy and never serves.
- • A fleet-wide replacement makes every instance fetch configuration at once, and the configuration store rate-limits — so the replacement fails everywhere simultaneously.
- • A patch tool reports success, the kernel is not rebooted, and the fleet is reported as compliant while running the vulnerable version for months.
- • An unattended-upgrade reboot policy restarts most of a fleet inside the same ten-minute window.
- • Termination loses buffered logs and local uploads, and leaves volumes and elastic addresses billing indefinitely.
- • Time from provision to serving traffic is the number that determines whether scaling can answer a burst — measure it as boot plus configure, separately.
- • Configure does not scale for free: every dependency it touches is hit once per instance, and a large replacement multiplies that instantly.
- • Capacity in a single family and zone is a real ceiling during a scale-out; a fleet with one acceptable instance type has a single point of provisioning failure.
- • Patch and restart cost grows linearly with fleet size unless the answer is replacement from an image rather than in-place maintenance.
- • Patching is the security obligation the guest OS creates, and reboot discipline is what turns patching into protection rather than paperwork.
- • Secrets fetched during configure live in guest memory for the instance's lifetime, so long-lived instances mean long-lived secrets — see Secrets in Infrastructure.
- • The metadata service is reachable during configure and forever after; require its hardened access mode so an application-side request-forgery bug is not credential theft.
- • Local disk is not erased on termination in any way you can verify, so encrypt volumes and treat anything written locally as needing encryption at rest.
- • Shell access during run is where audited, reproducible infrastructure quietly stops being either; log it and prefer replacing a machine to fixing one — see Audit Trails.
- • Billing runs from boot to terminate, so time spent in configure is paid time serving nothing, multiplied by every replacement.
- • Long-lived instances are cheap in provisioning effort and expensive in accumulated operational risk, which arrives as incident time rather than as invoice lines.
- • Terminate is a cost event that frequently fails to happen: orphaned volumes, addresses and snapshots outlive their instances for years.
- • Patching and reboot coordination is engineer time that scales with fleet size, and it is the cost that pushes teams toward immutable replacement.
- • Time-to-ready split into boot and configure — you cannot design scaling or diagnose slow deploys without the split.
- • Load balancer target health, which is the one common signal produced outside the guest that depends on the application answering.
- • Filesystem free space on every mount, alerted well before full.
- • Patch level joined with uptime per instance, which is the only way to see the patched-but-not-rebooted state.
- • Instance age distribution across the fleet: a long tail of old machines is a drift and reproducibility report in one number.
- • The signal that lies: the provider status check. It reports that the virtual machine exists and its kernel responds, which stays true while the machine sits in an emergency shell.
- • Do not manage the lifecycle at all: a managed container service or function platform provisions, patches and replaces for you, and removes five of these seven states.
- • Replace instead of patch. Building a new image and rolling the fleet turns patch, restart and drift into one operation you already perform, and is usually less work than in-place maintenance — see Mutable Servers and Immutable Images.
- • A provider-managed patch service for a fleet you genuinely must keep long-lived: less control, far less coordination, and a real answer for a small team.
- • For a single low-traffic internal machine, a documented maintenance window and a manual reboot is honest and adequate. Not everything needs a rollout strategy.
- • In-place patching buys speed and continuity of local state, and charges drift plus a reboot whose outcome nobody has tested.
- • Replacement buys a continuously tested boot path, and charges an image pipeline and a strict rule that nothing valuable lives on local disk.
- • Longer instance lifetimes buy stability and charge accumulated manual change, longer-lived secrets and a harder reboot.
- • Doing more work in the image buys fast, reliable configure and charges a slower build and more frequent rebuilds.
Instance lifecycle: mutable vs immutable
mutable provision → boot → configure → run → patch → restart → terminate
↑______________|
the box is edited in place; the image no longer describes it
immutable build image → provision → boot → run → replace → drain → terminate
↑___________________________________|
the box is never edited; a change means a new imageWhat people believe, and what is true
The instance status check tells me the application is healthy.
It tells you the virtual machine exists and its kernel responds. An instance sitting at an emergency shell passes both checks.
A long uptime is a sign of a stable machine.
It is a sign that the boot path has not been tested recently. The longer the uptime, the more unverified manual change the next reboot will reveal.
Automatic security updates mean the fleet is patched.
Only if something reboots for kernel updates. Patched-but-not-rebooted is the most common form of false compliance on a VM fleet.
Terminating an instance cleans up its resources.
Root volumes often delete; additional volumes, snapshots and reserved addresses often do not. Orphaned storage is one of the most reliable line items in any long-lived account.
Go deeper
Overview
Seven states from provision to terminate. Each fails in a specific way, and patch and terminate are the two that get skipped.
Practical
Write down, for each state, which signal would reveal a failure. Add the two that are almost always missing: patch level joined with uptime, and an orphaned-resource report. Then set a maximum instance age so restart is exercised on purpose.
Advanced
Move work out of configure and into the image. Every network dependency in the startup path is a way for a fleet-wide replacement to fail all at once, and it is also the largest component of time-to-ready — which is the number that decides whether autoscaling can answer your bursts.
Internals
Boot failures are invisible to guest-side monitoring because guest-side monitoring is a service inside the guest and it starts late. That is why the serial console — a virtual device the hypervisor exposes, written by the guest kernel before any of your software runs — is the only diagnostic that spans the gap, and why load balancer target health is the most valuable ordinary signal: it is produced outside the guest and only passes when something inside it answers.