Recreate: Stop Everything, Then Start the New Thing
The simplest strategy, an outage by design — and the only honest answer when two versions of your system genuinely cannot coexist.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
When is deliberately taking the service down the correct way to deploy, and what does that window actually consist of?
Some changes cannot be made while the old version is still running: an exclusive lock, a singleton job, a file format only one version understands. For those, overlapping versions is not a safety feature, it is the failure.
Recreate is the primitive strategy you use before you know better. Any real system should be doing rolling deploys.
Rolling a change that cannot coexist produces something worse than downtime: a mixed state where two versions corrupt each other's shared data, and the corruption outlives the rollout.
- Rolling a change that cannot coexist produces something worse than downtime: a mixed state where two versions corrupt each other's shared data, and the corruption outlives the rollout.
- Teams that consider recreate beneath them run it accidentally anyway — a rolling deploy where every new instance crashes on startup drains to zero and becomes an unplanned recreate, with no chosen window.
- The downtime is assumed to be the restart time. It is shutdown drain plus process start plus dependency connection plus cache warm plus readiness — and the last two dominate on real services.
- Because recreate is treated as trivial, nobody rehearses it, so nobody knows that the previous artifact does not start any more.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The sequence is: stop accepting new traffic, drain in-flight requests, terminate old processes, start new processes, wait for readiness, accept traffic. Every one of those steps has a duration, and the outage is their sum, not the middle one.
- The property recreate buys is that at no point are two code versions live. That is the *only* thing it buys, and for the workloads that need it, it is the thing they need.
- Termination is where recreate goes wrong quietly. If the old process is killed rather than signalled, in-flight requests die and any work it was holding is lost (Graceful Shutdown).
- Startup is where it goes wrong loudly. A recreate has no old version to fall back to: if the new one does not start, the outage does not end when the deploy does.
What the window is actually made of
Teams quote their recreate downtime as the container start time because that is the number the deploy tool prints. The user-visible window starts when the load balancer stops getting healthy responses and ends when it starts getting them again, and that is a longer interval at both ends.
Reconstruct it once, on a real service, and the dominant term is nearly always warm-up rather than start-up.
- T+0schangeDeploy begins. Instance is marked out of the load balancer pool; new connections stop arriving.
- T+2sactionIn-flight requests still completing. This is the drain, and skipping it is how deploys drop requests (Draining: Stopping Without Dropping).
- T+12sactionTermination signal sent; process finishes its current work and exits. A hard kill here is where the lost work happens.
- T+14ssignalLast old process gone. From here until readiness, nothing serves — the actual outage begins.
- T+16schangeNew process starts. Runtime boots, configuration is read and validated (Validate at Startup, Fail Clearly).
- T+25sactionDependency connections established: database pool, cache client, message broker.
- T+55ssignalCaches cold. The process is up and answering, but slowly, and downstream load is elevated because every read is a miss.
- T+70srecoveryReadiness passes; load balancer returns the instance to the pool; first real request served (Probes: Readiness, Liveness and Startup).
- T+3mrecoveryLatency returns to its usual shape as caches fill. The deploy tool reported success at T+70s; the system recovered here.
Times are illustrative of the shape, not measurements. The teaching is the ordering and which term dominates: the gap between "deploy succeeded" and "system healthy" is the warm-up, and it belongs to the outage.
Workloads where coexistence is the bug
The case for recreate is not simplicity, it is correctness. These are workloads where a rolling deploy does not reduce risk, it introduces a failure mode that recreate does not have.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Rolling deploy of a leader-elected singleton | Two schedulers both believe they hold leadership for a few seconds | The lease outlives the process that held it, so the new instance acquires while the old one still thinks it has it | Recreate, or make the leadership lease shorter than the drain and the work idempotent (Job Scheduler Reliability) |
| Rolling deploy across an on-disk format change | Old instances fail to read files the new ones wrote | Two versions sharing a volume with one format between them | Recreate, or make the format change additive and two-phase (Expand, Migrate, Contract) |
| Rolling deploy of a worker whose message schema changed incompatibly | Messages land in the dead-letter queue during the rollout window | v1 consumers pull messages that only v2 can parse | Recreate the consumer fleet, or make the consumer tolerant before the producer changes (Dead Letter Queues Are an Operation) |
| Rolling deploy holding an exclusive database lock | New instances block on startup and never reach readiness | Only one process can hold the resource, and the old one has not exited | Recreate — this workload has a capacity of one by construction |
| Recreate where startup fails | Outage continues past the planned window with nothing serving | No old version remains to fall back to | Validate config and dependencies before terminating; keep the previous artifact one command away (Rollback: Only Useful If It Is Actually Safe) |
The one place recreate is not a choice
maxSurge and maxUnavailable on a Deployment's rolling update, and the safe shape is surge at least one with unavailable zero. A VM autoscaling group expresses the same choice as whether an instance refresh maintains minimum healthy capacity; a managed platform usually decides for you and does not tell you which it chose.A rolling deploy where every new instance fails readiness does not stay rolling forever. Depending on the platform, it either halts partway — leaving a permanently mixed fleet — or drains the old instances anyway and leaves you at zero. The second case is a recreate you did not schedule, at a moment you did not choose.
That is why the surge and unavailability settings matter: they decide whether a bad rollout stalls with the old version still serving, or grinds the service down to nothing while reporting progress.
take 1 old instance out
-> start 1 new instance
-> new instance fails readiness
-> platform retries, keeps taking old ones out
-> serving capacity falls with each batch
-> unscheduled outage, reported as "deploying"add 1 new instance alongside
-> new instance fails readiness
-> no old instance was removed
-> rollout stalls at full old capacity
-> alert on stalled rollout
-> deploy failed, service unaffectedWhether a failed rollout becomes an outage is decided entirely by whether you surge before you remove. Adding capacity first costs one extra instance for the length of the deploy and converts a class of outages into a stalled pipeline.
How to do it properly
Most important first.
- Use it deliberately, for the workloads that earn it: single-instance services, scheduled jobs and workers, leader-elected singletons, and any change where two versions writing the same state is unsafe.
- Measure the window before you need it. Time a recreate in a lower environment and know which of drain, start, warm dominates (A Successful Deploy Is Not Evidence of a Healthy System).
- Make shutdown correct first: handle the termination signal, stop accepting, finish in flight, then exit (Graceful Shutdown).
- Verify the previous artifact still starts, on a schedule, so the reversal is not a hypothesis (Rollback: Only Useful If It Is Actually Safe).
- For user-facing services, put something in front of the gap — a maintenance response is a better experience than connection refused, and it tells you the window is real.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains it — that is the design. Every user is affected for the length of the window, and the only lever is how short and how well-announced the window is.
What can go wrong
- The new version fails to start and there is nothing serving. Recreate converts a deploy failure directly into an outage.
- A long cold start — cache warm, JIT, connection pools, index load — turns a planned seconds-long window into a minutes-long one.
- Clients that do not retry treat the window as a hard failure, and the queue of retries at the far end arrives all at once when you come back (Retry Storms: The Load You Generated Yourself in Observability terms).
- Rollback is a second identical window, which teams underestimate because they are counting one outage rather than two.
- "Recreate is for beginners." It is the correct strategy for singletons, exclusive workloads and incompatible formats, all of which exist in mature systems.
- "The downtime is the restart time." It is drain plus start plus warm plus readiness, and warm is usually the largest term.
- "We have a maintenance window, so downtime is free." Free for users, perhaps. It is not free for the queue building up behind you, or for jobs whose schedule you just missed.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The observed window matches the expected one, measured from last successful request to first successful request rather than from the deploy tool's clock.
- No requests were killed mid-flight: in-flight count reached zero before termination.
- The service reached readiness, served real traffic, and its dependencies reconnected — not merely that the process is running.
- Deploy the previous artifact by the same mechanism, and accept a second window of the same length.
- Because rollback is symmetrical, the cost of being wrong is exactly double the cost of the deploy. That is a reason to be more certain before a recreate than before a canary, not less.
- If the change touched shared state that the old version cannot read, the rollback is not a rollback at all and you are rolling forward whether you planned to or not (Roll Forward: When Going Back Is the Harder Option).
- Automate the sequence and its gating — drain, terminate, start, wait for readiness, verify — because a hand-run recreate at 2am skips the drain.
- Keep the scheduling of the window human where users are affected. When to take a deliberate outage is a business decision, not a pipeline decision.
- You are trading availability for simplicity and for the guarantee of no coexistence. On a system where that guarantee is not needed, it is a bad trade.
- The simplicity is real and worth something: there is no mixed-version window to design for, so ordinary changes carry less compatibility burden (Version Coexistence: N and N+1, in Both Directions).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe stop-drain-start-warm sequence is the same on VMs, containers and bare processes. What differs is who runs it: an orchestrator with a Recreate strategy, a systemd restart, or a person typing.
- SCALE-SPECIFICAt one instance and modest traffic, a few seconds of downtime per deploy costs less than the engineering to avoid it. Above the point where deploys are frequent and traffic is continuous, the accumulated windows become the availability budget (Error Budgets: Unreliability You Are Allowed to Spend in Observability terms).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.