Deployments: Declaring What Should Be Running
A Deployment is desired state — this many replicas of this image, with this rollout policy — that a controller works toward continuously, including after failures nobody scripted.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do I say "this version, this many, replaced safely" and have it stay true?
Pods do not recreate themselves, do not roll forward to a new image, and have no memory of what was running before. Something has to hold the intent and act on it every time reality drifts.
Think of a Deployment as a deploy command with YAML syntax: you apply it, it deploys, the job is done. kubectl apply is a verb like deploy.sh.
It is not an event, it is a standing instruction. If you delete pods, they come back; if a node dies, replacements appear; the instruction keeps executing long after your command returned.
- It is not an event, it is a standing instruction. If you delete pods, they come back; if a node dies, replacements appear; the instruction keeps executing long after your command returned.
- That makes "did it work?" a separate question. The apply succeeded; the rollout may still be stalled at three of six replicas with the rest failing readiness (Probes: Readiness, Liveness and Startup).
- Treating it as a command invites imperative edits — scaling by hand, patching an image — which succeed and are then silently reverted the next time the manifest is applied (Manual Production Changes).
- The rollout policy is invisible to people who think in commands, so surge and unavailability defaults decide their capacity during every deploy without anyone choosing them.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A Deployment declares a pod template, a replica count, a selector and an update strategy. Everything else follows from the controller comparing that to what exists (Reconciliation: The Loop Under Everything).
- On a template change the controller creates a new ReplicaSet and scales it up while scaling the old one down, according to the strategy — it does not mutate existing pods (ReplicaSets: The Layer You Should Not Manage).
- The default strategy is a rolling update bounded by
maxSurge(how many extra pods may exist above the desired count) andmaxUnavailable(how many of the desired count may be missing). Those two numbers are your capacity policy during a deploy. - Progress is gated on readiness, not on the container starting. A new pod counts toward availability only once its readiness probe passes, which is what stops a broken version from replacing a working one (Probes: Readiness, Liveness and Startup).
- Old ReplicaSets are kept, scaled to zero, up to
revisionHistoryLimit. That retained history is what makes an undo possible without re-running the pipeline. - A rollout that never becomes ready does not fail loudly by default; it stalls, reports a progress condition after a deadline, and leaves both versions running (Apply Is Not Running).
A Deployment is desired state, not an action
Everything in this manifest is a statement about what should be true, including the two numbers that decide what happens to your capacity while it is becoming true.
1apiVersion: apps/v12kind: Deployment3metadata:4 name: checkout5spec:6 replicas: 67 revisionHistoryLimit: 5 # how many old ReplicaSets stay available to roll back to8 minReadySeconds: 20 # ready for this long before it counts as available9 selector:10 matchLabels:11 app: checkout # immutable in practice: changing it is a new object12 strategy:13 type: RollingUpdate14 rollingUpdate:15 maxSurge: 2 # may run 8 pods briefly16 maxUnavailable: 0 # never drop below 6 available17 template:18 metadata:19 labels:20 app: checkout # must match the selector, or nothing is managed21 spec:22 containers:23 - name: app24 image: registry.example.com/checkout@sha256:9f3e1c...25 readinessProbe:26 httpGet:27 path: /ready28 port: 8080maxUnavailable: 0 with maxSurge: 2 is the "never lose capacity" setting and requires the cluster to have room for two extra pods. Invert them and the rollout is free but takes capacity away exactly when a new version is under scrutiny.
What a rolling update looks like minute by minute
The value of walking a rollout in time is that it makes the coexistence window concrete. There is a period — usually minutes, occasionally forever — where both versions are behind the same Service and either may answer any request.
The timeline below is a rollout that goes wrong in the most common way: the new version starts, passes readiness, and is worse. Nothing in the platform stops it, because nothing in the platform knows what "worse" means.
- T+0schangeNew image digest applied; controller creates a new ReplicaSet at 2 replicas
- T+15ssignalTwo new pods Running but not Ready; old ReplicaSet still at 6
- T+40schangeNew pods pass readiness and enter the Service endpoints; old scaled to 5
- T+1msignalError rate rises from 0.2% to 3%, concentrated in the new version's pods
- T+2msignalRollout continues regardless — readiness passes, so the controller sees success
- T+3mactionOperator pauses the rollout, freezing the current mix rather than completing it
- T+4mactionRoll back to the previous revision; old ReplicaSet scales back to 6
- T+5mrecoveryError rate returns to baseline; new ReplicaSet at 0, retained for inspection
The controller behaved correctly throughout. The gap it cannot fill is between "the process reports itself ready" and "the release is good" — that gap is what canary analysis exists for (Canary Analysis: Compared Against What?).
Where the desired state can disagree with reality
These are the situations where the object and the cluster say different things. Each has a distinct signature and a different first move, and confusing them costs the most time during an incident.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| New pods never become Ready | Rollout stuck; old version still serving | Readiness probe failing — often a dependency, not the app (Probes: Readiness, Liveness and Startup) | Read probe failure events before application logs; roll back if the dependency is not recoverable now |
| Pods stay Pending | Rollout stuck with zero new pods | No node has capacity for the declared requests (The Scheduler, and Why a Pod Is Pending) | A capacity problem: add nodes or reduce requests, not an application fix |
| ImagePullBackOff | Rollout stuck immediately | Bad digest, missing registry credential or a pull rate limit | Verify the artifact exists and the node can authenticate (Artifact Registries) |
| Replicas keep changing on their own | Capacity moves with no deploy | An autoscaler owns the replica count and the manifest also sets it | Remove replicas from the manifest where an autoscaler owns it (Horizontal Pod Autoscaling) |
| Rollout completed, behaviour wrong | Healthy pods, unhealthy service | Readiness measured the process, not the release | Roll back on the version-scoped signal, then fix verification (A Successful Deploy Is Not Evidence of a Healthy System) |
| Pods restart mid-rollout | Error spike during every deploy | Grace period shorter than in-flight requests (Graceful Shutdown) | Raise the grace period and handle the termination signal; drain connections first (Draining: Stopping Without Dropping) |
How to do it properly
Most important first.
- Reference images by digest rather than by a mutable tag, so the desired state names exactly one artifact (Tags Versus Digests).
- Set
maxSurgeandmaxUnavailabledeliberately against your capacity.maxUnavailable: 0keeps full capacity through the rollout and requires headroom for the surge (Headroom). - Make the readiness probe mean "this replica can serve", and make sure it does not merely test that the port is open — a rollout is only as safe as the signal gating it.
- Use
minReadySecondswhen your failure shows up shortly after start rather than at start, so the rollout does not race past it. - Watch the rollout as a rollout: replica counts by version, error rate by version, and a stop condition you decided before you started (Canary Analysis: Compared Against What?).
- Keep the manifest in version control and apply it from there, so the cluster reflects a reviewed artifact rather than someone's terminal history.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A rolling update touches every replica of the service, so an unhealthy version reaches all traffic unless readiness stops it — the gate, plus a canary in front of it, is the containment (Canary: One Percent, Then Five, Then Watch).
What can go wrong
- Rollout stalls halfway: new pods never become ready, old pods are still serving, and the service is running two versions indefinitely. Availability is fine, which is why nobody notices until a compatibility problem surfaces (Version Coexistence: N and N+1, in Both Directions).
maxUnavailableset too high, so a deploy removes more capacity than the remaining replicas can absorb and the rollout itself causes the latency spike (Building a Capacity Model).- A rollout that completes successfully into a broken version, because readiness tested liveness rather than correctness. Nothing in the platform checks whether the new version is *good* (A Successful Deploy Is Not Evidence of a Healthy System).
- Scaling changes made by an autoscaler and then overwritten by an applied manifest that hard-codes
replicas, producing a sudden capacity drop with no related deploy (Horizontal Pod Autoscaling). - Selector edited on an existing Deployment — an immutable field on most clusters — so the apply fails, or on older objects orphans the existing pods.
- "The deploy finished when the command returned." The command returned when the API server stored your intent. The rollout is a separate, observable process.
- "A successful rollout means the release is good." It means the new pods passed readiness. Whether the version is correct is a question for your verification, not the controller's (A Successful Deploy Is Not Evidence of a Healthy System).
- "Deployments handle stateful workloads if I attach a volume." They give pods interchangeable identities and no ordering, which is exactly what stateful systems cannot use (StatefulSets: Identity, Storage and Order).
- "Rolling back is risky." Rolling back to a known-good artifact is usually the safest available action; the risk lives in the state changes that accompanied the release, not the artifact swap.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The rollout reports the expected number of updated, ready and available replicas, and the number of old replicas reaches zero.
- Requests are served by the new version specifically: version distribution in logs or metrics shifts, rather than an aggregate error rate staying flat (Deploys on the Same Timeline as the Symptom).
- Error rate and latency for the new version match the baseline of the old one, measured over a window rather than glanced at.
- A deliberately broken image stalls the rollout instead of completing it — the cheapest test that your readiness gate is real.
- Roll back by restoring the previous desired state: reapply the previous manifest, or roll back to the previous revision, which scales the old ReplicaSet up and the new one down using the same mechanism as any other rollout (Rollback: Only Useful If It Is Actually Safe).
- It is fast because the old ReplicaSet and its image are still there — no rebuild, no pipeline. This is the strongest practical argument for keeping revision history.
- It does not undo anything outside the cluster. A migration, a published event, or a consumed message is not covered (A Migration and a Deploy Are One Event).
- Automate applying from a reviewed repository, with the digest injected by the pipeline rather than edited by a person (Promotion).
- Automate rollout observation: fail the pipeline if the rollout does not reach the expected ready count within a deadline.
- Do not automate rollback on any single alert without thought — an automated rollback into a schema the old version cannot read is a worse outage than the one it was reacting to (Roll Forward: When Going Back Is the Harder Option).
- Declarative desired state makes recovery free and makes emergency imperative action feel like fighting the system — which it is, because the controller will undo it.
- Rolling updates avoid downtime and guarantee a window where two versions serve at once, which pushes real compatibility work onto every change (Version Coexistence: N and N+1, in Both Directions).
- Surge capacity for a zero-unavailability rollout is capacity you pay for continuously to use occasionally.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThe Deployment object and its ReplicaSet mechanics are Kubernetes'. An ECS service performs the same replace-in-batches behaviour against a target group's health checks; a VM autoscaling group does it by instance refresh; a PaaS does it invisibly and gives you a rollout status instead of a strategy.
- SIMPLIFIEDOmits pod disruption budgets, deployment pause, and the interaction with node draining — all of which change how much capacity you actually keep during a rollout.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.