ReplicaSets: The Layer You Should Not Manage
A ReplicaSet keeps N pods matching a template alive. It exists so that a Deployment can roll out by scaling two of them in opposite directions — and that is the only reason you should ever look at one.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is this extra object between my Deployment and my pods, and when does it matter?
A rolling update needs two populations of pods to exist at once, each with its own count, each independently maintained. Something has to own "keep exactly N of this exact template running" so that something else can own "shift N from one template to another".
Ignore it entirely — it is generated, it has an unreadable name with a hash on the end, and everything works without knowing it exists.
Ignoring it is mostly right and fails in exactly two places: reading a stalled rollout, and understanding a rollback. Both are moments when you can least afford to be learning a new object.
- Ignoring it is mostly right and fails in exactly two places: reading a stalled rollout, and understanding a rollback. Both are moments when you can least afford to be learning a new object.
- The pod names people paste into incident channels contain the ReplicaSet hash, so "which version is this pod?" is answerable from the name — if you know what the name is made of.
- Old ReplicaSets scaled to zero look like leftover junk and get deleted "for tidiness", which throws away the fast rollback path (Rollback: Only Useful If It Is Actually Safe).
- When a rollback is unexpectedly slow, the reason is usually that the old ReplicaSet is gone and its pods must be created from scratch, including image pulls.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A ReplicaSet has one job: ensure the number of pods matching its selector equals its replica count. It creates pods when there are too few and deletes them when there are too many.
- A Deployment owns several ReplicaSets — one per pod template revision. Changing the template creates a new one; changing only
replicasdoes not. - A rolling update is therefore two scale operations run against each other, bounded by surge and unavailability: new ReplicaSet up, old ReplicaSet down, gated on readiness (Deployments: Declaring What Should Be Running).
- The hash in a ReplicaSet name is derived from the pod template, which is why an identical template produces the same ReplicaSet and a one-character change produces a new one.
- Old ReplicaSets are retained at zero replicas up to
revisionHistoryLimit. A rollback scales one of them back up — no rebuild, no registry round trip for images already on the nodes. - ReplicaSets select pods by label, and ownership is recorded on the pod. Two controllers whose selectors overlap will fight over the same pods, each seeing the other's as its own surplus.
Three objects, three jobs
The reason for the split is separation of concerns between controllers, and it is worth stating explicitly because the middle row is the one people cannot place.
| Object | Its single job | Do you write it? |
|---|---|---|
| Deployment | Move from one pod template to another, safely, and remember previous ones | Yes — this is your interface |
| ReplicaSet | Keep exactly N pods matching one exact template alive | No — generated, one per template revision |
| Pod | Run the containers on one node | No — generated, replaced rather than repaired |
Reading a stalled rollout
This is the one time the object earns your attention. The output below is what a stuck rollout looks like: the new revision has pods that exist and are not ready, and the old revision is still carrying the traffic.
Read it as two numbers per revision — how many exist, and how many are ready. The gap between them is the whole diagnosis, and it points at readiness rather than at scheduling or image pulls, which would show up as pods that do not exist at all.
1NAME DESIRED CURRENT READY AGE2checkout-7d4b9c5f68 2 2 0 6m <- new template, nothing ready3checkout-6c88f4a2b1 6 6 6 9d <- previous template, serving everything4checkout-5f9a1e77c3 0 0 0 21d <- retained revision, the rollback targetTwo pods exist and none is ready, so scheduling and image pull both succeeded and readiness is failing — read the pod events, not the application logs. The third row is not junk: it is the revision a rollback would scale back up in seconds.
Two controllers, one set of pods
The one genuinely dangerous thing at this layer is a selector that matches pods another controller also owns. Each controller counts the other's pods as its own, concludes it has too many, and deletes some. The other controller recreates them. Nothing errors; pods just churn.
It is rare in hand-written manifests and common in copy-paste-and-rename, where the name changes and the label does not.
Two Deployments named `checkout` and `checkout-canary`, both with `selector.matchLabels: {app: checkout}` and both with `app: checkout` on their pod template.Selectors of `{app: checkout, track: stable}` and `{app: checkout, track: canary}`, with matching template labels — so a Service can still select `app: checkout` and reach both deliberately.Selectors define ownership, and overlapping ownership is a fight rather than an error. The better form also makes the shared Service intentional: one address in front of both tracks is precisely how a canary is served (Canary: One Percent, Then Five, Then Watch).
How to do it properly
Most important first.
- Do not create, edit or scale ReplicaSets directly. The Deployment controller will reconcile your change away, and in the meantime you have two sources of truth.
- Do read them during a stalled rollout. The counts per ReplicaSet tell you exactly how far the rollout got and which template the surviving pods came from (Reading a Broken Workload).
- Keep
revisionHistoryLimithigh enough for the rollbacks you actually perform, and low enough that the object list stays readable. The retained objects hold no pods and cost nothing but clutter. - Make labels specific enough that no two workloads' selectors can overlap — including across namespaces where your tooling copies manifests around.
- When you need many pods from one template with no rollout semantics at all, that is still a Deployment. A bare ReplicaSet is almost never the right answer.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Direct interference at this layer is contained by the Deployment controller reverting it within seconds — the exception is overlapping selectors, which can churn every pod of two workloads at once.
What can go wrong
- Overlapping selectors between two controllers: each repeatedly deletes pods it considers surplus, producing a churn loop that looks like random restarts.
- A ReplicaSet that cannot create pods — quota exhausted or a rejecting admission policy — so replicas stay below desired with a condition on the object nobody reads.
- History pruned to nothing, making rollback a fresh rollout with image pulls at exactly the wrong moment.
- Someone scales a ReplicaSet by hand during an incident; it works for thirty seconds, then the Deployment controller restores the previous count and the "fix" evaporates (Manual Production Changes).
- A stalled rollout left indefinitely, so two ReplicaSets serve permanently — which is fine for capacity and quietly wrong for anything version-sensitive (Version Coexistence: N and N+1, in Both Directions).
- "I should manage ReplicaSets for finer control." The finer control is illusory; the Deployment controller reconciles it away.
- "An old ReplicaSet at zero replicas is wasted resources." It holds no pods. It is a saved revision, and deleting it costs you rollback speed.
- "The ReplicaSet decides the rollout pace." It maintains a count. The Deployment controller decides the counts, gated on readiness (Probes: Readiness, Liveness and Startup).
- "Two ReplicaSets with pods means something is broken." During a rollout that is the normal state. It is only a problem when it persists.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- During a healthy rollout, the new ReplicaSet's ready count climbs and the old one's falls, and their sum stays within surge and unavailability bounds.
- After a rollout, exactly one ReplicaSet has non-zero replicas.
- A rollback completes in seconds rather than minutes, which is the observable proof the old ReplicaSet was still there.
- The ReplicaSet is the rollback mechanism: restoring a previous revision scales the corresponding old ReplicaSet back up.
- Rolling back at this layer directly — scaling ReplicaSets by hand — is not a rollback, because the Deployment's desired state still names the bad template and will reassert it.
- Automate nothing here on purpose. This layer is managed by a controller and should stay that way.
- Automate the alert instead: a Deployment whose ready replicas have differed from desired for longer than a rollout should take is the signal worth having (Alert on Symptoms, Not on Causes).
- The extra object is genuine conceptual overhead for a benefit — fast, artifact-free rollback — that only appears on bad days. That trade is why it exists and why it is invisible on good ones.
- Retaining history keeps rollback fast and keeps images pinned on nodes, at a small cost in object count and node disk.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThe ReplicaSet is Kubernetes' way of splitting "maintain a count" from "shift between templates". ECS has no user-visible equivalent — a service tracks task set revisions internally; a VM autoscaling group performs an instance refresh with no second group to inspect. The behaviour is common; only Kubernetes gives you the intermediate object to read.
- SIMPLIFIEDDeliberately omits the older ReplicationController and the ownership garbage-collection rules, neither of which changes how you should work with the object: you should not.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.