Pets and Cattle, Read Carefully
A useful metaphor about replaceability that becomes dangerous when treated as a rule — because some infrastructure genuinely is irreplaceable, and a database treated as cattle is a data loss event.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Should every server be disposable, and what happens to the ones that are not?
Teams need a way to talk about which infrastructure can be replaced without ceremony and which cannot. The industry adopted a metaphor for it, and the metaphor is memorable enough that it now gets applied to things it was never about.
Named, hand-tended servers are the old way. Modern infrastructure is identical, numbered, and replaced when unhealthy — so make everything cattle.
A primary database replaced on failure is a restore from backup, not a replacement. If the automation does it, it does it at machine speed and before anyone reads the alert.
- A primary database replaced on failure is a restore from backup, not a replacement. If the automation does it, it does it at machine speed and before anyone reads the alert.
- Systems where identity is load-bearing — a broker with a stable id and a partition assignment, a node in a consensus group, a member of a replica set — break when identity is treated as interchangeable.
- The metaphor implies the problem is sentimentality. It is not. Some components hold the only copy of something, and the correct response to that is care, not shame.
- Applied to the whole estate, "cattle" removes the vocabulary for the parts that need careful treatment, so nobody argues for it and the care quietly stops happening.
- Auto-replacement on a health check failure is exactly the right behaviour for a stateless replica and exactly the wrong behaviour for a node that is slow because it is recovering.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The metaphor is about replaceability, and it dates from the era when almost every server was hand-built and hand-named. Its real content is: prefer components that can be recreated from a recorded definition, because recreatable components make recovery mechanical.
- Replaceability is not a binary. It is a function of three things: whether the component holds authoritative state, whether its identity is referenced by other components, and how long it takes to rebuild to a serving condition.
- A stateless API replica scores low on all three: replace it. A cache replica holds no authoritative state but takes an hour to warm: replaceable, at a cost you should know. A primary database scores high on all three: replacement is a recovery procedure with an RPO attached (RTO and RPO).
- The right target is not "make everything cattle". It is to reduce the number of components that score high, and to give the remaining ones real operational care: backups you have restored, failover you have rehearsed, and automation that escalates rather than acts.
- Managed services are the most common way teams reduce that number honestly. The component is still careful; the care is someone else's operational responsibility, which is a trade with its own terms (Shared Responsibility in Cloud sets out where the line sits).
The three questions, applied honestly
Replace the two buckets with three questions, and the interesting components stop being ambiguous. Notice that the answers move over time: a component that was disposable when it was built acquires state, and nobody re-runs the classification.
| Component | Holds authoritative state? | Identity referenced? | Rebuild to serving | Treatment |
|---|---|---|---|---|
| Stateless API replica | No | No | Seconds | Replace automatically, on any health failure |
| Cache node | No | Sometimes, if sharded by identity | Minutes to hours, warming | Replace automatically; know the warm-up cost (Operating a Cache) |
| Queue consumer | No — the queue holds it | No | Seconds | Replace automatically; ensure in-flight work is acked correctly (Operating Queues and Scheduled Work) |
| Broker / consensus node | Yes, partially | Yes — id and membership | Minutes, plus re-replication | Automate rejoin; require a human for removal |
| Database replica | Copy, not authority | Yes, if promotable | Hours, seeding from primary | Automate rebuild; require a human for promotion |
| Primary database | Yes | Yes | Restore time plus data loss window | Never automatic destruction. Protection at the provider, restore rehearsed |
The metaphor, and what it was answering
It is worth stating the original argument fairly, because it was a good one. In the era it came from, servers were individually built, individually named, and individually irreplaceable — not because they held anything important, but because nobody had written down how they were made. Recovery meant a person remembering.
The correction was: make the definition the source of truth so recovery becomes mechanical. That argument is entirely right and is the argument for everything else in this module.
What the metaphor does badly is the second half. It has no vocabulary for components that are irreplaceable because of what they hold rather than because of how they were built, and it carries a tone that makes arguing for care sound like nostalgia. Both halves need saying.
every component is cattle -> auto-replace on health failure -> no interactive access anywhere -> no special handling for the database failure mode: a slow-recovering primary is terminated by the remediation that was protecting it
reduce the number of careful components
-> everything derivable from a definition
-> everything stateless auto-replaced
-> the few that hold authority get:
protection at the provider,
rehearsed restore,
a human on the irreversible stepThe goal — fewer irreplaceable components — is achievable and valuable. The rule — no irreplaceable components — is false for any system that stores anything, and pretending otherwise removes the care from exactly the components that need it most.
Deciding what a component gets
Run this per component when it is created and again whenever its responsibilities change. The output is not a label, it is a set of operational commitments.
Something in this component is unhealthy. What is automation allowed to do?
when No authoritative state, identity not referenced, rebuild is fast.
cost You must genuinely have moved the state off it — verify rather than assume (Immutable Infrastructure).
when No authoritative state, but rebuild is slow or warm-up is expensive.
cost A restart hides the cause; you need the diagnostic captured before it happens (Production Debugging).
when Holds a replica or participates in a quorum. Removing it may be correct, but not automatically.
cost Time to human response is now in your availability budget.
when Holds authoritative state. The wrong automated action is unrecoverable.
cost Slower mitigation, deliberately — and it obliges you to have a rehearsed procedure ready (Runbooks).
How to do it properly
Most important first.
- Classify each component explicitly on the three questions above and write the answer down next to the service ownership record (The Ownership Record).
- For low-score components: automate replacement, replace routinely, and never allow interactive fixes (Immutable Infrastructure).
- For high-score components: automate everything except destruction and failover. Automate the backup, the restore test, the monitoring, the promotion procedure — and require a human to trigger the irreversible step (Restore Drills).
- Put provider-side protection on the high-score components so that an automated system that is wrong cannot delete them (Destructive Changes: What a Rename Really Does).
- Rehearse the replacement of the careful components on a schedule, in a real environment. A stateful failover that has never been performed is a plan, not a capability (Disaster Recovery as an Operation).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
For disposable components, the rollout mechanism contains it. For authoritative-state components, nothing contains it except a backup you have restored — which is the point of the distinction.
What can go wrong
- Auto-remediation that terminates a struggling stateful node, turning a degradation into a data recovery.
- A "cattle" policy applied to a broker or consensus node whose identity is referenced in configuration elsewhere, so replacement produces a split view of the cluster.
- The opposite failure, still common: a hand-built machine that has been running for years, cannot be rebuilt, and is a single point of failure nobody wants to touch.
- Treating a managed database as if it needed no care, because the care is invisible — until a version upgrade window or a failover event makes it visible (Operating a Production Database).
- Classification done once and never revisited, so a component that acquired state two years ago is still classified as disposable.
- "Cattle means we do not need backups." Backups are what makes a stateful component recoverable at all. The metaphor was never about data (Backup Operations).
- "Pets are a failure of engineering." A primary database is not a pet; it is a component with authoritative state. Using the metaphor to shame care into disappearing is how teams lose data.
- "Kubernetes makes everything cattle." It has a specific abstraction for identity-bearing workloads precisely because some things are not (StatefulSets: Identity, Storage and Order, Why Stateful Workloads Are Harder).
- "Managed means it is now cattle." It means the provider carries the careful part. Your recovery point objective, your upgrade windows and your failover behaviour are still yours to know.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For every stateful component you can name the last successful restore, its date, and how long it took.
- The disposable components have actually been replaced recently, without anyone noticing.
- The list of components that require care is short, written down, and shorter than it was last year.
- For disposable components, rollback is relaunching the previous artifact.
- For careful components, there is no rollback for destruction — only restore, with whatever data loss the recovery point implies. That asymmetry is the entire reason the distinction exists (Partial and Logical Data Recovery).
- Automate: replacement, health-based recycling and scaling for everything that holds no authoritative state.
- Keep human: destruction, failover and promotion of anything that does. Automation here should page, propose and prepare — not act. This is the clearest instance in the domain of a decision that must stay with a person (The Automation Trap).
- Careful components cost operational attention that disposable ones do not: rehearsals, runbooks, on-call familiarity. That cost is real and is the reason teams try to eliminate them.
- Eliminating them usually means adopting a managed service, which trades operational burden for less control over upgrade timing and failover behaviour.
- Keeping a human in the loop for stateful failover costs minutes of downtime that automation would have saved. Sometimes that is the wrong trade — the point is to make it deliberately, per component.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe three questions — authoritative state, referenced identity, rebuild time — apply to any component on any platform. The metaphor is a historical shorthand for the case where all three answers are "no".
- ORG-SPECIFICWhere the line falls is an organisational choice about acceptable data loss and acceptable operational burden. A team with a five-minute recovery point objective can automate more of this than one with a zero-loss requirement, and neither is more correct.
- SIMPLIFIEDPresented as three questions with a small number of outcomes. Real estates have components that are authoritative for one thing and disposable for another — a node that caches aggressively but also holds a durable write-ahead segment — and those need the questions asked per concern, not per host.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — quorum membership and why removing a node is a consensus decision rather than a capacity decision.