The Node Replacement That Took the Cluster With It
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Cluster manager | One node down and its ranges under-replicated. | Restore the replication factor as fast as possible. Start streaming immediately at full speed. | ✕ wrong |
| Source nodes | Streaming requests plus normal read traffic, both served from the same disks and NICs. | Serve everything; nothing is failing. | ✓ right |
| Clients | Reads slow across all keys, not only those on the replaced node. | The whole cluster is failing, not one node. | ✕ wrong |
| On-call engineer | One node lost, 23 healthy, and cluster-wide latency collapse. | A second, hidden failure must be in progress. | ✕ wrong |
3 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
Twenty-three of twenty-four nodes were healthy and serving. Commit before reading on: explain how losing one node degrades reads for keys that never lived on it, and say what the recovery policy was implicitly optimising for.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.