The question this answers
My monitoring says the node stopped sending heartbeats. What have I actually learned?
A timeout-based failure detector guarantees, at best, *eventual strong completeness*: a node that has genuinely crashed is eventually suspected by every correct node. It guarantees nothing about accuracy — a correct node may be suspected at any time, repeatedly, and there is no bound on how often.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The observer knows one fact: no message from the target has arrived at this observer within the chosen window. It does not know whether the target is dead, whether the target is slow, whether the path between them is broken, whether the target’s reply was dropped, or whether the observer itself is the one cut off from everything.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Five realities behind one silence
The mistake is the same shape as the timeout mistake, and it is worth listing the alternatives explicitly because engineers routinely consider only the first.
The node crashed. The case everyone designs for, and often the least likely one in a well-run system.
The network between you is broken. The node is healthy and serving every other observer. Your view is the outlier, and any action you take on it will conflict with the majority’s.
The node is overloaded. It is alive, its heartbeat thread is starved or its GC has paused it, and it will resume — usually still believing everything it believed before the pause.
The heartbeat was delayed or dropped. Queueing, packet loss, a saturated NIC. The node sent it; you did not get it in time.
You are the isolated one. This is the case that gets missed, and it is the most dangerous, because an observer that concludes "everyone else is dead" and acts accordingly is a single node making decisions on behalf of a cluster it has been cut off from. The correct response to "I cannot reach a majority" is to suspect yourself, not everyone else — and that inversion is why quorum-based membership exists and why a node should step down rather than take over when it loses contact.
- Node 5 — healthy, cut off
- n1believes “node 5 has failed”✕ and it is false
- n5believes “nodes 1-4 have all failed”✕ and it is false
- n5believes “it should take over as leader”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Completeness and accuracy, and why you only get one
The formal vocabulary is genuinely useful here. Completeness is "every crashed node is eventually suspected". Accuracy is "no correct node is ever suspected". A perfect failure detector has both, and exists only in a synchronous system where you can wait out a known bound and be certain.
In a partially synchronous system you can have completeness for free: set any timeout, and a dead node will eventually be suspected. Accuracy is what you cannot have, and the timeout value only chooses *how* you fail. A short timeout gives fast detection and frequent false suspicions of healthy nodes. A long timeout gives few false suspicions and a long window in which a genuinely dead node still holds its responsibilities. There is no value that gives you both, and tuning is choosing where on that curve to sit.
The practical consequence is that a failure detector should be treated as producing a suspicion, not a verdict — and downstream logic should be safe under a wrong suspicion rather than dependent on a right one. This is why the good designs never let the detector directly authorise anything: it triggers a *process* (an election, a quorum check, a lease expiry with a fencing token) which is designed so that suspecting a healthy node is survivable.
| Short timeout | Long timeout | |
|---|---|---|
| Detection latencytypical | Low — failover in seconds | High — a dead node holds its role for minutes |
| False suspicionstypical | Frequent, especially under load | Rare |
| Failure modetypical | Thrashing: repeated failovers, unstable leadership | Extended unavailability of whatever the dead node owned |
| Interaction with loadassumption | Load causes suspicion, suspicion causes failover, failover causes load | Degrades slowly and visibly |
Better detectors do not remove the problem
Several refinements genuinely help and none of them changes the fundamental limit. Phi-accrual detectors output a continuous suspicion level based on the observed distribution of heartbeat intervals, rather than a binary verdict at a fixed threshold — which adapts to a network that is generally slower at 3pm and lets different consumers of the signal choose different thresholds. Indirect probing, as used in SWIM-style membership, asks other nodes to probe the target before concluding anything, which specifically addresses "the path between us is broken" and "I am the isolated one". Quorum-based membership requires a majority to agree a node is out, which converts a local suspicion into a cluster decision.
All three reduce false positives. None of them can distinguish a crashed node from one that is paused and about to resume, because that distinction does not exist in the evidence. Which is why the last line of defence is never detection quality but fencing: when a node is evicted, invalidate its authority so that its return is harmless. A monotonically increasing token that the storage layer checks means an evicted node’s writes are rejected on arrival, no matter how confident it is. That is a safety property that does not depend on the detector being right, and it is the only kind worth relying on.
The design instinct to build: detection can be wrong; the consequences of detection must be safe anyway. Every hour spent tuning a timeout is worth less than the first hour spent making a wrong suspicion survivable.
- Phi-accrual: adapt to the observed interval distribution instead of a fixed threshold.
- Indirect probing: ask a third party before concluding, which catches path failures and self-isolation.
- Quorum membership: a majority must agree, so a single isolated observer cannot evict anyone.
- Self-suspicion: a node that cannot reach a majority should step down rather than take over.
- Fencing: make the evicted node’s return harmless, so being wrong is not catastrophic.
What a health check actually measures
Most production failure detection is not a heartbeat protocol between peers; it is a load balancer or an orchestrator calling an HTTP endpoint. The same analysis applies, plus one more failure mode that is specific and common: the health endpoint and the work path are different code. A service whose worker pool is wedged, whose database connection pool is exhausted, or whose downstream dependency is timing out can return 200 OK from /healthz all day, because that handler touches none of those things.
The opposite error is equally common and worse. A health check that verifies every downstream dependency turns one dependency’s outage into a cluster-wide eviction: all instances report unhealthy simultaneously, the load balancer removes all of them, and a partial degradation becomes a total outage. Cloud infrastructure has the liveness-versus-readiness distinction, and it is exactly this: liveness should ask "is this process broken in a way only a restart fixes"; readiness should ask "can this instance serve right now". Conflating them is how a slow database causes a mass restart that makes the database slower.
The rule that follows: a health check should test the instance, never its dependencies. Dependencies get circuit breakers and degraded modes, not eviction.
observer last heartbeat verdict actually ───────────────────────────────────────────────────────────── lb-a 0.4s ago healthy alive lb-b 12.1s ago SUSPECT alive (path b→n5 down) peer n1 11.8s ago SUSPECT alive peer n5 0.2s ago healthy alive n5 is alive. Two observers say otherwise. If either of them is allowed to act alone, the cluster now has a problem that n5 did not cause and cannot see.
Key points
- Silence has five causes; the node being dead is only one, and often not the likeliest.
- The observer being the isolated party is the case most often missed, and the most dangerous.
- Completeness is free; accuracy is impossible. The timeout value chooses which way you fail, not whether.
- Better detectors reduce false positives and never eliminate them.
- Fencing is what makes a wrong suspicion survivable, and it is worth more than any amount of tuning.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The target sends a periodic signal, or the observer periodically probes it.
- • The observer tracks the time since the last successful exchange.
- • When that exceeds a threshold — fixed, or derived from the observed distribution — the observer raises a suspicion.
- • The suspicion is either acted on locally (dangerous) or submitted to a quorum for a cluster-level decision (safe).
- • If the node is evicted, its authority is invalidated with a token so that its eventual return cannot corrupt anything.
- • Heartbeats are dropped or delayed while the node is healthy.
- • The heartbeat thread is starved while the work threads are fine, or the reverse.
- • The observer loses connectivity and interprets universal silence as universal death.
- • A GC pause exceeds the threshold and the node resumes believing nothing happened.
- • The health endpoint is served by a code path that is unaffected by whatever is actually broken.
- • Failover thrash: an aggressive timeout under load evicts healthy nodes, redistributing their work onto the remaining ones, which then also time out. The operator sees a leadership-change counter spiking and no hardware fault anywhere.
- • Isolated observer acts alone: a node that lost connectivity promotes itself. The operator sees two nodes reporting the leader role, and writes landing in two places.
- • Healthy-but-useless: liveness passes while no work progresses. The operator sees 100% health, zero errors, and a queue age climbing steadily.
- • Mass eviction from a dependency check: every instance reports unhealthy when one database is slow. The operator sees the entire pool drop out of rotation at once, converting a slowdown into an outage.
- • Late resurrection: an evicted node returns after a long pause and writes with pre-eviction assumptions. The operator sees writes from an instance that was decommissioned minutes earlier.
- • A local suspicion needs no coordination and confers no authority; that asymmetry is the entire safety argument.
- • Turning a suspicion into an eviction requires a majority, which costs a round trip and is unavailable to a minority — deliberately.
- • Fencing requires a monotonic token issued by something the storage layer also trusts, which makes the storage layer part of the safety protocol rather than a passive participant.
- • A node that is suspected but healthy continues doing its work correctly, which is precisely what makes the situation dangerous.
- • A majority partition can still evict and reassign; a minority partition should be able to do neither.
- • Fenced resources remain safe regardless of how wrong the detector was — that guarantee is independent of detection quality.
- • Detect: track suspicion events separately from actual failures, and compare them — the ratio is your false-positive rate.
- • Contain: require a quorum before any suspicion has consequences, and never let a single observer evict.
- • Recover: readmit a returning node only after it re-synchronises, and only with a new token.
- • Reconcile: check for work performed by a node during the window in which it was considered evicted.
- • Verify: exercise the path deliberately — pause a node beyond its threshold and confirm its return is harmless.
- • Suspicion events and eviction events as separate counters; a large gap between them means quorum is doing its job, and a small gap means single observers have too much power.
- • Time since last contact per observer-target pair, not a single global health boolean — disagreement between observers is the signal that a path is broken.
- • Leadership or ownership change rate; a healthy cluster changes leaders rarely and a thrashing one changes constantly.
- • Rejected fenced writes, which should be rare and non-zero — a zero here often means fencing is not actually wired up.
- • Any system where one node holds an exclusive role — a leader, a lock, a partition owner, a scheduled job.
- • Any place where a load balancer or orchestrator decides whether an instance receives traffic.
- • For stateless instances behind a load balancer where the worst case is a few failed requests, elaborate detection buys little over a simple readiness probe.
- • Aggressive detection on a system under variable load actively harms availability, because load causes suspicion and suspicion causes more load.
- • Leases rather than heartbeats: the holder must renew before acting, so a node that loses contact disqualifies itself without anyone needing to detect it.
- • Make exclusivity unnecessary: if any node may safely do the work because the operation is idempotent and conditional, no detector is on the critical path.
- • Delegate to a coordination service that has already solved this, and consume its session semantics rather than building detection.
- • Human-in-the-loop failover for rare, high-stakes transitions: slower, and sometimes the right trade when a wrong automatic decision is expensive.
Any timeout is wrong in one of two directions
What people believe, and what is true
A missed heartbeat means the node is down.
It means no heartbeat arrived at this observer. Five situations produce that, and in one of them the observer is the failed party.
Tuning the timeout correctly solves this.
Tuning chooses between false positives and slow detection. Neither is eliminated, and the right investment is making a false positive harmless.
A health check tells you whether the service works.
It tells you what the health handler does. If that handler does not touch the work path, it reports on a code path nobody uses.
The health check should verify all dependencies.
That turns one dependency’s degradation into a total eviction of your fleet. Dependencies need circuit breakers; health checks should test the instance.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
No heartbeat means no heartbeat arrived. The node may be dead, slow, unreachable, or perfectly fine while you are the one cut off. Treat it as a suspicion, never a fact.
Practical
Require a quorum before a suspicion has consequences. Have a node that cannot reach a majority step down rather than take over. Keep liveness checks about the instance and readiness checks about capacity, never about dependencies. Then make the evicted node’s return harmless with a fencing token — that is worth more than any timeout tuning.
Advanced
Chandra and Toueg classified detectors by their completeness and accuracy properties and showed that consensus is solvable in a partially synchronous system with a detector as weak as ◇W — eventually weak — which eventually suspects all crashed processes and eventually stops suspecting at least one correct process. That is a remarkably low bar, and it is the theoretical reason production systems can use crude timeout-based detection and still be safe: the detector is not required to be right, only to eventually stop being wrong about one node, and the safety of the protocol never depended on the detector in the first place.
Apply it
- ⚡ Under a traffic spike, your cluster starts failing over repeatedly with no hardware fault. Diagnose it and explain what makes the cycle self-sustaining.
- 💬 List everything that could cause a node to stop sending heartbeats. Which is the most dangerous to get wrong?
- 💬 Why should a node that cannot reach a majority step down rather than take over?
- 💬 Your health check verifies the database connection. What happens when the database is slow?