The question this answers
What does a consensus protocol assume, and what happens when the assumption is false?
Safety (agreement, validity, integrity) holds under asynchrony: arbitrary message delay, loss and reordering, and any number of crash failures. Liveness (termination) holds only under partial synchrony — the network must eventually deliver messages within some bound — and a majority of the configured membership must be reachable. Neither guarantee survives nodes that send incorrect information.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node cannot check any of these assumptions. It cannot tell whether a majority exists — only whether a majority has replied *to it, recently*. It cannot tell whether the network is in its "eventually well-behaved" phase or its bad phase; both look like waiting. It cannot detect a lying peer under a crash-stop model, because the model assumes lying does not happen. Every assumption is invisible from inside.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Assumption one: a majority is reachable
A consensus cluster of five nodes needs three to make progress. This is not a tuning parameter; it is the whole safety argument, because any two subsets of three from five must share a node, and that shared node is what prevents two conflicting decisions from both succeeding. Weaken the quorum and you have not made the system more available — you have made it capable of being wrong.
Consensus cannot make an unavailable majority available. If three of five nodes are down, no configuration, no timeout, no retry policy produces a decision. Operators reach for restarts and failovers at this point; nothing helps except restoring nodes or explicitly, dangerously, reconfiguring the cluster to a smaller membership — which is a decision to accept possible data loss, taken by a human, not by the protocol.
This is also why cluster size is a real design choice rather than a "more is better" dial. Larger clusters tolerate more simultaneous failures and make every single decision slower, because the majority is larger. See Quorums: What R + W > N Does and Does Not Buy for the counting argument in general form.
| Cluster size | Majority | Failures tolerated |
|---|---|---|
| 3protocol | 2 | 1 |
| 4protocol | 3 | 1 — no better than 3, and slower |
| 5protocol | 3 | 2 |
| 7typical | 4 | 3 — rarely worth the added latency |
Assumption two: the network eventually behaves
The FLP impossibility result proves that in a fully asynchronous system — no bound at all on message delay — no deterministic protocol can guarantee that every node eventually decides, if even one node may crash. This is not a gap waiting for a cleverer algorithm. It is a theorem.
Practical protocols escape it by assuming partial synchrony: the network may misbehave arbitrarily for arbitrarily long, but there is eventually a period during which messages arrive within a bound. During the bad periods the protocol makes no progress; during the good periods it does. Crucially, it is never *unsafe* during the bad periods — it just waits.
The engineering consequence is precise: timeouts in a consensus protocol are a liveness parameter, never a safety parameter. Set the election timeout too low and you get elections that interrupt healthy leaders and throughput collapses. Set it too high and failover takes longer. Neither setting can cause two leaders to both commit — that is guarded by Terms and Epochs: Making Stale Leaders Harmless, not by the clock.
raft: term 41 -> 42, starting election (no heartbeat for 1043ms) raft: term 42 -> 43, starting election (vote split: self=1, needed=2) raft: term 43 -> 44, starting election (no heartbeat for 1002ms) raft: dropped 812 append_entries in last 10s (peer unreachable) apply: committed index stuck at 9,338,201 for 47s # note: no data loss, no divergence — just no progress
Assumption three: nodes crash, they do not lie
Raft, Paxos, Zab and Viewstamped Replication all assume a crash-stop (or crash-recovery) failure model: a node either follows the protocol correctly or stops. It never sends a message that contradicts its own state, never forges a vote, never claims to have log entries it does not have.
That assumption is reasonable inside a trusted datacentre and unreasonable across a trust boundary. A corrupted disk that silently returns wrong bytes, a bug that double-votes in the same term, or a malicious participant all step outside the model, and the guarantees simply do not apply — the protocol is not "degraded", it is off-model. Tolerating that class needs a Byzantine protocol and a quorum of 3f+1 rather than 2f+1. See Byzantine Failures, and Why You Probably Do Not Assume Them.
The everyday version of this is subtler than malice: a node whose durable state was not actually durable. Raft requires that a vote and a log entry are on stable storage before being acknowledged. A node that acknowledges then loses the write after a restart — because fsync was disabled for speed — has lied, in exactly the sense the model forbids, and can cause two leaders in the same term.
What consensus explicitly does not promise
It is worth being blunt about the negatives, because most disappointment with consensus comes from expecting one of these.
None of these are shortcomings. They are the shape of the tool, and a design that needs one of them needs something else in addition — or instead.
- It does not prevent partitions. Nothing does. It decides who keeps working during one.
- It does not make an unavailable majority available. Down is down.
- It does not make reads fresh. A follower read is a stale read unless the protocol is extended — leader leases, read index, or routing reads through the log.
- It does not bound latency. It bounds it only when the network is well-behaved, and network behaviour is exactly what it cannot control.
- It does not protect data written outside the log. Any side effect a node performs without going through the protocol is unprotected by it — see Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.
- It does not survive losing a majority’s durable state. Restore-from-backup after that is a human judgement about which history to keep.
Key points
- Safety holds under full asynchrony; liveness needs partial synchrony plus a reachable majority.
- A majority is a safety mechanism, not a tuning knob — weakening it permits conflicting decisions.
- FLP means "eventually decides" is unachievable in a fully asynchronous model; timeouts buy liveness only.
- Timeout tuning affects throughput and failover time; it can never affect correctness.
- The crash-stop model excludes lying nodes — including nodes that acknowledge writes they did not durably persist.
- Consensus does not prevent partitions; it defines behaviour during one.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The membership is fixed and known: a majority is defined against the *configured* set, not against the reachable set.
- • Every decision requires acknowledgement from a majority of that configured set.
- • Acknowledgement requires the acknowledging node to have durably recorded its state first.
- • Failure detection is by timeout and is allowed to be wrong; a false suspicion triggers an election, not a decision.
- • When the network misbehaves, the protocol waits. When it behaves again, progress resumes from the last decided point.
- • A majority becomes unreachable and progress stops entirely.
- • A network that never enters a well-behaved period, so elections repeat indefinitely without a winner.
- • A node that loses acknowledged state after a restart, violating the model.
- • Membership changes performed carelessly, so two different "majorities" exist simultaneously across two configurations.
- • Clock or scheduling pauses long enough that a live node is universally suspected and repeatedly deposed.
- • Total write unavailability with every process healthy: the operator sees three of five nodes up, all logging "no quorum", CPU near zero, and clients timing out on writes while reads from stale followers still succeed.
- • Perpetual election: two nodes split the vote every round. The operator sees the term counter climbing by tens per second and zero committed entries — the fix is randomised timeouts, not a bigger cluster.
- • Divergent history after a power loss: a node that had
fsyncdisabled restarts having forgotten a vote it acknowledged. The operator sees two nodes claiming leadership in the same term in the logs, and a data mismatch that the protocol cannot explain. - • Split configuration during a membership change: the operator sees two disjoint groups each believing it has a majority, each accepting writes, and a reconciliation problem that has no automatic answer.
- • Cross-region collapse: a majority spanning two regions makes every write pay an inter-region round trip; the operator sees write latency step from 3 ms to 90 ms with no code change, because the leader moved.
- • The majority requirement *is* the coordination requirement — it cannot be optimised away, only relocated.
- • Where the majority physically sits determines the latency floor: a majority inside one datacentre is milliseconds; a majority spanning continents is tens of milliseconds and cannot be improved.
- • Membership changes are themselves decisions that must go through the protocol, which is why joint consensus (two overlapping configurations at once) exists.
- • Nothing already committed is lost, provided the durability assumption held.
- • The minority side retains its data and its ability to serve stale reads, but must not accept writes.
- • The system is unavailable for writes rather than inconsistent — the deliberate choice.
- • Detect: alert on quorum loss directly ("members acknowledging < majority"), which is a different signal from node down.
- • Contain: fail fast on the minority so callers shed load instead of queueing against a cluster that cannot answer.
- • Recover: restore enough members. Resist the reconfigure-to-smaller-cluster reflex — it converts an availability incident into a possible data-loss incident.
- • Reconcile: returning members catch up from the leader’s log; entries they hold that were never committed are truncated. See The Raft Log: Commit Index, Divergence and Reconciliation.
- • Verify: confirm all members report the same term and commit index, and that durable-write settings are actually enabled on every member.
- • Number of members acknowledging the leader within the heartbeat interval — availability margin, not node count.
- • Term/epoch increase rate; a healthy cluster changes term rarely.
- • Time since last commit, per member.
- • Whether durable-sync is enabled on every member — an audit item, not a metric, and the one that turns a survivable outage into data loss.
- • Round-trip latency to each peer at p99, since the majority is formed from the fastest peers.
- • Whenever you are about to claim a consensus system "guarantees consistency" — this lesson is the list of conditions attached to that claim.
- • When sizing a cluster and choosing where its members physically live.
- • During incident review, to separate "the protocol misbehaved" (almost never) from "an assumption was violated" (almost always).
- • As a reason to avoid consensus entirely — the assumptions are usually satisfied inside one datacentre, and the alternative is often worse.
- • When used to justify weakening quorum settings for availability; that trade is not available at the protocol level.
- • If the majority requirement is unaffordable, drop the requirement for agreement rather than weakening the quorum: use Leaderless Replication: Every Replica Accepts Writes with explicit conflict resolution and accept the weaker guarantee honestly.
- • If cross-region majorities are too slow, keep the consensus group inside one region and replicate asynchronously across regions, accepting bounded loss on region failure. See Active-Passive: Simple to Reason About, Rarely Tested.
- • If lying nodes are in the model, use a Byzantine fault-tolerant protocol with
3f+1members — a different and much more expensive tool. - • If the network is genuinely reliable and failure is rare, a single node with fast, verified restore is simpler and often more available in practice.
Quorum arithmetic and its fine print
✓ Every read quorum meets every write quorum
A reader therefore touches at least one replica that saw the write — provided everything on the right holds.
- Quorums are drawn from the same N home replicas — no sloppy quorum, no hinted handoff to a stand-in node.
- Membership is stable: every participant agrees which N nodes hold this key while the read and the write are in flight.
- A write that reached W replicas is durable on all W — an acknowledgement is not withdrawn by a later crash.
- The reader can tell which of the returned values is newest — a version, a vector clock or a monotonic timestamp, not a wall clock it merely trusts.
- Under those conditions every read quorum shares at least 1 node with every write quorum, so the last acknowledged write is visible to the read.
A sloppy quorum accepts W acknowledgements from nodes outside the home set during a partition. The count is met, the overlap is not, and the read misses the write.
Read-your-write for a single key, under the stated assumptions: any read touching R = 3 of N = 5 replicas sees at least one replica carrying the last write acknowledged by W = 3.
- ✕A sloppy quorum accepts W acknowledgements from nodes outside the home set during a partition. The count is met, the overlap is not, and the read misses the write.
- ✕A write fails partway: fewer than W replicas acknowledged, so the client saw an error, but some replicas kept the value. A later read can return a write that was reported as failed.
- ✕Last-write-wins resolution with unsynchronised clocks discards the newer value because the older writer had a faster clock. The overlap happened; the read still returned stale data.
What people believe, and what is true
Consensus keeps the system available during a partition.
It keeps the *majority side* available and deliberately stops the other. If there is no majority side, nothing is available for writes.
A four-node cluster is more fault-tolerant than three.
Both tolerate exactly one failure. The four-node cluster needs three acknowledgements instead of two, so it is strictly slower for the same tolerance.
Lowering the election timeout makes the system safer.
It makes failover faster and false elections more frequent. Safety is unaffected either way; throughput is not.
The protocol handles bad disks.
It assumes storage tells the truth. Silent corruption or lost acknowledged writes are outside the model, which is why serious implementations add checksums and refuse to start on detected corruption.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Consensus needs a majority of its members reachable and a network that eventually delivers messages. It assumes nodes crash rather than lie. Outside those conditions it stops, or its guarantees no longer apply.
Practical
Size clusters at 3 or 5, keep the members close enough that a majority round trip is cheap, and verify durable writes are on. Alert on quorum health rather than node health. When a majority is lost, restoring nodes is the fix; shrinking the cluster is a data-loss decision that needs a human.
Advanced
FLP shows no deterministic asynchronous protocol both guarantees safety and always terminates with one crash possible. Chandra and Toueg reframed this as the weakest failure detector needed for consensus, ◇W: eventually some correct node is never suspected. Real timeouts approximate ◇W, and the reason a mistuned timeout is harmless to correctness is that the failure detector is consulted only to *start* an election, never to decide one.
Internals
Membership change is where these assumptions are most easily broken. Naively swapping the configuration lets old-config and new-config majorities exist simultaneously with no overlap — two legitimate leaders. Raft solves it either with joint consensus (a transitional configuration requiring majorities in both old and new) or by restricting changes to one member at a time, which guarantees overlap. Any operational runbook that says "edit the peer list and restart" has skipped this.
Apply it
- 🔧 Given a 5-node cluster split 3/2, list precisely what each side can and cannot do, for reads and for writes.
- 🔧 Design an alert that fires on quorum loss but not on a single node restart, and explain the signal it uses.
- ⚡ A cluster spans two datacentres, three nodes in A and two in B. DC A burns down. What is the honest recovery story, and what did the topology cost you?
- 💬 What does a consensus protocol assume about the network, and what breaks if the assumption is false?
- 💬 Why is a four-node cluster not more fault-tolerant than a three-node one?
- 💬 Someone disabled fsync on the Raft members to improve write latency. What is now possible that was not before?
- 💬 Three of five nodes are permanently gone. Walk me through your options and their consequences.