The question this answers
How does Raft choose a leader, and why can the winner never be missing committed data?
At most one leader per term (from vote-once-per-term plus majority overlap), and leader completeness: any node elected leader in term T holds every entry committed in any term before T. Liveness — that some leader is eventually elected — holds only while a majority is reachable and message delays are eventually bounded, and depends on randomised timeouts to avoid perpetual vote splits.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A follower knows only how long it has been since a heartbeat arrived. A candidate knows how many vote grants have reached it — and cannot distinguish "I lost" from "my requests were dropped" from "the grants were dropped"; all three are the same silence. A leader knows when each follower last acknowledged it, which is a fact about the past, not about now. No node ever knows the cluster’s state; each knows only its own inbox.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Three states and the transitions between them
Every Raft node is in exactly one of three states. Followers are passive: they respond to requests and never initiate. Candidates are followers that got bored waiting and are campaigning. Leaders send heartbeats and serve clients. There is no fourth state, no observer role in the core protocol, and no "acting leader".
The transitions are equally sparse. A follower whose election timer expires becomes a candidate. A candidate that wins becomes leader; a candidate that hears from a legitimate leader reverts to follower; a candidate that times out starts a new term and campaigns again. Any node in any state that sees a higher term immediately becomes a follower — including a leader, and this rule takes precedence over everything else.
That last rule is what makes the protocol collapse gracefully. There is no reconciliation logic for "two leaders"; a leader that learns of a higher term simply stops being one, before doing anything else with the message.
1FOLLOWER --(election timeout, no heartbeat)--> CANDIDATE2CANDIDATE --(votes from a majority)--------------> LEADER3CANDIDATE --(AppendEntries from valid leader)----> FOLLOWER4CANDIDATE --(election timeout again)-------------> CANDIDATE (term+1)5LEADER --(sees a higher term, anywhere)-------> FOLLOWER6ANY --(sees a higher term, anywhere)-------> FOLLOWER # overrides allThe election itself
A candidate increments its term, votes for itself, persists both facts, and sends RequestVote to every peer. It carries two things beyond its identity: lastLogIndex and lastLogTerm, a summary of how complete its log is. Those two numbers are what make the safety argument work.
A peer grants the vote only if it has not already voted in this term and the candidate’s log is at least as up to date as its own. "At least as up to date" compares the last entry’s term first, then the index — a longer log with an older last term loses to a shorter log with a newer one, because a newer term means the entry was appended by a more recent leader.
A candidate that collects grants from a majority becomes leader for that term and immediately begins heartbeating, which suppresses any other node’s election timer. In the common case this is one round trip and the cluster is stable again in well under a second.
- N1 — lastLogTerm=7, lastLogIndex=104
- N2 — granted
- N3 — granted
- N4 — no reply
- N5 — refused: its lastLogIndex=106
- n1believes “I have a majority (self + N2 + N3)”✓ and it is true
- n1believes “N4 has crashed”✕ and it is false
- n5believes “N1 must not lead — my log is more complete”✓ and it is true
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Why the winner cannot be missing committed data
This is the argument worth carrying away, because it is short and it is the reason Raft needs no special recovery logic. An entry is committed only once it is stored on a majority. A candidate needs votes from a majority. Any two majorities of the same cluster intersect in at least one node.
So for any committed entry, at least one voter in any successful election must hold it. That voter will refuse to vote for a candidate whose log is behind its own. Therefore a candidate missing a committed entry cannot assemble a majority, and any node that does win already holds every committed entry. This is the Leader Completeness Property, and it is why leadership change never loses acknowledged data.
Note what this argument does not need: no clock, no accurate failure detection, no knowledge of who was leader before, and no reconciliation phase. It is quorum intersection plus one comparison rule.
entry X committed => stored on >= 3 of {N1..N5}
election won => votes from >= 3 of {N1..N5}
|A| >= 3, |B| >= 3, |A ∪ B| <= 5 => A ∩ B ≠ ∅
so: some voter V holds X.
V grants a vote only to a candidate whose log is >= V's.
=> the winner holds X. (for every committed X)Safety versus liveness — the distinction Raft makes explicit
Safety means nothing incorrect ever happens: no two leaders in a term, no committed entry lost, no two nodes applying different commands at the same index. Raft guarantees safety always — under arbitrary message loss, delay, reordering and duplication, arbitrary crashes, and arbitrarily bad clocks. There is no timeout setting and no network condition that can make Raft unsafe.
Liveness means progress eventually happens: a leader is elected, entries are committed, clients get answers. Raft guarantees liveness only under favourable conditions — a reachable majority, message delays that are eventually bounded, and election timeouts comfortably larger than the round-trip time.
Keeping these apart is the practical skill. When someone asks "is it safe to lower the election timeout to 50 ms?", the answer is: it cannot make the system incorrect, and it will probably make it stop making progress. Almost every Raft tuning question is a liveness question wearing a safety costume, and answering it as a safety question leads to needless caution in one place and misplaced confidence in another.
| Condition | Safety | Liveness |
|---|---|---|
| Messages lost, delayed, reordered, duplicatedprotocol | Intact | Degraded — may stall |
| Election timeout far too lowprotocol | Intact | Broken — election storms |
| Clocks wildly wrongprotocol | Intact | Degraded — bad timer behaviour |
| Majority unreachableprotocol | Intact | Broken — no progress at all |
| Term/vote not durably persistedassumption | **Broken** — outside the model | n/a |
Vote splits, and the fix that is just randomness
If several followers time out at the same instant, each becomes a candidate in the same term and votes for itself. With five nodes and three candidates, no one reaches three votes. Each times out again, increments the term, and repeats. The cluster is fully healthy and produces nothing.
Raft’s answer is to draw each node’s election timeout uniformly from a range — commonly 150–300 ms in a single datacentre. The chance that two nodes fire within one round trip of each other becomes small, and a split that does occur is resolved by the next round because the ranges are re-drawn. It is the same insight as Without Jitter, Every Client That Failed Together Retries Together: the failure is caused by synchronisation, so the fix is to desynchronise.
The tuning rule of thumb is broadcastTime << electionTimeout << MTBF. If the election timeout is not comfortably larger than a round trip, healthy leaders get deposed by normal jitter; if it is enormous, failover is slow. Neither end of that range affects correctness.
Key points
- Three states — Follower, Candidate, Leader — and a single overriding rule: seeing a higher term makes you a follower.
- A vote is granted at most once per term, and only to a candidate whose log is at least as up to date as the voter’s.
- Majority overlap plus that vote rule gives Leader Completeness: the winner already holds every committed entry.
- Safety holds unconditionally; liveness holds only under a reachable majority and eventually-bounded delays.
- Vote splits are prevented by randomised election timeouts, not by any protocol subtlety.
- Term and vote must be persisted before being acted on — the one way to make Raft unsafe.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Each follower runs an election timer with a randomised duration; a leader’s heartbeat resets it.
- • On expiry: term += 1, vote for self, persist (term, votedFor), send
RequestVote(term, lastLogIndex, lastLogTerm)to all peers. - • A peer grants iff it has not voted this term and the candidate’s log is at least as up to date; it persists its vote before replying.
- • On a majority of grants the candidate becomes leader and sends empty
AppendEntriesas heartbeats immediately. - • A candidate receiving
AppendEntriesfrom a leader with a term >= its own reverts to follower. - • On expiry without a winner, the candidate starts a fresh term and campaigns again, with a newly drawn timeout.
- •
RequestVotemessages are lost, so a candidate never reaches a majority despite being a legitimate winner. - • Vote grants are lost, so a candidate that did win never learns it and disrupts the cluster with another election.
- • Heartbeats are delayed past the election timeout on a healthy leader, causing an unnecessary election.
- • Several nodes time out together and split the vote repeatedly.
- • A node partitioned away campaigns repeatedly, inflating its term, then rejoins and deposes the healthy leader.
- • Persisted term or vote is lost across a restart, permitting two votes in one term.
- • Election storm: the operator sees the term counter climbing continuously, zero committed entries, and every node logging "starting election". Cause is almost always an election timeout below the real network round-trip spread, or timeouts that are not randomised.
- • Disruptive rejoin: a node isolated for ten minutes returns with a term hundreds higher and immediately deposes a healthy leader. The operator sees a term jump and a brief throughput dip at the exact moment connectivity was restored. Pre-vote eliminates this.
- • Leadership flapping on a saturated node: the operator sees leadership alternating between two members every few seconds, p99 latency in seconds, and disk or CPU pegged on the member that keeps winning because its log is most complete.
- • No leader after rolling restart: nodes restart faster than an election completes and each loses its heartbeat window. The operator sees every process up, every process a follower, and no commits for minutes.
- • Two leaders in term T in the logs: impossible under the protocol, therefore proof that a node lost its persisted vote. The operator should look for ephemeral storage or a disabled fsync, not for a protocol bug.
- • One majority round trip per election, then heartbeats at a fixed interval to retain authority.
- • Heartbeat traffic is O(n) per interval from the leader; it is the standing cost of keeping the agreement current.
- • Election cost is not the round trip but the gap: nothing commits between the old leader stopping and the new one starting.
- • Committed entries always survive an election, by Leader Completeness.
- • Uncommitted entries may be discarded — a client that received no acknowledgement has no claim.
- • With no majority reachable, no leader is elected and no writes are accepted; reads from followers remain possible but are stale.
- • Detect: graph term-per-member and elections-per-hour; alert on time-without-leader rather than on node liveness.
- • Contain: enable pre-vote, randomise timeouts, and use leadership transfer before planned maintenance.
- • Recover: restore majority connectivity; election and catch-up are automatic.
- • Reconcile: the new leader overwrites divergent follower tails via the
AppendEntriesconsistency check — see The Raft Log: Commit Index, Divergence and Reconciliation. - • Verify: one leader, converged terms, identical commit indexes, and durable storage confirmed on every member.
- • Term per member on one graph; divergence and rate of climb are both visible at a glance.
- • Elections per hour, and the identity of the node that triggers them.
- • Time-without-leader, integrated.
- • Per-follower heartbeat acknowledgement latency at p99 — the leading indicator of the next election.
- • Whether pre-vote is enabled, and whether the data directory is durable.
- • When you need automatic, correct failover for a replicated state machine and want an algorithm your on-call engineers can actually reason about.
- • When the operations authorised per election are numerous — the amortisation is what makes leader-based consensus efficient.
- • When you need a defensible answer to "could we have lost an acknowledged write?" — Leader Completeness is that answer.
- • Write-heavy workloads spread across regions: every write funnels through one node and pays a cross-region majority.
- • Environments where pauses are routine, since each pause costs an election and each election is a global stall.
- • Very large clusters, where the majority grows and every decision waits for more acknowledgements.
- • Multi-Paxos, which reaches the same guarantees with a different decomposition and no requirement that logs stay contiguous — see Paxos and the Other Protocols: What They Share and Where They Differ.
- • A managed coordination service so you consume elections instead of implementing them — see Coordination Services: The Primitives, Not the Product.
- • Static primary with manual failover: no elections, no split votes, a human as the failure detector, and much simpler operations at the cost of recovery time.
- • Partitioned ownership so each range elects independently, which bounds the blast radius of any one election.
Raft elections: three states, two rules
What people believe, and what is true
The node with the most log entries wins.
Comparison is by last entry’s *term* first, then index. A shorter log whose last entry came from a newer term is more up to date.
A shorter election timeout gives higher availability.
Below the network’s latency spread it manufactures elections that would not otherwise occur, and each election is a stall.
Raft guarantees the cluster keeps working.
It guarantees the cluster never does anything incorrect. Working requires a majority, which Raft cannot provide.
A newly elected leader has all the data.
It has all *committed* data. It may also hold uncommitted entries that will be discarded, and followers may still be catching up.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Nodes are followers until they stop hearing from a leader, then campaign for a numbered term. A majority of votes wins. Voters refuse candidates whose logs are behind, so the winner always has every committed entry.
Practical
Randomise election timeouts, size them well above round-trip time, enable pre-vote, and use leadership transfer for planned maintenance. Measure elections-per-hour and time-without-leader. Put term and vote on durable storage, and treat "two leaders in one term" in the logs as a storage bug rather than a protocol bug.
Advanced
The vote rule plus quorum intersection is the whole of Leader Completeness, and it is why Raft has no separate recovery protocol: the election *is* the recovery. Compare this with Viewstamped Replication, which reaches the same result with an explicit view-change phase that transfers state, and with Multi-Paxos, where a new leader must run a phase-1 round to learn what may already have been chosen at each position.
Internals
Pre-vote adds a pre-flight round in which a candidate asks whether peers *would* grant a vote, without incrementing any term. A node isolated for a long time therefore cannot return with an inflated term and depose a healthy leader; it discovers it is behind and rejoins quietly. CheckQuorum is its companion: a leader that has not heard from a majority within an election timeout steps down voluntarily, so it stops serving before anyone else has to reject it. Neither changes safety — both exist purely to stop the protocol disrupting itself.
Apply it
- 🔧 Construct a data-loss scenario that occurs if the log-completeness vote rule is removed, and identify the exact write that is lost.
- 🔧 Explain why pre-vote is a liveness feature and not a safety feature.
- ⚡ Five nodes; N1 leads term 7. N1 is paused for 8 seconds by GC. Describe what each of the other four does, what N1 does when it resumes, and what a client connected to N1 experiences throughout.
- 💬 Walk me through a Raft election from timeout to first heartbeat.
- 💬 Why does Raft compare last log term before last log index?
- 💬 Prove that a newly elected leader cannot be missing a committed entry.
- 💬 Is it safe to set the election timeout to 20 ms? Answer in terms of safety and liveness separately.