The question this answers
How does a cluster choose exactly one node to act — and how does that node know it still may?
At most one leader per term, always. Not "at most one leader at a time": two nodes may simultaneously believe they lead, in different terms, and the protocol guarantees only that the older term can commit nothing. Exactly one leader exists *eventually*, and only while a majority is reachable.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A leader knows when it last received acknowledgements from a majority. It does not know that it is the leader right now — leadership is a fact about the cluster, and the cluster is exactly what it cannot see. A candidate that receives no votes does not know whether it lost, whether its requests were dropped, or whether the replies were dropped; all three are silence. A follower knows only that no heartbeat has arrived recently, which is not the same as "the leader is dead".
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Why a leader at all
A leader is a coordination *shortcut*. Instead of running an agreement round for every operation, the cluster runs one agreement round to pick a leader, and then the leader orders everything by itself. A single node ordering operations is trivially consistent — it is just a program with a variable — so the expensive part happens once per leadership term rather than once per request.
That is a very good bargain when leadership is stable: one election per week authorising a billion operations. It is a very bad bargain when leadership flaps, because every election is a full stop for the whole system. This is why the practical measure of a leader-based system is not election speed but election frequency.
The leader is also the reason Leader-Based Replication: Buying Order With a Single Writer can offer a simple ordering story, and the reason a leader-based system has a single point of *throughput* even when it has no single point of failure.
The shape of every election protocol
Implementations differ in detail, but the skeleton is fixed. A node suspects the leader is gone, nominates itself under a new, higher generation number, collects votes from a majority, and begins acting. Every step is a decision that must survive the network being unhelpful.
The subtle rule is the one about voting at most once per term. Without it, two candidates could each collect a majority in the same term by asking the same nodes twice, and the whole safety argument evaporates. That single-vote record must be durable before the vote is sent — a node that forgets its vote after a restart can vote twice for the same term, which is the Consensus Is Not Magic: The Assumptions It Runs On durability requirement in its most concrete form.
1on election_timeout:2 currentTerm += 1 # new generation3 votedFor = self4 persist(currentTerm, votedFor) # MUST be durable BEFORE any request is sent5 votes = 16 for peer in cluster:7 send RequestVote(term=currentTerm, candidate=self, lastLogIndex, lastLogTerm)8 9on RequestVote(term, candidate, lastLogIndex, lastLogTerm):10 if term < currentTerm: return Reject(currentTerm) # stale candidate11 if term > currentTerm: currentTerm = term; votedFor = nil; step_down()12 if votedFor in (nil, candidate) and candidate_log_at_least_as_new():13 votedFor = candidate14 persist(currentTerm, votedFor) # again: durable first15 return Grant(currentTerm)16 return Reject(currentTerm)17 18on receiving votes from a MAJORITY of the configured membership:19 become_leader() # note: of THIS term onlyThe elected node cannot verify its own election
Here is the asymmetry that produces most leader-related incidents. When a node wins, it learns so by counting votes — a fact about the past. Nothing tells it when that fact expires. It keeps behaving as leader until *it* notices something, and the thing it notices is again an absence: peers stopped acknowledging.
So a leader partitioned away from the cluster continues to believe it leads for at least one timeout period, and possibly much longer if it is paused rather than partitioned. During that window the rest of the cluster has already elected a successor. Two leaders, both sincere. This is Split-Brain: Two Nodes, Both Certain They Are In Charge, and the protocol’s answer is not to prevent the belief but to make the stale leader’s actions ineffective — see Terms and Epochs: Making Stale Leaders Harmless.
The practical corollary: a leader must never act on leadership alone for anything outside the protocol. Writing to the replicated log is safe because the log rejects stale terms. Writing to an external object store, sending an email, or taking a lock is not safe, because those systems have never heard of terms. That is the gap Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely closes.
- N1 — last majority ack: 4s ago
- n1believes “I am the leader of term 7”✓ and it is true
- n1believes “I will still be the leader when my next write lands”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Elections that never finish
The second common pathology is the opposite of a stale leader: no leader at all. If several followers time out simultaneously, they all become candidates in the same term, split the vote, and none reaches a majority. Each then times out again, increments the term again, and repeats. The cluster is healthy, the network is fine, and throughput is zero.
The fix is embarrassingly simple and worth remembering because it generalises: randomised timeouts. Give each node an election timeout drawn from a range rather than a constant, and the probability that two nodes start in the same window collapses. It is the same jitter argument as Without Jitter, Every Client That Failed Together Retries Together, applied to leadership rather than to retries.
The third pathology is the flapping leader: a node that is slow but not dead keeps getting deposed, wins again because its log is the most complete, and is deposed again. The symptom is a term counter climbing steadily with occasional commits between elections.
Key points
- A leader is a coordination shortcut: one agreement round per term instead of one per operation.
- The guarantee is at most one leader *per term*, not at most one leader at a time.
- A leader learns it was elected; it can never learn that it still is.
- A node must vote at most once per term, and that vote must be durable before it is sent.
- Split votes are prevented by randomised election timeouts, not by cleverness.
- Leadership authorises actions inside the protocol only; external side effects need fencing.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Followers expect periodic heartbeats from the current leader.
- • A follower that hears nothing for its (randomised) election timeout increments the term and becomes a candidate.
- • The candidate votes for itself, durably records term and vote, and requests votes from all peers.
- • Peers grant a vote if they have not voted in this term and the candidate’s log is at least as up to date as their own.
- • A candidate with a majority becomes leader for that term and begins sending heartbeats, which suppress further elections.
- • Any node that sees a higher term than its own immediately steps down and adopts it — including a leader.
- • Vote requests are lost, so a legitimate candidate never reaches a majority.
- • Vote grants are lost, so a candidate that *did* win never learns it and starts another election.
- • Heartbeats are delayed past the election timeout, deposing a perfectly healthy leader.
- • Two or more candidates start in the same term and split the vote indefinitely.
- • A leader is paused by garbage collection or descheduling for longer than the timeout and returns believing it still leads.
- • Election storm: the operator sees the term counter climbing tens per minute, commit throughput near zero, and every node logging "starting election". Usually a too-short timeout or non-randomised timeouts, not a failing node.
- • Zombie leader: a partitioned leader keeps accepting client writes locally and returns success for operations that will never commit. The operator sees a node reporting itself leader with a term lower than the rest of the cluster, and clients whose writes vanish after the partition heals.
- • Leadership flapping on a slow node: the operator sees leadership oscillating between two members every few seconds, p99 write latency in the seconds, and disk or CPU saturation on the node that keeps winning.
- • No leader after a rolling restart: nodes restart faster than they can complete an election, each losing its heartbeat window. The operator sees a cluster that is fully up but has no leader for minutes.
- • Double vote after a crash: a member with non-durable vote state grants two votes in one term. The operator sees two leaders in the same term in the logs — a state the protocol says is impossible, and therefore proof that an assumption was broken.
- • One majority round trip per election. Everything after it is leader-local until the next election.
- • Heartbeats are continuous coordination at low cost: they are what convert "we agreed once" into "we still agree".
- • The cost of an election is not the round trip but the pause: no operation commits between the old leader stopping and the new one starting.
- • Committed entries survive any election — the vote rule requires a candidate’s log to be at least as up to date as the voter’s, so a node missing committed entries cannot win.
- • Uncommitted entries from a deposed leader may be discarded; a client that saw no acknowledgement must not assume they survived.
- • With no majority, the cluster has no leader and accepts no writes; it does not elect a leader on a minority side.
- • Detect: track "time without a leader" as a first-class metric; it is the true availability signal for a leader-based system.
- • Contain: have clients fail fast rather than queue against a leaderless cluster, and refuse writes at a node that believes it leads but has not heard from a majority recently.
- • Recover: restore connectivity or capacity; election is automatic and needs no operator action once a majority can talk.
- • Reconcile: the new leader forces its log onto followers, truncating uncommitted divergent tails. See The Raft Log: Commit Index, Divergence and Reconciliation.
- • Verify: after recovery, confirm one leader, one term, and identical commit indexes across members.
- • Leader identity and term over time — a graph that should be almost flat.
- • Elections per hour; anything above single digits is a problem to explain.
- • Time-without-leader, integrated over the day.
- • Per-follower heartbeat acknowledgement latency, which predicts which node will trigger the next election.
- • Whether any node reports itself leader while the majority reports someone else — the split-brain alarm.
- • When operations must be ordered and a single ordering authority is far cheaper than agreeing per operation.
- • When a resource must have exactly one owner — a shard, a scheduler, a compaction job, a cron.
- • When failover must be automatic and correct without a human choosing which replica was ahead.
- • When the workload is write-heavy and geographically spread: every write goes to one node, which becomes both a latency floor and a throughput ceiling.
- • When elections are frequent, because each one is a global pause.
- • When the "leader" performs external side effects, since leadership does not extend to systems outside the protocol.
- • Partitioned ownership: give each key range its own single owner so there is no global leader at all. Ownership assignment still needs agreement, but only rarely. See Cross-Partition Operations: Paying for What the Split Took Away for what this costs.
- • Leaderless quorum writes, where no node is special and conflicts are resolved after the fact — see Leaderless Replication: Every Replica Accepts Writes.
- • A static, human-assigned primary with manual failover: no election protocol, no split votes, and a person as the failure detector. Slower to recover, dramatically simpler to reason about.
- • An external lock service (etcd, ZooKeeper) holding the leadership lease, so your service does not implement consensus at all — see Coordination Services: The Primitives, Not the Product.
Electing one node — and knowing you are still it
What people believe, and what is true
Leader election guarantees there is only ever one leader.
It guarantees only one leader per term. Two nodes can believe they lead simultaneously; the older term simply cannot commit.
If a node was elected, it is the leader until it crashes.
It is the leader until a majority stops acknowledging it — which can happen without the node noticing, and without it crashing.
Faster election timeouts mean higher availability.
Below the network’s real latency spread they cause elections that would not otherwise happen, and each election is downtime.
The leader always has the most recent data.
It has all *committed* data, by the vote rule. It may be missing nothing, but a freshly elected leader can still be replicating a backlog, and it may hold uncommitted entries that will be discarded.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
One node is elected to act for everyone, so ordering becomes trivial. It is elected by majority vote for a numbered term, and every term has at most one leader.
Practical
Randomise election timeouts, persist term and vote before acting on them, and measure elections-per-hour rather than election speed. Treat "time without a leader" as your availability metric. Never let leadership alone authorise a side effect outside the cluster.
Advanced
The vote-granting rule is the safety hinge: a voter refuses a candidate whose log is behind its own. Combined with majority overlap, this guarantees any new leader’s log already contains every committed entry, so leadership change never loses committed data. Liveness, by contrast, rests entirely on randomised timeouts, which is why it can be tuned badly without ever becoming unsafe.
Internals
Two refinements matter in production. Pre-vote: before incrementing its term, a candidate asks whether peers *would* vote for it; this stops a partitioned node from returning with an inflated term and deposing a healthy leader. Leadership transfer: a leader going down for maintenance hands leadership to a caught-up follower explicitly, converting an election-shaped outage into a sub-millisecond handover. Both address disruption, not safety.
Apply it
- 🔧 Explain why a candidate whose log is behind must be refused, and construct the data loss that occurs if that rule is dropped.
- 🔧 Design a "am I still leader?" check a node can perform locally, and state precisely what it does and does not prove.
- 💬 Walk me through a leader election. What exactly makes two leaders in the same term impossible?
- 💬 Your cluster elects a new leader every four seconds. Where do you look first?
- 💬 A leader is partitioned away. What is it allowed to keep doing, and what must it stop doing?