The question this answers
Why do most replicated systems funnel every write through one node, when that node is obviously a bottleneck and a single point of failure?
All writes to a replicated object are totally ordered by whichever node is leader for that object, and every follower applies that same order. A read on the leader reflects every write the leader has acknowledged; a read on a follower reflects some *prefix* of that order, with no bound on how short the prefix is unless one is imposed.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A follower knows the position it has applied up to and the last message it received from the node it believes to be leader. It does not know whether that node is still leader, whether a new leader has been elected, or how far the current leader has advanced. A leader knows it *was* elected; it does not know it still holds the role — that is why a leader that has been partitioned away keeps confidently serving reads. See Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The order is the product
It is tempting to describe leader-based replication as "the leader has the data and the followers copy it". The more useful description is: the leader is a serialisation point. Concurrent writes arriving from many clients are given a single, definite order by the fact that one node decided the order, and every other copy then replays that order.
This is why a single writer is not merely a scaling compromise you tolerate — it is doing real work. Without it, two writes to the same key have no natural order, and the system must either invent one (see Last Write Wins Is Data Loss You Chose by Default, which loses data) or keep both and ask the application (see Only the Application Knows What the Merge Means). Multi-leader and leaderless designs pay exactly this price.
- node-a — accepts all writes; assigns positions
- node-b — applied position 4,102
- node-c — applied position 3,880
- f2believes “I am nearly current”✕ and it is false
- lbelieves “I am the leader”✓ and it is true
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Where reads are served decides what you have promised
The replication topology does not determine the guarantee a user experiences — the *read routing policy* does. The same leader-and-followers cluster gives you three different systems depending on where a read may land.
This is the single most consequential configuration decision in a replicated system, and it is usually made by whoever set up the connection pool.
| Read routing | What a reader can observe | What breaks |
|---|---|---|
| Leader onlytypical | Every acknowledged write, in order | The leader carries all load; followers are pure standby capacity |
| Any replicaprotocol | Some prefix of the write order, arbitrarily old | Read-your-writes and monotonic reads both, immediately |
| Follower, but leader for a user's own recent writestypical | Own writes always visible; other users' writes may lag | Requires tracking a per-session position — see [[read-after-write]] |
Failover is the hard part, and it is a different problem
Everything above is straightforward while the leader lives. The difficulty is concentrated entirely in the transition: the leader stops responding, and something must decide that a new leader exists. That decision is not a replication problem, it is a consensus problem — which is why leader-based replication is easy to describe and hard to operate.
Three questions have to be answered correctly and they are answered by different mechanisms. Is the old leader actually gone, or merely slow? — nobody can know, so this is a timeout, which means it is a guess (Crashed or Just Slow: The Distinction You Cannot Make). Which follower should be promoted? — ideally the one with the longest log, but that requires comparing positions across nodes that may not all be reachable. How do we stop the old leader acting if it comes back? — this is Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely, and skipping it is how you get Split-Brain: Two Nodes, Both Certain They Are In Charge.
node-a 12:04:31 WARN follower node-b heartbeat timeout (3s) node-a 12:04:31 INFO continuing as leader, quorum not required for reads node-c 12:04:33 INFO leader node-a unreachable, starting election (term 8) node-c 12:04:34 INFO elected leader, term 8, log position 4,102 node-a 12:04:36 INFO accepted write, position 4,103 <-- still term 7 node-c 12:04:37 INFO accepted write, position 4,103 <-- different write, same position node-a 12:04:52 INFO rejoined cluster, stepping down (term 8) node-a 12:04:52 WARN truncating 1 uncommitted entry
Key points
- The leader's real function is to impose a total order on writes without running an agreement protocol per write.
- Followers apply that order; a follower read is a prefix of it, not a stale snapshot of a moment.
- Read routing, not topology, determines what guarantee a user actually gets.
- A leader cannot know it is still the leader — only that it was elected. That gap is where split-brain lives.
- Failover is a consensus problem wearing a replication costume: is it gone, who is next, and how is the old one stopped.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • All write requests are routed to the leader for the object or partition.
- • The leader assigns each write a position in a durable ordered log and applies it locally.
- • The leader streams log entries to each follower; each follower applies them in order and reports its applied position back.
- • The leader acknowledges the client either immediately (asynchronous) or after enough followers report the entry (synchronous / quorum).
- • On leader loss, a promotion procedure elects a follower, which must reconcile any entries it lacks and reject writes from the previous leader.
- • The leader crashes with acknowledged entries not yet on any follower.
- • The leader is partitioned from followers but still reachable by clients, and keeps accepting writes.
- • A follower is promoted while behind, silently discarding the entries it never received.
- • Two nodes believe they are leader for the same object in different terms.
- • The replication stream disconnects and reconnects repeatedly, so lag oscillates rather than converging.
- • Split-brain writes: two nodes accept writes for the same key concurrently. The operator observes duplicate primary keys, or two versions of a row that "cannot both exist", and no error in either log.
- • Lost acknowledged writes after promotion: the customer has a confirmation email for an order that is not in the database. The write-loss window equals the replication lag at the moment of failure.
- • Failover that does not fire: the leader is slow rather than dead, health checks pass at the TCP level, and the system is fully unavailable to writers while every monitor reports the cluster healthy.
- • Flapping promotion: a marginal network causes repeated elections; the operator sees term numbers climbing steadily and write availability collapsing between elections.
- • Read-your-write violations at scale: after moving reads to followers, support volume rises for "my change did not save" with no corresponding error rate.
- • None per write in the steady state — this is the design's central economy. The leader decides alone.
- • Coordination is concentrated entirely in leader election and in the fencing that follows it, so its cost appears as an availability gap during failover rather than as latency per request.
- • Synchronous acknowledgement adds a per-write wait on followers — that is Synchronous Replication: Paying Latency for a Durability Guarantee, and it is a separate decision from being leader-based.
- • While a leader exists, ordering is preserved regardless of how far followers lag.
- • With no leader, the system is unavailable for writes but may still serve stale reads from followers.
- • After an unsafe promotion, ordering is preserved *going forward* but a suffix of the previous history has been discarded; the system is internally consistent and externally wrong.
- • Detect: distinguish "leader unreachable from monitoring" from "leader unreachable from a quorum of followers" — only the latter should trigger promotion.
- • Contain: fence the old leader before the new one accepts writes, using an epoch or term that storage and clients both check. Never rely on the old leader noticing.
- • Recover: promote the follower with the highest applied position among those reachable, and record which positions were lost.
- • Reconcile: replay or compensate the lost window against upstream systems that were told those writes succeeded.
- • Verify: run a scripted failover on a schedule. An untested promotion path is a second outage waiting behind the first.
- • Current term or epoch number, and its rate of change — a climbing term is the earliest signal of election flapping.
- • Per-follower applied position gap relative to the leader, so the write-loss window is a number you know before the failover rather than after.
- • Time to promote, measured end to end from last successful write to first successful write on the new leader.
- • Count of writes rejected due to fencing — a non-zero count means an old leader is still trying, which is exactly what you want to see caught.
- • Whether any client is still routing writes to the demoted node after promotion.
- • Workloads with a natural single writer per object, which is most transactional application data.
- • Any system that needs a definite order of writes without paying an agreement round trip on each one.
- • Teams that want conflict resolution to be *unnecessary* rather than correct — a single writer means conflicts cannot arise.
- • Write throughput beyond one node — the answer here is partitioning, so each partition has its own leader, not abandoning leadership.
- • Writers spread across regions, where every write pays the round trip to a distant leader. See Three Ways to Accept a Write in More Than One Place.
- • Requirements for write availability during a partition, which a single leader cannot provide to the minority side by construction.
- • Environments where the failover procedure cannot be tested, making the promotion path less trustworthy than the leader it replaces.
- • Partition the data and give each partition its own leader — the standard way to scale writes while keeping a single writer per object. See Hash Partitioning and the Modulo Trap.
- • Multi-leader replication if writers genuinely must be accepted in more than one location, accepting conflicts as the price. See Multi-Leader Replication: Accepting Writes in More Than One Place.
- • Leaderless replication if you would rather trade a definite order for uniform write availability. See Leaderless Replication: Every Replica Accepts Writes.
- • A consensus-replicated log (Raft) if you want leader-based replication where the promotion itself is safe by construction rather than by procedure. See The Raft Log: Commit Index, Divergence and Reconciliation.
Failover, and the gap a leader cannot see
- n1 — log 0 entries · committed 0
- n2 — log 0 entries · committed 0
- n3 — log 0 entries · committed 0
n1 follower term 0 — n2 follower term 0 — n3 follower term 0 — [x] committed (x) replicated but not committed
What people believe, and what is true
The leader always has the most recent data.
The *current* leader does, by definition of what it accepted. A node that believes it is leader and has been partitioned away has stale data and full confidence, which is worse than having neither.
Automatic failover makes the system more available.
It reduces downtime when the leader truly fails and creates a new outage class when it merely stalls. Failover without fencing converts a partial outage into data corruption.
A single leader is a bottleneck we should design away.
The single writer is buying you the absence of write conflicts. Removing it does not remove the work — it relocates it into conflict resolution, which is strictly harder.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
One node accepts writes and orders them; the others replay that order. The order, not the copying, is the product.
Practical
Decide read routing deliberately: leader-only for correctness-critical reads, followers for reporting and browse paths, and a session-pinned hybrid where users read their own writes. Then test the failover, because the promotion path is where every real incident happens.
Advanced
A leader is a lease over the right to order writes. Because no node can detect its own lease expiry reliably (Crashed or Just Slow: The Distinction You Cannot Make), safety cannot come from the old leader stepping down politely — it must come from the *storage or the followers* rejecting stale terms. Leader-based replication is therefore only as safe as its fencing, and this is the single most commonly skipped part of the design. See The Stale Lock Holder: A Paused Process Does Not Know It Was Paused and Leases: Authority With an Expiry Date.
Apply it
- 🔧 Given a cluster where the leader has position 4,103 and the two followers have 4,102 and 3,880, state exactly what is lost under each possible promotion, and what the fencing must prevent.
- 💬 Your leader stops responding to health checks but is still accepting writes from an application server on the same rack. What happens if you promote?
- 💬 Why is the single writer in leader-based replication doing useful work rather than just being a bottleneck?
- 💬 You move 80% of reads to followers and the error rate does not change, but support tickets rise. What is happening?