The question this answers
When is an entry actually committed, and what happens to entries a deposed leader wrote but never committed?
Log Matching: if two logs contain an entry with the same index and term, the logs are identical in every entry up to that index. State Machine Safety: if a node has applied an entry at index i, no other node ever applies a different entry at index i. An entry is committed — durable and never revocable — exactly when it is stored on a majority *and* was appended in the current leader’s term. Entries below that bar may be discarded without notice.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A follower knows the entries it holds and the commit index the leader last told it about; it does not know whether its most recent entries are committed, and it must not apply them until told. The leader knows which entries each follower has acknowledged — as of the last reply it received, which may be stale. A client whose request timed out knows nothing about whether its entry was committed, and there is no local check either party can perform to find out. Commitment is a property of the cluster, learned by counting, never observed.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Local append is not a commit
The leader receives a client command, appends it to its own log at the next index, stamped with the current term, and sends AppendEntries to every follower. At this moment the entry exists on exactly one machine and is worth nothing. If the leader crashes now, a successor may — legitimately, correctly — erase it.
The entry becomes committed when the leader has received acknowledgements from a majority, itself included. Only then may the leader advance its commitIndex, apply the entry to its state machine, and answer the client. This ordering is the whole of the durability contract: acknowledge on commit, never on append. An implementation that returns success at append time will lose acknowledged writes on failover, and it will do so silently.
The consequence for clients is uncomfortable and unavoidable. A client that times out cannot know whether its entry committed; it may commit seconds later. This is A Timeout Tells You Nothing About Whether It Happened at the storage layer, and the answer is the same: make the operation identifiable and retryable rather than trying to learn the outcome.
The commit index, and the rule that surprises people
commitIndex is the highest index known to be committed. The leader computes it by looking at what a majority has acknowledged, then piggybacks it on the next AppendEntries so followers learn how far they may apply. Followers never compute it themselves — commitment is decided by the leader and propagated.
The rule that catches everyone: a leader may not commit an entry from a previous term merely because it is now on a majority. Raft only advances commitIndex past an entry once an entry *from the leader’s own current term* has been replicated to a majority. Counting replicas of an old-term entry is not sufficient, because a subsequent leader could still overwrite it in a way that would contradict a commit already announced.
In practice a new leader appends a no-op entry in its own term immediately after election. Once that no-op commits, everything before it commits with it. If you have ever wondered why a freshly elected Raft leader writes an empty entry, this is why — and it is not an optimisation, it is required for safety.
leader state (5 nodes, currentTerm = 4):
nextIndex[] = { N2: 7, N3: 4, N4: 7, N5: 4 } # what to send next
matchIndex[] = { N2: 6, N3: 3, N4: 6, N5: 3 } # highest known replicated
replicated-on-majority = 3rd largest of {6 (self), 6, 3, 6, 3} = 6
commit rule: advance commitIndex to 6
ONLY IF log[6].term == currentTerm (== 4)
otherwise wait for a current-term entry to replicateDivergence: how logs come apart
Picture a leader in term 4 that accepts three client writes at indexes 7, 8 and 9, replicates none of them, and then crashes — or is partitioned, which is worse because it keeps accepting writes. Those three entries exist on one node and nowhere else.
Meanwhile N2 and N3 elect N2 in term 5. N2 is legitimate: by Leader Completeness it holds every *committed* entry, and entries 7–9 were never committed, so their absence is not a problem. N2 accepts its own client writes at indexes 7, 8 and 9, stamped with term 5.
Now two logs disagree at the same indexes with different terms. Both nodes behaved correctly. No client was misled, because nobody was ever told entries 7–9 in term 4 had succeeded. Divergence in uncommitted tails is a normal, expected state, not a bug — and the protocol must resolve it without human involvement.
Reconciliation: the consistency check that repairs everything
Every AppendEntries carries prevLogIndex and prevLogTerm — the entry immediately before the ones being sent. A follower accepts only if it has an entry at prevLogIndex whose term matches. If it does not, it rejects, and the leader decrements nextIndex for that follower and tries again with an earlier point.
This walks backwards until leader and follower agree on a common prefix. From there the leader sends everything after it, and the follower truncates its divergent tail and adopts the leader’s entries. The old leader’s three term-4 entries are deleted, unread and unmourned.
Two properties make this safe rather than reckless. First, Log Matching: agreement on one (index, term) implies agreement on the entire prefix, so a single matching point proves the whole history matches — that is why a backwards search is sufficient. Second, Leader Completeness: the leader doing the overwriting already holds every committed entry, so truncation can only ever destroy entries that were never committed and never acknowledged. The leader’s log is the log, and that is a safe thing to say only because of how leaders are chosen.
1# leader -> follower2AppendEntries(term, prevLogIndex, prevLogTerm, entries[], leaderCommit)3 4# follower5if term < currentTerm: return Reject(currentTerm)6if log has no entry at prevLogIndex: return Reject(hint=len(log))7if log[prevLogIndex].term != prevLogTerm: return Reject(hint=first index of that term)8 9# prefix agrees from here down (Log Matching)10for (i, e) in entries:11 if log has entry at prevLogIndex+1+i with a different term:12 truncate(from = prevLogIndex+1+i) # discard divergent tail13 append(e)14 15commitIndex = min(leaderCommit, index of last new entry)16apply committed entries in index order17return Ok18 19# leader, on Reject:20nextIndex[follower] -= 1 # or jump using the hint21retry # converges on the common prefixWhat this means for a client
The honest client-side contract is narrower than people assume. An acknowledged write is durable and will survive any number of leader changes. An unacknowledged write is in superposition: it may commit, it may be truncated, and there is no bound on when the question resolves — a partitioned leader’s entry can be truncated minutes later.
So a client that times out must not assume failure. It must retry with an identifier that lets the state machine recognise a duplicate, exactly as Idempotent Is a Property of the Whole Effect, Not the Write describes. And a read that must reflect all committed writes cannot simply be served by any follower: followers lag, and a follower’s commitIndex trails the leader’s. Linearizable reads require going through the leader with a confirmed quorum, or a read-index protocol — see Linearizability: An Operation Is an Interval, Not a Point.
Key points
- An entry on the leader’s disk is a proposal; an entry on a majority in the current term is a commit.
- Acknowledge clients on commit, never on local append.
- A leader may not commit a previous term’s entry by replica count alone — hence the no-op entry after election.
- Divergent uncommitted tails are normal after a leader change, not a defect.
- The
prevLogIndex/prevLogTermcheck walks back to a common prefix; the follower then truncates and adopts the leader’s log. - Truncation is safe because Leader Completeness means the leader already holds everything committed.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Client sends a command; the leader appends it locally with the current term at the next index.
- • The leader sends
AppendEntrieswithprevLogIndex/prevLogTermto every follower. - • Followers verify the previous entry matches, truncate any conflicting tail, append, and acknowledge.
- • The leader tracks
matchIndexper follower and computes the highest index replicated on a majority. - • If that index holds a current-term entry,
commitIndexadvances; the leader applies and answers the client. - • The new
commitIndexrides along on the nextAppendEntries, letting followers apply in the same order. - • On rejection, the leader decrements
nextIndex(or uses the follower’s hint) and retries earlier until the prefixes match.
- •
AppendEntriesis lost, so a follower falls behind and itsmatchIndexstalls. - • The leader crashes after appending locally but before replicating, orphaning entries.
- • The leader crashes after replicating to a majority but before answering the client — committed, but the client saw a timeout.
- • A follower crashes and restarts having lost entries it acknowledged, breaking the durability assumption.
- • A slow follower diverges far enough that the leader has already compacted the entries it needs, requiring a snapshot transfer instead.
- • Applying is not deterministic across nodes, so identical logs produce different state — a state machine bug that the protocol cannot detect.
- • Acknowledged writes lost on failover: an implementation acked on append. The operator sees clients reporting successful writes that are absent afterwards, and a truncation line in the old leader’s log at exactly those indexes.
- • One follower stuck far behind: the operator sees a flat
matchIndexfor one member, growing disk on the leader because compaction is blocked, and a cluster that will lose quorum if one more node fails despite all nodes being "up". - • Commit index frozen with a live leader: the leader cannot replicate a current-term entry to a majority. The operator sees the leader healthy,
commitIndexstatic, client writes timing out, and — the tell — the last log entry’s term lower thancurrentTerm. - • Snapshot storm: a follower is so far behind that the leader must ship a full snapshot; the transfer saturates the link and slows replication to everyone. The operator sees replication latency rising cluster-wide during a single member’s recovery.
- • Divergent applied state with identical logs: a non-deterministic state machine (map iteration order, a wall-clock read, a random seed). The operator sees identical
commitIndexon every member and different query answers — the worst of the set, because the protocol reports perfect health.
- • One majority round trip per commit, pipelined and batched so throughput is not one-commit-per-round-trip.
- • The commit latency is the latency of the *median* member of the fastest majority, so one slow follower is tolerated and two are not.
- • Reads are only cheap if you accept staleness; a linearizable read needs its own quorum confirmation and costs a round trip too.
- • Every committed entry survives any sequence of leader changes, by Leader Completeness.
- • Uncommitted entries may vanish at any later time, without notice and without an error to the client.
- • Log Matching holds at all times: any two logs agreeing at an (index, term) agree on the entire prefix.
- • With no majority,
commitIndexfreezes; nothing is lost and nothing progresses.
- • Detect: alert on
commitIndexnot advancing while a leader exists, and on per-follower replication lag in entries, not seconds. - • Contain: refuse to acknowledge before commit; fail fast when the leader cannot reach a majority rather than queueing.
- • Recover: the consistency check reconciles automatically; badly lagging followers are caught up by snapshot install.
- • Reconcile: divergent uncommitted tails are truncated by the protocol. Effects already emitted outside the state machine are not — they need Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely or idempotence.
- • Verify: confirm every member reports the same
commitIndexandlastApplied, and compare a state-machine checksum across members to catch non-determinism.
- •
commitIndexandlastAppliedper member on one graph. - • Per-follower
matchIndexgap from the leader, in entries — the real replication health signal. - • Commit latency at p99, which tracks the median majority member.
- • Snapshot install events and their duration.
- • A periodic state-machine hash comparison across members — the only way to catch non-deterministic apply.
- • When you need an ordered, durable sequence of operations that survives node loss with no manual failover.
- • When the state machine on top is deterministic, so replicating the log replicates the state exactly.
- • When you want a single explanation for both replication and recovery — in Raft they are the same code path.
- • When operations are independent and do not need a global order: you are paying for total ordering you never use.
- • When entries are large, since every entry crosses the network to every member and lands on every disk.
- • When one member is chronically slow, because it blocks compaction and erodes the failure margin without ever appearing down.
- • Per-partition logs so ordering is local to a key range and no global sequence exists — see A Topic Is Not One Log: Ordering Lives Inside a Partition and Cross-Partition Operations: Paying for What the Split Took Away.
- • Quorum writes without a log: no ordering guarantee, conflicts resolved after the fact. See Leaderless Replication: Every Replica Accepts Writes.
- • Asynchronous replication from a single primary: much faster, and bounded data loss on failover, which is the right trade for many systems. See Asynchronous Replication: The Loss Window You Chose.
- • Convergent data types that need no agreed order at all — see CRDTs: Deterministic Merge, Not Correct Merge.
Raft: five nodes, one log
Log divergence and truncation
What people believe, and what is true
Once the leader writes the entry, it is committed.
It is a proposal on one machine. A future leader may delete it, and no client will ever be told.
The client got a timeout, so the entry was not committed.
It may commit after the timeout. The client learns nothing from a timeout — see A Timeout Tells You Nothing About Whether It Happened.
Truncating a follower’s log means losing data.
Only uncommitted, unacknowledged entries are ever truncated. Committed entries cannot be, because the leader that overwrites necessarily holds them.
A follower can serve consistent reads because it has the log.
Its commitIndex lags the leader’s, so it may hold entries it must not apply and miss entries it has not received.
Replicating an old entry to a majority commits it.
Raft explicitly forbids that. A current-term entry must reach a majority first — hence the no-op after every election.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
The leader appends client commands to a log and copies them to followers. An entry counts as committed once a majority has it; only then is it applied and the client answered. Entries that never reached a majority can be deleted by the next leader.
Practical
Acknowledge on commit, not on append. Watch per-follower matchIndex gaps rather than node up/down. Expect log truncation after a leader change and treat it as normal. Keep the state machine deterministic — no clocks, no map iteration order, no randomness — and checksum applied state across members periodically.
Advanced
The current-term commit restriction is the subtlest rule in Raft. Without it, an entry replicated to a majority by an old leader could be reported committed and still be overwritten by a later leader that never saw it, breaking State Machine Safety. The no-op entry a new leader appends is the practical device that makes previous-term entries committable, and its absence is a classic bug in hand-rolled implementations.
Internals
Compaction turns the unbounded log into a snapshot plus a suffix. A follower behind the leader’s snapshot point cannot be repaired by the backwards nextIndex walk — the entries no longer exist — so the leader ships an InstallSnapshot instead, which is expensive and must be rate-limited to avoid starving normal replication. Compaction is also why a chronically lagging follower is an availability risk rather than a cosmetic one: the leader must retain log entries until the slowest member catches up, so one stuck follower grows the leader’s disk until it must choose between unbounded growth and snapshot-shipping.
Apply it
- 🔧 Construct the data-loss scenario that occurs if a leader commits a previous-term entry purely on replica count.
- 🔧 Explain why the backwards
nextIndexwalk is guaranteed to terminate at a correct common prefix. - 🔧 List three sources of non-determinism in a state machine and how you would detect each in production.
- ⚡ A leader accepts 500 writes while partitioned and acknowledges none of them. It rejoins 90 seconds later. Describe precisely what happens to those 500 entries and what each client saw.
- 💬 When exactly is a Raft entry committed?
- 💬 Two nodes have different entries at index 7. Which one is wrong, and how does the cluster resolve it?
- 💬 Why does a new Raft leader append a no-op entry?
- 💬 A client times out on a write. What are the possible fates of that entry?