The question this answers
If my replica is always a little behind, exactly how much data am I agreeing to lose, and when do I find out?
A write acknowledged by the leader is durable *on the leader*. It will reach the replicas eventually, in order, provided the stream stays connected and the leader survives long enough to send it. There is no bound on staleness at a replica, and no guarantee that any specific acknowledged write survives the leader's death.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The leader knows the position it has sent to each follower and the position each follower has confirmed; the difference between the two is the size of the loss window, and it is knowable in advance. A follower knows only its own applied position — it cannot tell whether it is 5ms or 5 minutes behind, because "no new entries" and "the connection is silently dead" look identical from where it stands.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The window is a number, so treat it as one
The usual conversation about asynchronous replication is qualitative — "there is a small chance of data loss". That is not a design input. The useful form is: at the moment of leader failure, the set of writes lost is exactly those acknowledged but not yet confirmed by any surviving replica, and its size is the current lag, which you are already measuring.
Once it is a number, it becomes a business question rather than a technical one. Losing 200ms of session-tracking writes is nothing. Losing 200ms of payment authorisations is an incident with a name and a postmortem. Same protocol, same number, entirely different answer — and the point is that you get to have the conversation before the failure rather than during it.
leader write position 4,912,338 replica-b confirmed position 4,912,331 gap: 7 writes (~15 ms) replica-c confirmed position 4,911,004 gap: 1,334 writes (~2.8 s) if the leader dies right now: promote replica-b -> lose 7 acknowledged writes promote replica-c -> lose 1,334 acknowledged writes neither reachable -> writes survive only if the leader's disk does
Lag is not a smooth quantity
Teams reason about lag as if it were a slowly varying number around a baseline. In practice it is bimodal: near zero almost always, and then very large during exactly the events that also threaten the leader. Bulk imports, schema migrations, a long transaction, a compaction storm, or a network incident all push it up — and a network incident is also the most likely cause of a failover.
The correlation is the danger. Your loss window is smallest when nothing is wrong and largest at the moment you most need to fail over. Averaged lag dashboards hide this completely; the number that matters is the lag distribution during incidents, which you will only have if you record the maximum rather than the mean.
Why this is usually still the right default
Given the above it can sound like asynchronous replication is a mistake people make. It is not. It is the arrangement that lets write latency be a property of one local disk rather than of the network, and for the overwhelming majority of data — content, activity, derived state, anything the client can regenerate — the loss window costs less than the latency would.
The failure is not choosing asynchronous replication. The failure is choosing it *implicitly*, by accepting a default, and then discovering during an incident that the business believed acknowledged meant durable. The honest version of this design has three artefacts: a stated recovery point objective, a measurement showing the actual window meets it, and a reconciliation path for the writes inside it.
- Write latency stays local — no network round trip on the critical path, and no dependency on a follower being healthy.
- A slow or dead replica cannot stall writes, so adding replicas does not reduce write availability.
- Replicas can be arbitrarily distant, which is what makes cross-region read locality affordable.
- The cost is concentrated entirely in failover, which is rare — so the expected cost is low even where the worst case is bad.
Key points
- The loss window is exactly the set of writes acknowledged but not yet on a surviving replica, and its size is the lag you already measure.
- Lag is bimodal and correlates with the incidents that cause failovers, so the window is largest exactly when it is used.
- Asynchronous replication is often correct — it keeps write latency local and keeps a slow replica from stalling writes.
- The mistake is choosing it implicitly. State the recovery point objective, measure the window against it, and know how you would reconcile.
- A follower cannot tell "no new writes" from "the stream is dead", so silence must be treated as suspect rather than as caught up.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The leader durably logs a write and immediately acknowledges the client.
- • A background process streams accumulated log entries to each follower at whatever rate the follower and network allow.
- • Each follower applies entries in order and periodically reports its confirmed position back to the leader.
- • The leader retains log entries until all followers (or a retention limit) have consumed them; past that limit a lagging follower must be re-seeded.
- • On promotion, the new leader's history simply ends at its last applied position — entries beyond it are gone, and any old-leader entries that arrive later must be discarded, not merged.
- • The leader dies holding acknowledged entries no follower received.
- • The replication connection dies silently and lag grows without a single error being logged.
- • A follower falls past the leader's log retention and can no longer catch up incrementally.
- • The old leader returns after promotion and attempts to stream entries that conflict with the new leader's history.
- • A long-running transaction or bulk operation on the leader produces a burst of entries the follower cannot apply at line rate.
- • Acknowledged-but-absent writes after failover: customers hold confirmation numbers for orders that do not exist. The count equals the lag at failure time, and no component reports it — you must compute it.
- • Unbounded lag with green dashboards: apply lag grows for hours because the stream is stalled, while "seconds since last event" reads near zero on a quiet shard.
- • Re-seed cascade: a follower drops out of retention, a full snapshot copy begins, the resulting read load slows the leader, and a second follower falls out of retention during the copy.
- • Divergent old leader: the demoted node comes back and, without fencing, replays entries into a history that has moved on. The operator observes rows that reappear after deletion.
- • Read-your-write failures at the application layer, appearing as "the save did not work" support tickets with no corresponding server errors.
- • Zero on the write path — this is the entire point, and it is why write latency is a local property here.
- • Coordination is deferred to failover, where the cost appears as lost data rather than as latency. You did not avoid the cost, you changed its currency.
- • Choosing which follower to promote is a comparison across nodes, and doing it correctly requires reaching them — which is precisely what may not be possible during the incident.
- • Every surviving replica holds a valid *prefix* of the leader's history — internally consistent, just incomplete. It is never a mixture.
- • Reads continue to be servable from replicas at unbounded staleness.
- • Writes acknowledged inside the loss window are gone with no record in the surviving system; only external records (client logs, upstream systems, the message that triggered the write) can reveal them.
- • Detect: monitor position gap per follower and the maximum gap over a window, not the mean. Alert on stream disconnection separately from lag.
- • Contain: stop routing reads to a follower past its staleness budget, and pause any bulk operation that is inflating lag while a leader is under stress.
- • Recover: promote the follower with the highest confirmed position that you can reach, and record the gap you accepted.
- • Reconcile: replay from the upstream source of truth — the queue, the request log, the payment provider — into the new leader. This is why an append-only ingress log is so valuable here. See The Log Is Not a Queue and Reconciliation Is a Component, Not a Cleanup Script.
- • Verify: compare counts and checksums against upstream systems after the incident, rather than declaring recovery when the service returns.
- • Position gap per follower, plus its maximum over the last day — the maximum is your real recovery point objective.
- • Stream connection state and time since last successful transfer, distinct from time since last applied entry.
- • Log retention headroom expressed in the same units as the gap, so "how close to a re-seed are we" is answerable at a glance.
- • A post-failover write-loss count, computed by comparing the old leader's final position against the promoted node's.
- • Application-level reconciliation deltas against upstream systems — the only signal that surfaces lost acknowledged writes.
- • Regenerable or low-value-per-record data: analytics events, activity feeds, caches, search indexes, derived state.
- • Write paths where latency is the product — anything interactive where an extra network round trip is felt.
- • Cross-region replicas for read locality, where synchronous replication would put continental latency on every write.
- • Clusters with many replicas, where synchronous acknowledgement would multiply the ways a write can stall.
- • Financial, legal or identity records where an acknowledged write that vanishes creates an obligation you cannot honour.
- • Systems that hand the client a receipt or identifier the client will use later — the identifier outlives the data.
- • Any workflow where a downstream system acts on the acknowledgement, since the downstream effect survives while its cause does not.
- • Environments where the lag is unmeasured, which converts a known bounded risk into an unknown unbounded one.
- • Semi-synchronous acknowledgement, which removes the loss window for a single round trip to the nearest follower. See Synchronous Replication: Paying Latency for a Durability Guarantee.
- • Asynchronous replicas plus a synchronous local one — near-zero loss for zone failure, no cross-region write cost. See Active-Passive: Simple to Reason About, Rarely Tested.
- • Write to a durable append-only log first and treat the database as a consumer, so the loss window is reconstructible by replay. See The Log Is Not a Queue.
- • Quorum writes, which give a derivable promotion-safety property rather than a procedural one. See Quorums: What R + W > N Does and Does Not Buy.
- • Accept the window explicitly and invest in reconciliation instead of prevention — often the best value where the upstream record already exists. See Reconciliation Is a Component, Not a Cleanup Script.
The loss window is a number — so treat it as one
| step | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| leader | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 |
| replica | 0 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 |
What people believe, and what is true
Replication lag is a performance problem.
It is a correctness budget. Lag sets your recovery point objective and the staleness your readers can observe; a latency dashboard cannot express either.
Lag is usually a few milliseconds, so the risk is negligible.
Lag is bimodal and spikes during the same events that cause failovers. The number that matters is the lag during incidents, not the median in steady state.
After failover the data will catch up once the old leader returns.
Entries beyond the new leader's history must be discarded, not merged — replaying them would corrupt a history that has already moved on. Recovering them is a manual reconciliation, not an automatic one.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
The leader says "done" before the copies have it. The writes in between are what you lose if the leader dies — that gap is the whole design.
Practical
Turn the window into a number: measure position gap per follower and its daily maximum, then check that number against what the business believes about acknowledged writes. Alert on stream disconnection separately from lag, because a stalled stream can report zero lag.
Advanced
Asynchronous replication is optimal when the expected cost of the loss window is below the aggregate latency cost of synchronous acknowledgement — and because failovers are rare, that inequality holds far more often than intuition suggests. The mistake is almost never asynchronous replication itself; it is failing to build the reconciliation path that makes the window recoverable. See Reconciliation Is a Component, Not a Cleanup Script and Source of Truth: The Question Every Inconsistency Incident Is Really Asking.
Apply it
- ⚡ A checkout service acknowledges an order, the primary is lost 80ms later, and the customer has a confirmation email. Design the reconciliation path that makes this recoverable.
- 💬 How much data would you lose if the primary died right now? Walk me through how you would answer that with the metrics you already have.
- 💬 Your replication lag dashboard shows a flat 0ms and the replica is four hours behind. How is that possible?
- 💬 The old primary comes back after a failover with 400 writes the new primary never saw. What do you do with them, and why is "replay them" wrong?