Replication Lag: Reads That Are Correct and Stale
Replicas turn read capacity into a purchase, and the price is time. Lag is not a failure until the application assumes it is zero — and every read-after-write bug in a replicated system is that assumption meeting reality.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Lag is a capacity signal before it is a correctness problem
A replica applies the primary's write stream. If it can apply changes at least as fast as they are produced, lag stays near the network round trip. If it cannot — because apply is more serialized than the parallel writes that produced it, because the replica is also serving heavy reads, or because a long-running query blocks apply — lag grows, and it grows in the same unbounded way any queue does when arrival exceeds service (Queueing: Why Systems Get Slow Before They Get Broken).
That makes lag a leading indicator worth alerting on well before it becomes a user-visible correctness issue. Rising lag during peak write traffic says the replica is at its apply capacity, which is a capacity conversation. Lag that recovers immediately when writes subside is a burst-absorption story. Lag that never recovers is a replica that is falling permanently behind and will eventually need rebuilding.
Measure it two ways, because they answer different questions. Time lag ("this replica is 2.4 s behind") is what the application cares about for staleness decisions. Byte lag ("this replica is 340 MB behind in the write stream") is what capacity planning cares about, and it keeps meaning something when write traffic stops — time lag on an idle system can read as zero while the replica is still far behind in bytes.
The read-after-write bug, and why it is intermittent
The failure is always the same: a write goes to the primary, the immediately following read is routed to a replica, and the replica has not applied that write yet. The user updates their profile and the next page shows the old name. It works perfectly in development, where there is one database. It works most of the time in production, because lag is usually small and the read usually arrives later than the apply. It fails during write bursts — precisely when the system is busiest and the incident is most expensive.
The intermittency is what makes this expensive to diagnose. The bug is a race between apply latency and the user's next request, so its reproduction rate depends on load, and the load that produces it is the load nobody can reproduce locally. Teams spend days on "sometimes the update does not save" tickets that are not save failures at all.
The fixes trade differently and should be chosen per read, not globally. Route reads that follow a write from the same user to the primary for a bounded window; or have the write return a position token and require the read to wait for a replica that has reached it; or make the interface not need the read at all by rendering from the write's own response. The last one is free at runtime and is the one people forget — the data was in the write response the whole time. This is the same contract question API Design frames as Consistency as a Contract Clause: what does the API promise about reading your own writes?
| Strategy | How it works | Good for | What it costs |
|---|---|---|---|
| Render from the write response | The mutation returns the new state; the UI uses it directly | The common case — the screen right after a save | Nothing at runtime; requires the write endpoint to return the representation |
| Sticky-to-primary window | After a write, route that user's reads to the primary for N seconds | Simple, broadly effective | Primary read load rises with write rate; N must exceed p99 lag |
| Wait for position token | Write returns a log position; read waits until a replica has applied it | Strong guarantee where it genuinely matters | Added read latency; requires engine and driver support |
| Read from primary always for this path | Certain endpoints never use replicas | Small set of correctness-critical reads | Gives up the read scaling for that path |
| Accept staleness explicitly | Document the read as eventually consistent; show a freshness hint in the UI | Feeds, counts, analytics, dashboards | Product decision, not a technical one — needs a real owner |
| Not a fix | Lower the lag alert threshold | Nothing | Lag is a race, not a threshold; smaller lag makes the bug rarer, not absent |
"Just add a read replica" moves the problem
Adding replicas is the standard answer to read load, and it is genuinely effective for read-heavy workloads. What it does not do is add write capacity: every replica applies the *entire* write stream, so writes remain a single-node problem and each new replica adds its own copy of the apply work plus the network cost of shipping the stream to it. A workload that is write-bound gets no relief and a slightly larger bill.
It also converts a simple system into a distributed one with a staleness contract, and that contract is now load-bearing across the whole application. Each read path acquires a decision it did not have before: primary or replica, and if replica, what staleness is acceptable. Teams that add replicas without making those decisions explicitly have not removed a problem; they have distributed it into every feature that reads data, where it surfaces as intermittent bugs owned by nobody.
The panel below is the read worth taking before and after adding a replica. Note the last row: primary write throughput is unchanged, because it was always going to be. If the constraint was write capacity, the correct conversation is partitioning, batching, or a different data model (Partitioning and Sharding), and replicas were never the answer.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Read capacity | 3x | Real and immediate — reads distribute across primary plus two replicas. | normal |
| Primary read load | -62% | The intended effect. | normal |
| Primary write throughput | unchanged | Every replica applies the full write stream; replication adds no write capacity. | smoking gun |
| Replica apply lag p99 | 2.4 s during write bursts | Apply is more serialized than the writes that produced it. | suspect |
| Read-after-write defects | 0 → 7 open tickets | Intermittent "my change did not save" reports appear within a week. | smoking gun |
| Failover RPO exposure | 0 → up to p99 lag | An unplanned failover can lose whatever the promoted replica had not applied. | suspect |
Key points
- Lag is a queue: it grows when apply rate falls below write rate, and it is a capacity signal before it becomes a correctness problem.
- Measure lag in both time and bytes, at p99 — average lag hides the write-burst spikes when read-after-write bugs actually happen.
- Read-after-write failures are a race, so they are intermittent, load-dependent, and never reproduce in a single-database development environment.
- Choose a staleness strategy per read path; rendering from the write response is free and routinely overlooked.
- Replicas add read capacity and zero write capacity, and they convert a simple system into one with a staleness contract in every feature.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1User → support: "I updated my address and it reverted" — intermittent, no error, not reproducible on request.
- 2Application → routing: the write went to the primary; the redirect's read was routed to a replica by the read/write splitter.
- 3Replica → state: p99 apply lag during the evening write burst is 2.4 s, well beyond the ~200 ms between the write and the follow-up read.
- 4Lag → cause: apply on the replica is more serialized than the concurrent writes that produced the stream, so it falls behind under burst.
- 5Cause → root cause: the application assumes read-after-write consistency that the replicated topology never promised, and no read path declares its staleness tolerance.
- • "The write failed." The write succeeded on the primary; the read went somewhere that had not applied it yet.
- • "Average lag is 90 ms, so we are fine." Read-after-write bugs happen at the p99 spike, not the average.
- • "Lower the lag and the bug goes away." It becomes rarer and remains a race. Rare intermittent bugs are harder to diagnose, not better.
- • "Add replicas to handle the load." Replicas add read capacity only; a write-bound system gets a bigger bill and a staleness contract.
- • "Lag is zero, the replica is caught up." On an idle system time lag reads near zero regardless of byte position — check both.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Replication lag per replica in seconds and in bytes, at p50 and p99, retained long enough to correlate with write-traffic bursts.
- • Apply rate against write rate on the primary, which shows whether a replica is keeping up or falling behind structurally.
- • The fraction of reads served by replicas versus primary, per endpoint, so sticky-to-primary windows have a visible cost.
- • Read-after-write defect reports as a tracked signal — they are the user-visible measurement of a lag policy that is wrong.
- • Longest-running query on each replica, since a long read can block apply and turn a read-scaling win into a lag incident.
- • Return the new state from the write and render from it, removing the follow-up read entirely where the UI allows.
- • Route post-write reads for that user to the primary for a window comfortably larger than p99 lag, per path rather than globally.
- • Use position tokens (log sequence number / GTID) where a genuine guarantee is required and the engine and driver support it.
- • Declare staleness explicitly for paths that tolerate it — feeds, counters, dashboards — and surface freshness in the interface rather than pretending it is current.
- • Treat sustained lag as capacity: reduce replica read load, isolate long-running analytics queries, or address write volume at the model level ([[partitioning-and-sharding]]).
- • Reproduce under load: a test that writes then immediately reads at production-like write rates should show zero stale reads after the fix.
- • Confirm primary read load rose by the expected amount after adding a sticky window — if it did not, the routing is not taking effect.
- • Track read-after-write defect reports over the following weeks; this class of bug is measured in tickets, not in dashboards.
- • Check p99 lag against the chosen sticky window regularly, since a window that was safe at last quarter's write volume may no longer be.
- • Sticky-to-primary windows restore correctness and return read load to the primary, partially undoing the scaling the replicas were added for.
- • Waiting for a position token adds latency to reads and couples the application to engine-specific replication internals.
- • Every additional replica adds full apply work and stream bandwidth, so read scaling has a real per-replica cost on the primary side.
- • Explicit staleness is honest and requires product decisions about what users are told, which is slower than a technical fix.
- • An alert on replica lag p99 crossing the sticky-to-primary window, which is the moment the correctness assumption silently expires.
- • An integration test that runs against a replicated topology with artificial lag injected, asserting each read path's declared staleness tolerance.
- • A documented staleness contract per read path, reviewed when routing changes, so "primary or replica" is never an implicit default.
- • A monitor on longest replica query, catching the analytics query that blocks apply before it becomes a lag incident.
Accuracy
Performance numbers are conditional. These are the conditions.
- DATABASE-SPECIFICLag semantics, apply parallelism and position-token mechanisms differ sharply — PostgreSQL streaming replication with LSNs, MySQL GTIDs with multi-threaded apply, and managed cloud replicas all behave differently under burst.
- ILLUSTRATIVEThe 2.4 s lag, the 62% primary read reduction and the ticket counts are a constructed shape showing the trade, not measurements.
- WORKLOAD-SPECIFICWhether replicas help at all depends on the read/write ratio. A write-bound workload gains nothing from them.