Replication

Monotonic Reads: Never Let Time Run Backwards

A reader who has seen a value must never subsequently see an older one. Without this, a page refresh can un-post a comment and a polling client can watch a counter oscillate — and unlike stale data, moving backwards is something users interpret as the system being broken.

▶ Run the lab

The question this answers

The question

Why does hitting refresh sometimes show older data than the previous load, and why is that so much worse than merely being stale?

The guarantee — the property claimed, and its scope

Monotonic reads: if a session reads a value at state S, every later read in that session returns S or a state at least as recent. It does not promise freshness — a session may sit arbitrarily far behind — only that it never moves backwards.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

A replica knows its own applied position and can compare it against a floor the client presents. It does not know what the client saw previously unless the client tells it. The entire mechanism rests on the client carrying forward the highest position it has observed, because no replica can reconstruct another replica's history.

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?What does a node know?How does it work?What can fail?How does it fail?Where is coordination?What holds under failure?How does it recover?How would you know?What is the simpler thing?
session guaranteesstalenessreplicationmonotonicity

Stale is confusing; going backwards is broken

These are different failures and users treat them differently. A stale read shows something a little old — annoying, forgivable, often unnoticed. A non-monotonic read shows something the user has already seen superseded: the comment they watched appear now absent, the balance that went up and then down, the order status that regressed from "shipped" to "processing".

The reason it lands so much harder is that it violates a model humans hold unconditionally — that time moves one way. A user seeing stale data concludes the system is slow. A user seeing data move backwards concludes the system is losing their data, and starts taking screenshots. The engineering cost of preventing it is small; the trust cost of allowing it is not.

Two reads, two replicas, and a page that appears to un-savetypical
UserReplica 1 (current)Replica 2 (behind)GET /order: deliveredGET /orderGET /order (load balanced elsewhere): deliveredGET /order (load balanced elsewhere)has v7 (write) at t=1has v7read -> v7 ("shipped") (read) at t=3read -> v7 ("shipped")refresh -> v4 ("processing") (read) at t=7refresh -> v4 ("processing")finally applies v7 (write) at t=11finally applies v7t=1time →t=11
delivereddelayed (dashed, long)duplicated (×2)dropped — stops short, never arriveswriteread
Neither replica is faulty. The load balancer chose differently on the second request, and the session experienced time reversing. The fix is not to make replicas equal — it is to make the session carry what it has already seen.

The fix is the same shape as read-after-write

Read-your-writes carries a floor derived from your *writes*. Monotonic reads carries a floor derived from your *reads*. Same plumbing, one extra update: after every read, raise the session floor to the position the replica served from.

That symmetry is worth noticing because it tells you the two guarantees compose for free — a single session floor updated on both writes and reads gives you both properties at once, and that combination covers the overwhelming majority of user-visible consistency complaints in replicated systems. See Session Guarantees: The Underrated Middle Ground for the full set of four.

1type SessionFloor = { position: number }
2
3async function read(session: SessionFloor, key: string) {
4 const replica = pickReplicaAtLeast(session.position) // may return the leader
5 const { value, servedAtPosition } = await replica.read(key)
6
7 // The crucial line: what I have seen can only go up.
8 session.position = Math.max(session.position, servedAtPosition)
9 return value
10}
11
12async function write(session: SessionFloor, key: string, v: unknown) {
13 const { position } = await leader.write(key, v)
14 session.position = Math.max(session.position, position) // read-your-writes
15}
One floor, raised by both writes and reads, gives both guarantees

Where non-monotonic reads sneak in even when you think you fixed it

Sticky routing is the usual mitigation and it is a partial one. It holds while the pin holds, and the pin does not hold across a deploy, a rebalance, a replica restart, a connection-pool refresh, or a client that reconnects on a different network. Each of those is a moment when a session silently moves to a replica behind the one it was on.

The subtler sources are worth listing because they are invisible in a single-service view. A CDN or HTTP cache can serve a response older than one the browser already rendered. A fan-out page that assembles data from several services can be internally non-monotonic even if each service is monotonic alone. A retry that lands on a different replica than the original attempt. A background poller in a different process than the one that rendered the page. Each needs the floor to travel with the request, which is why this is a propagation problem more than an algorithm problem.

  • Sticky sessions break at deploys, rebalances, restarts and network changes — precisely the moments nobody is looking.
  • HTTP caches and CDNs can serve a response older than one already delivered; ETag/If-None-Match alone does not prevent regression. See apiLinks conditional-requests.
  • Composite pages can regress in aggregate even when each backing service is individually monotonic.
  • Any hop that drops the freshness floor silently downgrades the guarantee to eventual consistency.

Key points

  • Monotonic reads forbids going backwards; it says nothing about freshness.
  • Users forgive stale data and do not forgive time reversing — the trust cost is disproportionate to the technical severity.
  • The mechanism is a session floor raised on every read, which is the same plumbing as read-your-writes.
  • One floor updated by both reads and writes gives both guarantees, covering most real staleness complaints.
  • Sticky routing is a partial fix that fails at deploys, rebalances and reconnects — the floor must travel with the request.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works
  • Each read response carries the position the serving replica had applied.
  • The session records the maximum position it has ever observed.
  • Subsequent reads present that maximum as a requirement.
  • A replica behind the requirement waits, redirects, or declines — it must not serve the older value.
  • The floor is monotonic by construction, so no ordering logic beyond a maximum is required.
What can fail at the boundary
  • The floor is not propagated through a hop and the requirement is lost.
  • A replica is chosen without checking the floor because the routing layer does not know about it.
  • A cache layer returns a response older than the floor.
  • Positions become incomparable after a failover or a re-seed.
  • The session identity changes — new tab, new device, cleared storage — and the floor resets to zero.
How it fails — what an operator sees
  • Refresh regression: the user reloads and sees an older state than the previous render. Reported as "it lost my comment", investigated as a data-loss bug, and found to be routing.
  • Oscillating widget: a polling component alternates between two values as successive requests land on replicas at different positions — a status flipping between "shipped" and "processing" every few seconds.
  • Post-deploy spike: sticky routing is dropped when instances are replaced, and non-monotonic complaints cluster in the minutes after every deploy with no error-rate change.
  • Composite page inconsistency: a dashboard shows a total that disagrees with the itemised list beneath it, because the two were assembled from replicas at different positions.
  • Cache-induced regression: a CDN serves a stale cached response after a fresher one was already delivered to the same client, so the regression happens outside the application entirely.
Where coordination is required
  • None between replicas — a replica only compares a client-supplied number against its own state.
  • The cost is propagation discipline: the floor must survive every hop, retry, cache and process boundary, which is a cross-cutting concern like tracing or deadlines.
  • This is dramatically cheaper than any global ordering guarantee, and it is the right first purchase for user-visible consistency problems.
What still holds under failure
  • When replicas lag, monotonic sessions get slower or move to the leader; they never move backwards.
  • When the floor is lost, the session degrades to eventual consistency — which is the pre-existing behaviour, not a new failure.
  • The guarantee is per-session, so a partition affects only sessions whose floor exceeds the reachable replicas' positions.
How it recovers
  • Detect: instrument a monotonicity check in the client — compare each response's served position against the last one and count regressions. This is a direct measurement, not a proxy.
  • Contain: route sessions whose floor exceeds all healthy replicas to the leader rather than serving them an older value.
  • Recover: regressions cease as soon as replicas advance past the floors in play; no manual step.
  • Reconcile: invalidate floors after a failover that changes log lineage, so sessions restart from a comparable baseline.
  • Verify: a synthetic client that reads repeatedly through the real edge — including the CDN — and asserts non-regression, since the CDN is where this most often reappears.
How you would know
  • Client-side regression counter: how often a response was older than the previously observed one. This should be zero and is usually never measured.
  • Distribution of served positions per replica, to see how wide the fleet's spread is.
  • Fraction of requests arriving without a floor — the leak indicator.
  • Regression rate broken down by deploy, since sticky-routing loss clusters there.
  • Cache hit responses whose position is below the requesting session's floor.
When it helps
  • Any UI that renders the same data repeatedly — dashboards, polling views, infinite scroll, status pages.
  • Systems that have spread reads across many replicas with variable lag.
  • Mobile clients that reconnect frequently and land on different backends each time.
When it hurts
  • Single-replica or leader-only read paths, where the property already holds and the plumbing is pure cost.
  • Batch and analytical reads where a session concept does not exist and nobody is comparing successive results.
  • Cases where the real requirement is freshness rather than monotonicity — monotonic reads will not make the data current and should not be sold as if it will.
Simpler alternatives
  • Leader-only reads: trivially monotonic, and correct until leader read load matters.
  • Sticky routing to a single replica — cheap, partial, and honest about being partial.
  • Bounded staleness: refuse to serve from a replica more than X behind, which limits how far back a regression can go without eliminating it.
  • Client-side merge: keep the highest version seen in the client and never render an older one, which fixes the display without touching the read path.
  • Read-your-writes alone, if the complaints are all about the user's own edits rather than about regression. See Read-After-Write: Letting a User See Their Own Change.

Refresh, and the comment is gone again

Refresh, and the comment is gone again
Two reads in one session, served by two replicas at different positions. The second read returns an earlier state than the first — time ran backwards for that user.
readstepserved bypositionwhat the user saw
#12near1comment posted
#23far0old ← went backwards
#34near1comment posted
#45far0old ← went backwards
Monotonic reads violated
A read in this session returned an earlier state than a previous read in the same session. Both replicas are healthy and both answers are individually legal under eventual consistency — the system never promised the session would move forward. Users forgive stale data; they do not forgive time reversing, because a comment that appears and then disappears reads as data loss, not as lag.
Monotonic reads forbids exactly one observation: going backwards. It says nothing about freshness. The mechanism is the same plumbing as read-your-writes — one floor per session, raised on every read as well as every write — which is why one token buys both guarantees and covers most real staleness complaints. Sticky routing is the partial fix people reach for first, and it fails at deploys, rebalances and reconnects, because the guarantee has to travel with the request rather than living in a routing table.
simplifiedTwo replicas, fixed delays, one write. Real deployments have more replicas and noisier lag, which makes the regression intermittent rather than reproducible — worse to diagnose, identical in mechanism.

What people believe, and what is true

Claim

Monotonic reads means the reader sees fresh data.

Reality

It means the reader never sees older data than before. A session can hold a monotonic view that is an hour behind, forever, and the guarantee is fully satisfied.

Claim

If each service is monotonic, the page is monotonic.

Reality

A page assembled from several sources can regress in aggregate — the total and the list can come from replicas at different positions. Composition needs a shared floor.

Claim

Sticky sessions give us this.

Reality

They give it while the stickiness holds. Deploys, rebalances, restarts and reconnects break it, which is why regressions cluster right after every deploy.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

Once a session has seen a value, it must never see an older one. Stale is tolerable; going backwards reads as data loss.

Practical

Return the serving position with every read, keep the maximum in the session, send it with subsequent reads, and let replicas below it defer to the leader. Then measure regressions client-side — it is the only place the property is directly observable.

Advanced

Monotonic reads is the read-side dual of read-your-writes, and both are single-scalar projections of the causal order. Once you need the same property *across* sessions — you must not see an effect before its cause, regardless of who wrote it — a scalar no longer suffices and you need per-source versions. That is exactly the step from a session floor to a vector, and from here to Causal Consistency: Never Show an Effect Before Its Cause. See Vector Clocks: Buying Concurrency Detection at O(N).

Apply it

Interview questions
  • 💬 A user refreshes and sees an older order status than before. Nothing errored. What guarantee is missing and how do you provide it?
  • 💬 Why do non-monotonic read complaints cluster immediately after deploys?
  • 💬 Your dashboard shows a total that disagrees with the rows below it. Each service is individually monotonic. Explain.