Eventual Consistency in Practice
Once work happens after the response, some reads are stale — and the engineering is in bounding it, showing it honestly, and knowing when it has stopped converging.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
My write returned 200 and the next read does not reflect it. Is that a bug, and what do I owe the user?
A user changes their display name. It updates immediately in their profile, appears in search within a minute, and shows on old comments whenever the cache turns over.
Write, publish an event, return 200. The consumers will catch up in a moment; it is fine.
The client immediately re-fetches the list it just changed, gets the pre-change value from a read replica, and renders the old name — so the UI appears to have discarded the user's edit (Read Replicas From the Application).
- The client immediately re-fetches the list it just changed, gets the pre-change value from a read replica, and renders the old name — so the UI appears to have discarded the user's edit (Read Replicas From the Application).
- The user refreshes, sees the old value again, edits again, and now you have two writes and a support ticket.
- A test writes and then asserts on a read, passes locally where everything is synchronous, and fails in CI 4% of the time.
- A downstream consumer fetches the entity by id on receiving the event, hits a replica that has not caught up, gets "not found", and dead-letters a perfectly valid event.
- The consumer has been dead for six hours, so "eventually" is now "never", and nothing distinguishes that from ordinary lag (Queue Backlog).
What is actually happening
- Eventual consistency is a claim about convergence, not about correctness: given no new writes and a working pipeline, all replicas and derived stores will agree. Both conditions are load-bearing and the second one fails often.
- The staleness a user experiences is the sum of concrete, measurable delays: replication lag, queue lag, consumer processing time, index refresh interval, cache TTL. None of them are mysterious; each is a number you can observe (TTL and Expiry).
- Read-your-writes is the specific guarantee users actually notice. They tolerate other people's changes arriving late; they do not tolerate their own change vanishing.
- The failure that matters is not lag, it is stall. A pipeline that is thirty seconds behind is working. A pipeline that is thirty minutes behind and growing is broken, and the two look identical if you only watch a snapshot.
- Eventual consistency is not something you opt into by using events — it is what you already have the moment there is a replica, a cache, a CDN or a consumer. The choice is whether it is designed or discovered.
The staleness budget is a sum you can actually compute
Teams talk about eventual consistency as if it were a philosophical property. It is arithmetic. Each hop between the commit and the read contributes a delay you can measure, and the user-visible staleness for a given screen is the sum along that screen's path.
Writing the sum down changes the conversation from "search feels stale" to "the indexer contributes forty seconds of the sixty-second budget, and the rest is the refresh interval". One of those is actionable.
| Hop | What it contributes | How you see it | What shrinks it |
|---|---|---|---|
| Primary commit | Zero — this is the moment of truth | The write's own latency | n/a |
| Replica apply | Replication lag | Engine lag metric per replica | Fewer, closer replicas; route the writer to primary |
| Outbox relay | Poll interval plus publish time | Age of oldest unpublished outbox row | Shorter poll, or change-data-capture |
| Broker to consumer | Consumer lag | Oldest unprocessed message age | More consumers, up to the partition count |
| Consumer work | Processing plus batch window | Per-message duration histogram | Smaller batches — at the cost of throughput |
| Index refresh | Engine's near-real-time interval | Engine setting; probe write-to-visible | Shorter refresh, at indexing cost |
| Cache TTL | Up to the full TTL | TTL configuration; hit rate | Invalidate on the event instead of expiring (Cache Invalidation) |
| Client store | Until the client refetches | Client instrumentation only | Return the resource from the write |
Read-your-writes is the only one users notice
Users are remarkably tolerant of other people's changes arriving late and completely intolerant of their own disappearing. That asymmetry tells you where to spend effort: not on making the whole system consistent, but on making each user's own writes visible to them immediately.
The options below are ordered roughly by how much they cost. The first one is free and is skipped surprisingly often, because the write endpoint was designed to return 204 No Content and nobody revisited it.
How does this user's own change become visible to them without waiting for convergence?
when Almost always. The client already has the round trip; use it.
cost The write response now carries the read model, and the two must stay in step (Three Models, Not One).
when The client will re-fetch a list or a related resource the write response cannot contain.
cost Primary read load proportional to write rate; needs a per-session marker that survives across instances (Stateless Services).
when The UI can render the new state confidently and reconcile later.
cost The client now holds business logic, and a rejected write means a visible rollback.
when The work genuinely takes time — reindexing, media processing, a long workflow.
cost A polling or notification contract, and a real state in the UI (Long-Running Operations: 202 and the Job Resource in API Design).
when Other people's data, analytics, aggregates, dashboards.
cost You owe the user a visible "as of" timestamp rather than a silently old number.
Lag is fine. Stall is an outage.
The single most useful operational idea in this lesson: a healthy asynchronous system is always behind. Being behind is not the alarm. Being behind and not catching up is the alarm, and distinguishing them requires watching the derivative, not the value.
The check below is deliberately about progress rather than about depth. A queue at depth zero with a dead consumer and a queue at depth zero with a healthy one are identical on a depth graph and completely different in reality.
1type LagSample = { oldestUnprocessedAgeMs: number; at: number }2 3/**4 * Two independent conditions, because they mean different things:5 * - breachedBudget: users are seeing staleness beyond what we promised6 * - stalled: the pipeline is not making progress at all7 * A pipeline can breach the budget while recovering (fine, watch it)8 * or stall while inside the budget (not fine, page someone).9 */10function assess(samples: LagSample[], budgetMs: number) {11 const latest = samples[samples.length - 1]12 const earliest = samples[0]13 const window = latest.at - earliest.at14 15 const breachedBudget = latest.oldestUnprocessedAgeMs > budgetMs16 17 // If nothing is being processed, the age of the oldest message18 // grows exactly as fast as wall-clock time.19 const ageGrowth = latest.oldestUnprocessedAgeMs - earliest.oldestUnprocessedAgeMs20 const stalled = window > 0 && ageGrowth >= window * 0.9521 22 return { breachedBudget, stalled }23}The stall test is the important half: when a consumer dies, the age of the oldest unprocessed message advances one millisecond per millisecond. That ratio, not any absolute threshold, is what separates "slow" from "stopped" — and it needs no tuning per topic.
How to build it
Most important first.
- Return the authoritative result from the write itself. The most effective fix for read-your-writes is to not make the client re-read at all — respond with the updated resource (Status Codes From the Server's Side).
- Route a writer's own subsequent reads to the primary for a bounded window, keyed on the session, so the person who made the change always sees it (Read Replicas From the Application).
- Make the in-between state explicit in the contract rather than pretending it does not exist: a
processingstatus, a job id to poll, or a version token the client can wait for (The Async Job Pattern in API Design covers the contract shape). - Choose consistency per read, not per system. The account balance on the payment screen and the same balance in a monthly summary have different requirements and should be served differently.
- State the convergence bound as an objective and monitor against it: "search reflects an edit within 60 seconds at p99" is a commitment you can alert on. "Eventually" is not.
- Design consumers to be safe when the fact is not yet readable: pass what they need in the event, or treat not-found as retryable rather than as a permanent failure (Naming Events).
What can go wrong
- Read-your-writes solved with a client-side sleep. It works on a fast day and produces flaky behaviour on a slow one.
- Sticky primary reads applied globally "to be safe", which removes the point of having replicas and moves the load back onto the primary (Sticky Sessions).
- A convergence bound published to users but never monitored, so it becomes a promise no one is checking.
- A consumer that dead-letters on not-found, permanently dropping events that were only a few hundred milliseconds early.
- Two derived stores converging at very different rates, so the same value differs between two screens and support cannot reproduce either.
- Compensating logic that assumes convergence has happened by the time it runs — a reconciliation that "fixes" data that was merely in flight.
- A read arriving between the primary commit and the replica applying it — the canonical read-your-writes race, resolved by routing rather than by timing (Backend Races).
- Two writes converging in different orders in two derived stores, so search and the API disagree until both settle (Keeping a Search Index in Sync).
- A consumer fetching an entity by id before the producing transaction is visible on its replica, producing a spurious not-found for a fact that is true.
- Permission and access revocation must not be eventually consistent in the direction of permissiveness. A revoked session or role that takes a minute to propagate is a minute of unauthorized access (Where the Check Belongs).
- Cached or indexed authorization decisions are the common form of this: the check passed once and the answer outlived the grant (Cache Invalidation).
- Deletion requests are a legal boundary in many jurisdictions, and "eventually deleted from the index, the cache, the dead-letter queue and the analytics store" needs to be an actual, verified list (Keeping a Search Index in Sync).
- "Eventual consistency means the data is sometimes wrong." It means reads may be behind. The source of truth is not wrong; the projections are late, and they converge — while the pipeline runs.
- "It is eventually consistent, so it will sort itself out." Only if something is still processing. A stalled consumer converges to nothing and looks the same from the outside (Writing Event Consumers).
- "Adding a cache does not change consistency." It adds a convergence window with its own TTL, and it is usually the largest one in the system (Caching in Backends).
- "Users cannot tell." They can, precisely and immediately, for their own writes. They largely cannot for other people's, which is why the design effort belongs on read-your-writes.
- "We are strongly consistent because we use one Postgres." Any read replica, cache, search index, CDN or client store reintroduces staleness regardless of the database's guarantees.
Operating it
- Replication lag per replica, consumer lag per group, index refresh interval and cache TTL, all on one dashboard. Together they are the user-visible staleness budget (Replication Lag: Reads That Are Correct and Stale in Performance covers the measurement).
- A synthetic write-then-read probe per derived store, recording time-to-visible. It is the only measurement that matches what a user experiences.
- Alert on the rate of change of lag, not only on its value. Lag that is high and shrinking is recovery; lag that is low and growing is the start of an incident.
- Count reads served from the primary due to a read-your-writes rule. If that number climbs, your replicas are doing less than you think.
- More replicas and more derived stores means more independent convergence windows, and the user-visible staleness becomes the maximum across all of them rather than the average.
- Under write bursts, lag grows on every derived path at once, which is exactly when users are most likely to be looking — so the worst staleness coincides with peak attention.
- Read-your-writes via primary routing does not scale with write-heavy workloads: at high write rates a large share of reads qualify for primary routing and the replicas idle.
- Cross-region deployments add a physical lower bound on convergence that no amount of tuning removes (Cross-Region Latency Is Physics, Not Configuration in Performance).
- Strong consistency for a read costs the primary's capacity and, across regions, real latency. It is the right choice for balances and permissions and the wrong default for everything.
- Returning the resource from the write makes the write endpoint heavier and couples it to the read model — usually worth it, and not free (Three Models, Not One).
- Exposing a
processingstate is honest and adds a state to every client. Hiding it is simpler until the day it is wrong. - Monitoring convergence per derived store is a real amount of instrumentation for something that is invisible when healthy.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALApplies to any system with a replica, cache, index, CDN or asynchronous consumer, independent of stack.
- DATABASE-SPECIFICWhat "read your writes" costs depends on the engine and topology: with synchronous replication a replica read is already current at commit; with asynchronous replication it is not, and the application must route. Some managed offerings expose a session or LSN token that lets a replica wait for a specific write, which changes the design from routing to waiting.
- SCALE-SPECIFICA single primary with no replicas, no cache and synchronous side effects has none of these problems, and the machinery here would be pure cost. The problem arrives with the first replica or the first consumer.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — consistency models, session guarantees and what a system can promise across a partition.
- — System Design — choosing per-read consistency across a whole product rather than per endpoint.