The question this answers
Why can I not just read the other service’s state the way I read a variable?
A node can guarantee the contents of its own memory and nothing else. A value obtained from another node is guaranteed only to have been true at that node at some point at or before the moment the message was sent.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows its own state exactly, and knows remote state only as a dated snapshot: "X was 7 as of the message I received 40ms ago". It does not know whether X is still 7, whether its own knowledge is the newest, or whether anyone else has a different answer. Treating a received value as current is the single most common source of distributed bugs.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The variable that does not exist
In one process, account.balance names a location. Two threads reading it may race, but they are racing over *the same bytes*, and the memory model tells you exactly what synchronisation makes one thread’s write visible to the other. Concurrency owns that machinery, and it is genuinely powerful: a mutex plus a happens-before edge gives you an authoritative current value.
Across machines there is no location. There is a value in service A’s memory, a value in the database, a value in the cache, and a value in the response B received two seconds ago — four copies, each correct as of a different moment. There is no operation that reads "the balance"; there is only "ask someone, and receive an answer about the past". A mutex does not help because there is nothing for it to protect: a lock on A’s copy says nothing about B’s. That is the point of Distributed Locks: What They Are Actually For and of Concurrency’s own warning that a local lock is not a distributed one.
The design consequence is that every piece of state needs a named owner and every other holder of that state is holding a cache. Once you say that out loud, questions that were vague become concrete: how stale may this copy be, who invalidates it, and what does a reader do when it discovers it was wrong?
Message passing is the only primitive
Everything nodes can do to each other reduces to sending messages. There is no shared write, no atomic compare-and-swap across the boundary, no volatile read. If you want the *effect* of a shared variable, you build it: elect an owner, route all operations through it, and accept that the owner is now a coordination point with its own availability, throughput ceiling and failure mode.
This is why the actor model and channel-based designs transfer so well to distributed systems and why shared-memory idioms transfer so badly. An actor owns its state and communicates only by message — which is not a stylistic preference at this scale, it is the only thing the hardware offers. Concurrency teaches both models; here one of them is simply unavailable.
It also explains a recurring category of bug: an in-process cache in a service that runs on many replicas. Each replica has its own copy, invalidation reaches one of them, and the system behaves differently depending on which replica served the request. The code looks single-node and the deployment is not.
1// One process: this works. The mutex creates a happens-before2// edge, so the read sees the write.3mutex.lock()4if (seats.available > 0) { seats.available -= 1; confirm() }5mutex.unlock()6 7// Many replicas: this compiles, runs, passes tests on one instance,8// and oversells. Each replica has its own `seats`, its own mutex,9// and no way to observe the others.10mutex.lock()11if (seats.available > 0) { seats.available -= 1; confirm() } // <-- one copy of many12mutex.unlock()13 14// What the boundary forces: a single owner of the invariant, and a15// conditional write that fails if the copy you read was stale.16const res = await db.query(17 'UPDATE seats SET available = available - 1 WHERE id = $1 AND available > 0',18 [showId],19)20if (res.rowCount === 0) soldOut()Staleness is a quantity, not a flaw
The instinct on meeting this problem is to try to eliminate staleness. That is available — route every read through the owner, or through a quorum — and it costs a round trip on every read plus unavailability whenever the owner is unreachable. For most reads that is a bad trade, which is why almost every real system serves most reads from a copy.
The productive move is to make staleness explicit and bounded. How old may this be before a decision made on it becomes wrong? For a product listing, seconds are fine. For "does this user still have permission", the answer determines whether you can cache a token at all — Security’s treatment of that trade-off is worth reading. For "are there seats left", no bound is acceptable and the decision has to move to the owner.
The useful reframing: you are not choosing between correct and stale data. You are choosing which *decisions* are allowed to run on a copy. A read that only displays information can tolerate a great deal; a read that guards a write usually cannot, and the fix is almost never a fresher read — it is a conditional write that fails when the assumption the read made no longer holds.
- Reads that display → serve from a copy, show the age if it matters.
- Reads that decide, where the decision is reversible → serve from a copy, detect and compensate.
- Reads that guard an invariant → do not decide on the read at all; make the write conditional and let it fail.
- Reads that must be authoritative → route to the owner or a quorum, and pay the availability cost knowingly.
What replaces the shared variable
Four constructions do almost all of the work, and each appears in its own module later. A single owner — all operations on this state go to one place, which is the simplest correct answer and the reason leader-based replication dominates. A quorum — a majority overlap makes a read see the newest acknowledged write, under stated assumptions. A conditional write — compare-and-set on a version, so a stale read cannot silently win; this is optimistic concurrency and it crosses the boundary better than any lock. A mergeable type — data structured so that concurrent updates converge without agreement.
Notice that only the last one avoids coordination entirely, and it does so by restricting what operations are expressible. That trade — expressiveness for coordination-freedom — recurs throughout the domain, and it is the honest core of "eventual consistency": not weaker guarantees for free, but a different set of operations that happen not to need agreement.
Key points
- There is no shared location across machines — only copies with ages.
- A received value is a fact about the sender’s past, not about the present.
- A lock in one process protects nothing on another machine.
- The fix for a stale read that guards a write is usually a conditional write, not a fresher read.
- Single owner, quorum, conditional write, mergeable type — those are the four replacements for a shared variable.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • State is created and mutated at some node, which is its owner for that moment.
- • Other nodes learn about it only by receiving a message describing it.
- • Between the owner’s write and the message’s arrival, every other node holds a value the owner has already superseded.
- • Any node acting on its copy is acting on the owner’s past, and may be acting on an assumption that no longer holds.
- • To make the action safe, the assumption must be re-checked where the state actually lives — as a version predicate on the write.
- • An update message is lost, so a copy stays stale indefinitely rather than briefly.
- • Updates arrive out of order and an older value overwrites a newer one.
- • An invalidation reaches some replicas and not others, so behaviour depends on routing.
- • Two nodes update their own copies concurrently and both believe they succeeded.
- • A node restarts and repopulates from a source that is itself behind.
- • Oversell: an in-process counter on many replicas allows more commitments than capacity. The operator sees a business-level violation with no error logs and a total that exceeds the limit by roughly the number of replicas.
- • Sticky stale cache: one replica missed an invalidation and serves old data. The operator sees users reporting inconsistent results that "fix themselves on refresh" — the refresh landed on a different replica.
- • Lost update: two services read-modify-write the same record and the second overwrites the first. The operator sees a field whose value corresponds to neither of two known concurrent operations.
- • Regression on restart: a node repopulates from a lagging replica and appears to move backwards in time. The operator sees data that was correct become incorrect immediately after a deploy.
- • Reading a copy needs no coordination and gives no guarantee about currency.
- • Making a copy authoritative requires routing to an owner or a quorum — one extra round trip per operation, and unavailability when the owner or the quorum cannot be reached.
- • A conditional write buys most of the safety for none of the read cost: it does not prevent a stale read, it prevents a stale read from causing an incorrect write.
- • Each node continues to serve from its own copy, so availability is high and divergence grows.
- • The owner’s state remains authoritative and correct throughout; it is the copies that are wrong.
- • Any invariant that requires seeing all copies at once is unenforced until they reconverge.
- • Detect: compare copies against the owner on a schedule; divergence rarely announces itself.
- • Contain: bound the age of any copy used for a decision, and refuse to decide on a copy older than the bound.
- • Recover: repair from the owner rather than from a peer copy, so an error cannot propagate laterally.
- • Reconcile: for concurrent updates that both succeeded, apply a documented merge rule — see Only the Application Knows What the Merge Means and Version Vectors: Making the Conflict Visible.
- • Verify: re-check the invariant itself after reconvergence, not merely that the copies now match.
- • Age of every cached or replicated copy at the moment it is used, propagated with the value rather than inferred.
- • Conditional-write rejection rate — a healthy non-zero number that spikes when staleness rises.
- • Per-replica divergence from the owner, sampled, so a stuck replica is visible before a user finds it.
- • Ratio of decisions made on copies to decisions routed to the owner, which is the real map of where correctness depends on freshness.
- • Reasoning this way is essential wherever the same logical state exists in more than one place — which, once there is a cache or a replica, is everywhere.
- • It is most valuable when an in-process data structure is about to be relied on for correctness in a service that scales horizontally.
- • Routing genuinely read-only display data through an owner for freshness buys latency and an availability dependency for no correctness gain.
- • Introducing version predicates on records that only one writer ever touches adds ceremony with no contention to protect against.
- • Keep the state in one place and do not copy it: a single database row is a shared variable with a real memory model, and is the right answer more often than the architecture diagram suggests.
- • Make the state immutable and append-only, so copies can never disagree about an existing value — only about how much of the log they have seen.
- • Choose a data type that merges deterministically, so concurrent copies converge without an owner.
- • Push the decision to the owner as a command rather than pulling state to the decider: send "decrement if positive" instead of reading, deciding and writing.
No shared variable — only copies with ages
// reads-then-writes on a copy: the gap is structural
const stock = replica.get("sku-9") // true at the owner, some time ago
if (stock > 0) owner.decrement("sku-9") // <- the owner may already be at 0
// the same operation with no gap: the check happens where the state lives
owner.decrementIf("sku-9", { expectedVersion: stock.version })
// -> either it applies, or it fails and tells you your copy was oldWhat people believe, and what is true
A distributed lock gives me the shared variable back.
It gives you mutual exclusion *if* the lock service is reachable, the lease has not expired unnoticed, and the holder is fenced when it is. See The Stale Lock Holder: A Paused Process Does Not Know It Was Paused — the failure mode is a holder that no longer holds it and does not know.
A fresher read fixes the race.
It narrows the window. Any read followed by a separate write has a gap; only a conditional write closes it.
Strong consistency means everyone sees the same value at the same time.
It means operations appear to take effect in some single order consistent with real time. Nobody observes a global instant — that is not available.
My in-memory cache is fine because it has a short TTL.
A TTL bounds how long a copy is wrong, not whether it is. If the decision cannot tolerate the TTL, the TTL is not the mechanism you need.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
There is no shared memory across machines. Every node holds a copy, every copy has an age, and "the current value" is not something any node can observe.
Practical
List every place a piece of state exists. Name one owner. Everything else is a cache, so give it an age bound and decide what a reader does when it turns out to be wrong. Where a read guards a write, move the check into the write as a version predicate instead of trying to make the read fresher.
Advanced
The reason a mutex works in a process and not across machines is that a mutex is not really about exclusion — it is about establishing a happens-before edge in a memory model that both parties obey. There is no shared memory model between machines, so the edge has to be constructed explicitly out of messages, which is what Lamport clocks and version vectors are for. This is why Happens-Before: The Only Ordering You Actually Have belongs to the foundations of this domain rather than to an advanced corner of it.
Apply it
- 🔧 Find an in-process cache in a horizontally-scaled service and write down what user-visible behaviour differs depending on which replica handles the request.
- 💬 Why does a mutex not solve a race between two replicas of the same service?
- 💬 A read shows 5 seats available and the write oversells. Where is the bug, and why is a fresher read not the fix?
- 💬 What is the difference between a stale read and an incorrect read?