The Distributed Systems Lab

Two machines, one rack. The first genuinely distributed row. Clocks agree approximately; silence has five causes. Both clauses of the test hold: a component your correctness depends on fails independently of you, and the only way you learn its state is a message that can be lost, delayed, reordered or duplicated. Two nodes or two thousand — the reasoning is the same.

try {
  const receipt = charge(order)   // returned => it happened, once
  markPaid(order, receipt)
} catch (e) {
  markFailed(order)               // threw => it did not happen
}

The interesting question is never how fast a remote call is. It is what may I conclude from what I observed. Three of these eight worlds produce identical evidence at the caller, and two of those three left durable effects behind. What crosses the boundary is data you deliberately sent — an id, a deadline, an idempotency key. Everything a local call gets from the runtime for free must be carried explicitly.

TCP removes reordering and duplication at byte level and converts loss into delay. Your own retries reintroduce duplication at the application level, and a connection reset discards in-flight data while telling you nothing about how much the peer consumed.

Every copy carries the owner’s latest version from step 11 onward. Until then, a reader on a replica sold stock that no longer existed — and nothing errored, because from that replica’s point of view its value was correct. A fresher read narrows the window; it does not close it. Any read followed by a separate write has a gap. The fix for a stale read that guards a write is a conditional write — send the version you believed and let the owner reject you — not a faster refresh.

// reads-then-writes on a copy: the gap is structural
const stock = replica.get("sku-9")   // true at the owner, some time ago
if (stock > 0) owner.decrement("sku-9")   // <- the owner may already be at 0

// the same operation with no gap: the check happens where the state lives
owner.decrementIf("sku-9", { expectedVersion: stock.version })
// -> either it applies, or it fails and tells you your copy was old

A node knows its own clock reading. It cannot measure its offset against a peer — only against a time server, over a round trip whose two directions it assumes took equal time. Comparing two hosts’ timestamps inherits the sum of both errors. Causality is the ordering that needs no assumption: it is established by messages, it is partial, and the events it refuses to order are genuinely unordered.

The cheapest rung on the ladder. Most "we need to scale" is this one. State the goal as a number with a horizon before choosing a topology: 3× read in 12 months. A requirement in that form can be evaluated; "we need to scale" cannot, and it is how teams end up with a sharded system that only ever needed read replicas.

Each service here is individually 99.90% available and individually fast. The operation is 99.6% available, because availability is a property of the operation across all of its dependencies rather than of any one service. Every crossing adds a median 12 ms that a local call did not cost, and a chance of the third outcome that a local call could not produce. A module boundary inside one deployable buys most of the organisational benefit at none of this price, and it is the reversible option — moving a module boundary is a refactor; moving a service boundary is a data migration plus a contract change plus a coordinated deploy.

No invariant spans this cut, so each side can enforce its own rules locally. One order crosses it rarely, which is what a well-placed boundary looks like. The durable form of a boundary is single-writer ownership: separate storage is a consequence of that rule, not the rule itself.

At "partitions" scope the predicate can be evaluated by no single owner, so the price is paid in round trips and in availability coupled across participants. Nothing here is checking the predicate. Turn the check on: this failure class produces no errors, so an unmeasured invariant is an unenforced one.

Weakening is a design tool, not a defeat. Localise: make the seat its own partition and the invariant becomes single-owner. The four legitimate moves are localise, bound, detect-and-compensate, and choose operations that commute. The one that is not available is leaving a required invariant unenforced and calling it eventual consistency — convergence preserves a violation faithfully.