The question this answers
How much ordering does my system actually need, and what does each level cost?
Four delivery guarantees, in increasing strength. None: messages may arrive in any order. FIFO: messages from the same sender arrive in send order; nothing is promised across senders. Causal: if a → b then every node delivers a before b; concurrent messages may be delivered in different orders at different nodes. Total: every node delivers every message in the same order, whether or not that order reflects causality.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
Under none and FIFO, a node knows only what has arrived, and cannot tell a missing message from a slow one. Under causal, a node knows whether it has all the dependencies of the message it is holding — but not whether more messages are coming. Under total, a node knows a message's position in the global sequence only once the ordering protocol has *decided* it, which is precisely why total ordering costs a round trip and blocks when a majority is unreachable.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The ladder
Each rung rules out a class of anomaly and adds a cost. Read the table as a menu with prices, not as a quality scale — the correct choice for most systems is not the top rung.
Two properties of the ladder are worth stating explicitly. First, the strengths are nested: total implies causal implies FIFO implies none. A system providing total order provides all of them. Second, the costs are not smoothly graded: the step from causal to total is qualitatively different from the others, because it is the step from "computable locally from metadata" to "requires agreement between nodes". That single step is where availability goes.
| Guarantee | Rules out | Cost | Available under partition? | |
|---|---|---|---|---|
| Noneprotocol | Nothing | Anything | Zero — this is what the network gives you | Yes, fully |
| FIFO (per sender)protocol | Reordering of one sender's own messages | A per-sender sequence number and a small reorder buffer | Per-sender counter; buffer on the receiver | Yes, fully |
| Causalprotocol | Effect before cause; reply before comment; post visible after unfollow | Causal metadata on every message; buffering until dependencies arrive | O(N) metadata, unbounded buffering delay | Yes — this is the ceiling for available systems |
| Totalprotocol | Any disagreement between nodes about order | All of the above, plus divergent replica states | Consensus: a round trip and a reachable majority per decision | No — blocks on the minority side |
FIFO is cheaper and weaker than people assume
FIFO is the ordering most systems accidentally have, because TCP provides it per connection and most brokers provide it per partition. It is genuinely useful: it means a sender's own sequence of actions is never scrambled, which handles a large fraction of real bugs.
What it does not handle is any dependency that passes through a *third party*. Alice posts a comment via service X; Bob replies via service Y. Two senders, so FIFO promises nothing, and Bob's reply can be delivered before Alice's comment. The anomaly is not exotic — it is the ordinary case whenever a user's action on one path causes an action on another.
The other trap: FIFO per *connection* or per *partition* is not FIFO per *sender*. Reconnect and you may have a new connection with an independent order. Rebalance a consumer and the partition assignment changes (Rebalancing: Everyone Stops So the Partitions Can Move). Increase partition count and the key-to-partition mapping shifts, so two messages for the same entity land in different partitions and the ordering you were relying on quietly disappears — with no error and no deploy of your code.
Causal order: the best you can have while staying up
Causal delivery holds a message until every message it depends on has been delivered. The dependency set comes from the causal metadata discussed in Vector Clocks: Buying Concurrency Detection at O(N), and the mechanism is a buffer plus a check.
This eliminates every effect-before-cause anomaly, which covers the overwhelming majority of ordering bugs users actually notice. And it does so with no agreement between nodes — each node decides locally when a message is ready, from information the message carried. That locality is why causal consistency remains available during a partition (Causal Consistency: Never Show an Effect Before Its Cause, CAP: What the Theorem Actually Says), and why it is the strongest model you can offer without giving up availability.
The costs are real and worth naming. Metadata scales with the number of actors. The buffer can grow without bound if a dependency is lost, so you need a policy for when it fills — usually requesting the missing dependency, and eventually falling back to a full state sync (Anti-Entropy: Repairing Divergence Nobody Reported). And delivery latency is now determined by the *slowest dependency*, so a single stuck message stalls everything causally downstream of it, on that replica only.
Crucially, causal order still permits nodes to disagree about concurrent messages. Node 1 may deliver x then y; node 2 may deliver y then x; both are correct. If your application needs those two to be seen identically everywhere — because an invariant spans them — causal is not enough and you have arrived at total order.
Total order, and why the last step costs so much more
Total order means every node delivers every message in the same sequence. It is the model that makes distributed programming feel like single-machine programming: replicas applying the same sequence of deterministic operations end up in the same state, which is state-machine replication, which is how a replicated database keeps its replicas identical (The Raft Log: Commit Index, Divergence and Reconciliation).
The price is discontinuous. Deciding a position in a global sequence is not computable from information a message carries, because a message that has not arrived yet might belong earlier. Somebody has to decide, and every node has to accept the decision — which is agreement, which is consensus. Total Order Broadcast Is Consensus Wearing a Different Hat shows the equivalence explicitly.
Practically this means: a round trip to a quorum before delivery; unavailability on the side of a partition without a majority; a throughput ceiling set by the sequencing point; and latency bounded below by the distance to the furthest quorum member (The One Number You Cannot Optimise for the multi-region version). None of that is an implementation weakness to be optimised away — it is what agreement costs.
The design conclusion, which is the actual point of this lesson: pick the weakest ordering that preserves your invariants, and localise the strong ordering where you genuinely need it. Most systems need total order for a small subset of operations — the ones enforcing a uniqueness or balance invariant (Start From the Invariant, Not From the Architecture) — and causal or FIFO for everything else. Applying total order globally because it is easier to reason about is the most common way teams buy an availability problem they did not need.
operation needs why -------------------------------------------------------------------- update user avatar none last one is fine, no invariant append to activity feed causal reply must not precede comment increment view counter none commutative; order irrelevant transfer between accounts total balance invariant spans both writes assign a unique username total uniqueness is a global invariant update per-user preferences FIFO single writer per key, own order matters Roughly: two of six operations need consensus. Applying it to all six buys nothing for four of them and costs availability for all six.
Key points
- None ⊂ FIFO ⊂ causal ⊂ total: each strictly stronger, each strictly more expensive.
- FIFO orders one sender's messages only, and is per connection or partition in practice — a reconnect or rebalance can silently remove it.
- Causal delivery eliminates effect-before-cause anomalies with local decisions and no agreement, which is why it survives partitions.
- Causal order still lets nodes disagree about concurrent messages. If that disagreement breaks an invariant, you need total order.
- Total order requires consensus: a round trip, a reachable majority, and unavailability for the minority side.
- Choose the weakest ordering that preserves your invariants, and localise total order to the operations that genuinely need it.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • None: deliver each message on arrival. No state, no buffer.
- • FIFO: sender attaches a per-destination sequence number; receiver buffers out-of-order arrivals until the gap is filled.
- • Causal: sender attaches its causal metadata; receiver holds a message until every dependency in that metadata has been delivered locally.
- • Total: messages are submitted to a sequencing protocol that assigns positions; a node delivers position
konly after positions1..k-1. - • Total order additionally requires the sequence to be agreed and durable, which is where consensus enters.
- • A reorder buffer grows without bound because the message filling the gap was lost.
- • A consumer rebalance or reconnection resets a per-sender sequence and the receiver sees a gap it can never fill.
- • Partition count changes and two messages for the same key take different paths, silently dropping the FIFO property they relied on.
- • The sequencer for a total order becomes unavailable and every ordered operation stops.
- • Causal metadata is stripped by an intermediary, converting causal delivery into FIFO without any error.
- • Stalled key under causal delivery: one replica stops applying updates for a specific entity because a dependency never arrived. The operator sees that entity frozen on one replica while everything else is healthy, and no errors anywhere.
- • Unbounded buffer growth: memory climbs on receivers holding messages waiting for dependencies, ending in an OOM that looks like a memory leak rather than an ordering failure.
- • Ordering silently lost after a scaling change: partition count increases, per-key ordering disappears, and the operator observes a rise in "impossible" state transitions with no code change to blame.
- • Total-order stall on the minority side: after a partition, the smaller side accepts no writes at all. The operator sees a hard availability drop confined to one zone, with healthy processes and healthy hosts.
- • Head-of-line blocking in a totally ordered stream: one slow operation delays every subsequent one, and latency rises across unrelated keys because they share a sequence.
- • None and FIFO: no coordination between nodes; FIFO needs only per-sender state.
- • Causal: no agreement, but nodes must exchange enough metadata to reconstruct dependencies — coordination in bandwidth, not in round trips.
- • Total: agreement per message (or per batch), requiring a reachable majority. This is the availability cost, and it is the reason Coordination Couples Availability is a design axis rather than a performance detail.
- • A common and effective middle path is to make total order *scoped*: totally ordered within a partition, causally ordered across partitions, so the expensive guarantee applies only where an invariant lives.
- • None and FIFO keep delivering during a partition; nodes simply diverge.
- • Causal keeps delivering, and correctly reports cross-partition writes as concurrent afterwards.
- • Total stops on the side without a majority. That is not a bug; it is the guarantee being honoured.
- • After healing, causal systems have conflicts to merge; totally ordered systems have a single history and nothing to merge — the work was paid for up front.
- • Detect: monitor reorder-buffer depth and age per replica; age is the actionable signal, exactly as with queues — Performance owns the queue-age argument, linked below.
- • Contain: bound buffers explicitly, with a defined policy on overflow — request the dependency, drop with an alert, or fall back to state transfer.
- • Recover: fetch missing dependencies directly from a peer; for large gaps, take a snapshot rather than replaying.
- • Reconcile: for systems that ran without the ordering they assumed, reconcile the resulting state — see Reconciliation Is a Component, Not a Cleanup Script.
- • Verify: assert the property you claim by injecting reordering in test and confirming no downstream anomaly appears.
- • Reorder buffer depth *and* age per replica and per key range.
- • Rate of messages delivered out of causal order, which should be zero if you claim causal delivery — an excellent invariant to assert continuously.
- • For total order: time to sequence a message, and the count of operations rejected because no majority was reachable.
- • Head-of-line blocking indicator: latency of the median operation in an ordered stream versus the slowest concurrent one.
- • Sequence gaps per sender, which detect lost FIFO after a reconnect or rebalance.
- • Explicitly choosing a rung helps most when a team is about to apply the strongest one everywhere by default — this lesson is the counter-argument, with prices attached.
- • Causal is the right default for user-facing replicated data where availability matters and effect-before-cause anomalies are visible.
- • Total is right, and worth its cost, for the small set of operations enforcing a global invariant.
- • Imposing total order on high-volume, commutative operations (counters, view logs, telemetry) buys nothing and costs throughput and availability.
- • Causal delivery hurts when dependency chains are long and one slow link stalls everything behind it.
- • Relying on FIFO from infrastructure you do not control is a latent failure: the guarantee can disappear during a reconfiguration you did not make.
- • Design operations to be commutative so ordering stops mattering — the cheapest possible answer, and the one behind CRDTs: Deterministic Merge, Not Correct Merge.
- • Partition so that everything needing an order shares a partition, then use the partition's local total order: A Topic Is Not One Log: Ordering Lives Inside a Partition.
- • Use idempotent, self-describing operations so a reordered delivery is detectable and discardable: Idempotent Is a Property of the Whole Effect, Not the Write.
- • Order at read time rather than at write time — store an unordered set with causal metadata and resolve when someone looks: Only the Application Knows What the Merge Means.
- • Accept the anomaly and correct it in the interface, which is a legitimate choice when the cost of the anomaly is a mild confusion and the cost of ordering is availability.
Four orderings, four prices
- 1m2reply to that commentbefore its cause
- 2m1comment on the photo
- 3m4post the close-friends photobefore its cause
- 4m3remove Bob from close friends
- 5m5update avatar
- 1m1comment on the photo
- 2m5update avatar
- 3m4post the close-friends photobefore its cause
- 4m3remove Bob from close friends
- 5m2reply to that comment
| Rules out | Cost | Available under partition? | |
|---|---|---|---|
| Noneprotocol | Nothing | Zero — this is what the network gives you | Yes, fully |
| FIFO (per sender)protocol | Reordering of one sender’s own messages | A per-sender sequence number and a small reorder buffer | Yes, fully |
| Causalprotocol | Effect before cause; a reply before its comment | Causal metadata on every message; buffering until dependencies arrive | Yes — the ceiling for an available system |
| Totalprotocol | Any disagreement between nodes about order | Consensus: a round trip and a reachable majority per decision | No — blocks on the minority side |
operation needs why -------------------------------------------------------------------- update user avatar none last one is fine, no invariant append to activity feed causal a reply must not precede its comment increment view counter none commutative; order irrelevant transfer between accounts total the balance invariant spans both writes assign a unique username total uniqueness is a global invariant update per-user preferences FIFO single writer per key, own order matters
What people believe, and what is true
The queue guarantees ordering, so my messages are in order.
It guarantees ordering within a partition or connection. Across partitions, across producers, or after a rebalance, there is no such guarantee.
Causal ordering means all nodes see the same order.
It means all nodes agree on the order of *causally related* messages. Concurrent messages may legitimately be delivered in different orders on different nodes.
Total order is just causal order plus tie-breaking.
Tie-breaking is deterministic and local; total order requires all nodes to agree on the same sequence including messages that have not arrived everywhere yet. That is consensus.
Stronger ordering is safer, so choose total when unsure.
Stronger ordering trades availability for order. Choosing total "to be safe" makes your system unavailable during partitions to protect invariants most of your operations do not have.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Four levels of ordering: none, per-sender FIFO, causal, and global total. Each rules out more anomalies and costs more coordination, and the last one costs consensus.
Practical
List your operations and mark which have an invariant spanning two writes. Those need total order, scoped as narrowly as possible. The rest need causal at most. Monitor reorder-buffer age, and never assume a FIFO guarantee survives a rebalance or a partition-count change.
Advanced
The jump from causal to total is the jump from locally computable to agreed. Causal delivery is decidable from metadata a message carries; a total position is not, because an unarrived message may belong earlier. This is exactly why total-order broadcast and consensus are equivalent, and why the FLP impossibility applies to the top rung of the ladder and not the others.
Apply it
- 🔧 Take a list of your system's write operations and assign each the weakest ordering that preserves its invariants.
- 🔧 Instrument a claimed causal-delivery path with an assertion that a message is never delivered before a dependency, and run it under injected reordering.
- ⚡ After scaling a topic from 6 to 12 partitions, support reports users seeing profile updates apply out of order. Nothing was deployed. Explain.
- ⚡ A team proposes routing every write through a consensus group "for consistency". Estimate what that costs them, in availability and in latency, and what they get.
- 💬 Name the four ordering models and what each costs.
- 💬 Your broker gives per-partition ordering. What breaks when you double the partition count?
- 💬 Which of your operations genuinely need total order? How would you argue the case for one of them?