The question this answers
I have to pick a consistency level for this operation. How do I decide without guessing?
This lesson provides a procedure, not a system property: name the invariant, determine whether it is locally checkable, price the coordination it would require, and choose the weakest model that protects it — recorded per operation, with the partition behaviour specified. The guarantee you end up with is whatever that procedure selects, and its scope is one operation, not the system.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The design question reduces to a question about node knowledge: can a single node, using only state it holds locally, determine that this operation is safe? If yes, the operation needs no coordination and any model will do. If no — if safety depends on what some other node has accepted — then coordination is required and the model must be strong enough to obtain it. Every decision below is that one question applied.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The procedure
Five steps, in order. The discipline is doing them in order, because starting at step four ("which database?") is how teams end up with a cluster-wide setting that is simultaneously too strong for 90% of traffic and too weak for the operation that matters.
Step three is the one people skip and the one that pays. Most operations in most systems require no coordination at all, and finding that out is what makes the remaining coordination affordable.
1. NAME THE INVARIANT Not "we need consistency". "No two users may hold the same handle." "An order is never charged twice." "Stock never goes negative." If you cannot write it as a sentence about the data, you do not yet have a requirement — you have an anxiety. 2. ASK IF ONE NODE CAN CHECK IT Can a single replica, from local state alone, decide this operation is safe? Appending to a set: yes. Claiming a unique handle: no. Decrementing stock past zero: no. Editing a document body: yes. -> "no" is the ONLY reason to buy coordination. 3. PRICE THE COORDINATION How many nodes must be reached, how far away are they, and what happens to this operation during a partition? A cross-region linearizable write is ~100ms and unavailable on the minority side. Is the invariant worth that, on THIS operation's volume? 4. CHOOSE THE WEAKEST MODEL THAT HOLDS eventual -> session guarantees -> causal -> linearizable per key -> strictly serializable. Stop at the first one that protects the invariant. Going further is money spent on nothing. 5. WRITE DOWN THE PARTITION BEHAVIOUR For this operation: serve stale, refuse, or degrade? Unwritten, this is decided by a timeout default during your first incident.
The ladder, and where most operations land
Walking the ladder from the bottom rather than the top is the single most useful habit here. Each rung costs more and forbids more, and the correct choice is the first rung that forbids what must not happen.
In practice the distribution is lopsided. The large majority of operations in a typical application land on the first two rungs — they need convergence and coherent per-user behaviour, and nothing more. A small number carry real invariants and need the top rungs. Recognising that the split exists, and that it is not 50/50, is what makes strong consistency affordable where it is genuinely needed.
| Model | Buy it when | Typical share of operations |
|---|---|---|
| Eventualtypical | Convergence is all that is required; no user-visible coherence needs | Large — feeds, counts, caches, recommendations, telemetry |
| Session guaranteestypical | A user must see their own activity coherently | Large — nearly every user-facing read path |
| Causaltypical | Cross-user ordering must be coherent: replies, edits, notifications | Small — collaborative and social features |
| Linearizable per keytypical | A single-object invariant must never be momentarily violated: locks, uniqueness, CAS | Small — but these are the operations that hurt when wrong |
| Strictly serializabletypical | An invariant spans multiple objects and must hold in real-time order | Smallest — ledgers, inventory transfers, financial state |
Three heuristics that resolve most arguments
Monotonic operations need no coordination. If an operation only ever adds information — appending to a set, incrementing a counter that is never checked against a limit, recording an event — replicas that have seen the same operations agree, in any order. Coordination becomes necessary exactly when the operation involves a negation: *no* duplicate, *not* below zero, *at most* one. That is the CALM result stated usefully, and it converts "do we need strong consistency?" into a question you can answer by inspecting the operation. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
Move the invariant, do not strengthen the system. If two fields must stay consistent, putting them in one object turns a distributed problem into a local one. Reserving stock as an explicit row with a compare-and-set turns an inventory invariant into a single-key linearizable operation. Restructuring is nearly always cheaper than raising the consistency level of everything around it. See Start From the Invariant, Not From the Architecture.
Price the violation, not just the coordination. "What does it cost if this invariant breaks once a month?" is a question with an answer. For a duplicate username: a support ticket. For a double-charged card: a chargeback, a refund, and a trust cost. For an oversold seat: a compensation policy that may already exist. Some invariants are cheaper to violate and reconcile than to enforce, and saying so out loud is an engineering judgement rather than a lapse. See Reconciliation Is a Component, Not a Cleanup Script.
- Adds information only → no coordination needed, whatever the model.
- Involves a negation, a limit, or a uniqueness claim → coordination is required and no merge rule substitutes.
- Invariant spans objects → restructure into one object before reaching for distributed transactions.
- Violation is cheap and reconcilable → consider allowing it and detecting it, rather than preventing it.
Worked example: an ordinary checkout
One flow, five operations, four different answers. Note that only one of them needs coordination at all, and it is not the one people usually reach for first.
The product listing and the recommendations converge and nobody notices. The cart is per-user and needs the user's own view to be coherent — session guarantees. Order status needs read-your-writes so the confirmation page is not empty. Stock reservation is the one real invariant, and it is single-key, so it needs linearizable compare-and-set on one row and nothing more. Payment capture is idempotent by key rather than consistent by model, which is a different tool entirely — see Idempotent Is a Property of the Whole Effect, Not the Write.
| Operation | Invariant | Model chosen | Partition behaviour |
|---|---|---|---|
| Browse productstypical | None | Eventual | Serve stale from local replica |
| Cart contentstypical | User sees own cart coherently | Session guarantees | Serve with session floor; leader fallback |
| Reserve stockprotocol | Never oversell | Linearizable CAS on one key | Refuse on the minority side |
| Order statustypical | User sees own order | Read-your-writes | Serve with floor; leader fallback |
| Capture paymentprotocol | Never charge twice | Idempotency key, not a consistency model | Retry safely on either side |
Key points
- Start from the invariant, stated as a sentence about the data. Without one you cannot choose.
- Coordination is needed exactly when a single node cannot verify safety from local state.
- Walk the ladder from the bottom and stop at the first model that protects the invariant.
- Operations that only add information need no coordination; negations, limits and uniqueness do.
- Restructuring the data to make an invariant single-object is usually cheaper than strengthening the model.
- Decide per operation, and write down the partition behaviour before the partition happens.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Enumerate the operations in the flow rather than treating the system as one thing.
- • For each, write the invariant it must preserve, or "none".
- • Classify each invariant as locally checkable or not.
- • For those that are not, price the coordination in latency and in partition-time availability.
- • Select the weakest model that protects the invariant, and record the partition behaviour alongside it.
- • Re-run the exercise when the deployment topology changes — a single-region decision does not survive a multi-region rollout unexamined.
- • The invariant is never stated, so the decision defaults to whatever the cluster setting was.
- • A cluster-wide level is applied to operations with entirely different needs.
- • The decision is made for one region and silently inherited by a multi-region deployment where the coordination cost is ten times higher.
- • The partition behaviour is left unspecified and gets decided by a client timeout default during an incident.
- • An invariant that spans objects is assumed to be covered by per-key linearizability.
- • Over-coordination: uniform strong consistency makes the whole system as slow as its most demanding operation, and the latency is blamed on the database rather than on the choice. Observed as p99 that does not respond to capacity increases.
- • Under-coordination: a real invariant runs at a weak level and is violated rarely under load. Observed as oversells, duplicate identities or negative balances that appear a few times a month with no error signal.
- • Multi-region regression: a consistency decision that was cheap in one region becomes a cross-continental round trip. Observed as a latency step change tied to a topology rollout, not to load.
- • Undefined partition behaviour: during the first real partition, each service does whatever its client library's default timeout implies, and the resulting mix of stale answers and errors is impossible to reason about afterwards.
- • Wrong-tool substitution: an idempotency requirement is addressed by raising the consistency level, which is expensive and does not fix duplicates. Observed as continued double-effects despite a strong-consistency migration.
- • The entire procedure is a method for finding the minimum coordination that preserves correctness.
- • Coordination bought where it is not needed costs latency in normal operation and availability during partitions — it is charged twice. See Coordination Couples Availability.
- • Coordination avoided where it *is* needed costs correctness, and the bill arrives later and larger.
- • Operations at weak levels continue during a partition and may diverge.
- • Operations at strong levels refuse on the minority side, and that refusal is the guarantee working.
- • A mixed system behaves in a mixed way, which is only comprehensible if the per-operation decisions were written down.
- • Detect: assert the invariants continuously over the data, since a model mismatch produces no errors and only invariant checks reveal it.
- • Contain: make degraded consistency levels explicit and observable rather than silent.
- • Recover: for weak-level operations, run repair and convergence checks; for strong-level ones, resume when quorum returns.
- • Reconcile: for invariants violated while running at too weak a level, reconciliation is application work — and having decided in advance which invariants those are is what makes it tractable. See Reconciliation Is a Component, Not a Cleanup Script.
- • Verify: revisit the per-operation table after every topology change and after every incident.
- • Continuous invariant assertions — the single most valuable signal, because model mismatches are silent otherwise.
- • Consistency level in force per operation, as a metric dimension.
- • Latency attributable to coordination, per operation, so the price of each decision is visible.
- • Quorum-loss and stale-serve counts, showing which side of each partition decision is actually being exercised.
- • Drift between the documented per-operation table and the levels observed in production.
- • Any new service touching replicated data, before the first consistency level is configured.
- • Latency investigations where coordination is suspected but has never been attributed per operation.
- • Multi-region rollouts, which invalidate single-region consistency decisions silently.
- • Post-incident review after an invariant violation, where the question is which rung was needed.
- • Single-node systems, where the answer is strict serializability for free and the analysis is wasted.
- • Prototypes and internal tools where the invariants are trivial and the exercise delays useful work.
- • As a way to avoid deciding — the procedure exists to reach a decision quickly, not to generate documents.
- • Default to the strongest model and revisit only if latency becomes a problem — a legitimate strategy for small single-region systems where coordination is cheap.
- • Design the operations to be monotonic so the question never arises. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It and CRDTs: Deterministic Merge, Not Correct Merge.
- • Allow the violation and reconcile it, where reconciliation is cheaper than prevention. See Reconciliation Is a Component, Not a Cleanup Script.
- • Move the invariant into a single object so per-key linearizability suffices. See Start From the Invariant, Not From the Architecture.
Choosing a consistency model without guessing
| What it protects | What it costs | Where most operations actually belong | |
|---|---|---|---|
| Eventualprotocol | Nothing observable | Nothing at write time | Anything with no invariant: feeds, comments, view counts |
| + session guaranteestypical | A session’s view of its own activity | A token the client carries | Almost every user-facing read path |
| + causalprotocol | Effects never precede their causes | Dependency metadata on every update | Threads, comment trees, collaborative state |
| Linearizable per objectprotocol | A single-copy view of one object | Coordination on every operation on that object | Locks, leases, uniqueness, one-shot actions |
| Strictly serializableassumption | Groups of operations, ordered by real time | Coordination plus concurrency control | The genuinely transactional handful |
What people believe, and what is true
Pick one consistency model for the system.
Consistency is a per-operation decision. A single cluster-wide level is simultaneously too strong for most traffic and too weak for the operations carrying invariants.
Start by choosing a database.
Start by naming the invariant. The invariant determines what coordination is required; the database is the last decision, not the first.
Stronger is safer, so default to strong.
Stronger is safer per operation and less available during partitions, and it costs latency everywhere. A system too slow to use is not safe either. Buy the weakest rung that holds.
If we need one strong operation we need a strong system.
One linearizable key alongside an eventually-consistent store is a completely ordinary architecture, and usually the cheapest correct one.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Name the invariant. Ask whether one node can check it alone. If yes, no coordination is needed. If no, buy the weakest model that protects it — for that operation only.
Practical
Build the per-operation table: invariant, model, partition behaviour. Expect most rows to need nothing stronger than session guarantees, and expect one or two to carry the real invariant. Then assert those invariants continuously, because a wrong choice produces no errors.
Advanced
The local-checkability test is CALM: a computation has a coordination-free distributed implementation if and only if it is monotonic. Every invariant that needs coordination contains a negation — no duplicate, not below zero, at most one — and every one that does not is purely additive. That reduces model selection from taste to inspection, and it explains why moving an invariant into a single object works: it converts a distributed non-monotonic check into a local one, without weakening anything. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It and Start From the Invariant, Not From the Architecture.
Apply it
- 🔧 Take one flow in a system you work on, write its per-operation table with all three columns filled, and find at least one operation currently paying for coordination it does not need.
- ⚡ A social app adds a second region. Re-run the per-operation table and identify which decisions the topology change invalidated.
- ⚡ An inventory service oversells twice a month. Determine whether the correct fix is a stronger model, a restructured invariant, or a reconciliation process.
- 💬 Walk me through choosing a consistency level for "reserve one seat on this flight". What is the invariant, and what is the weakest model that protects it?
- 💬 Which operations in a checkout flow need coordination and which do not? Justify each.
- 💬 Your team wants to migrate everything to a strongly consistent database to fix a duplicate-charge bug. What do you say?