Agreement Costs Round Trips
Every guarantee that several machines agree on something is paid for in round trips. A quorum write is at least one; consensus is more; a distributed lock is two plus however long the holder keeps it. The guarantee is often worth it — the cost is never zero.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The round-trip price list
Coordination mechanisms differ enormously in what they guarantee and, correspondingly, in how many network round trips they need before anyone can proceed. Because a round trip has a floor set by distance (see Cross-Region Latency Is Physics, Not Configuration), the same mechanism can be nearly free within a rack and brutal across regions — which is why coordination decisions and topology decisions cannot be made independently.
The table below is about *round trips*, not about correctness. Each mechanism buys a genuinely different guarantee, and paying more round trips for a guarantee you actually need is a good trade. The failure mode is paying for a guarantee you did not need: a distributed lock protecting an operation that was already idempotent, or a strongly consistent read serving a page that would have been fine with data a second old.
The most expensive item on the list is usually not in the table at all: the *hold time* of a lock. A lock held across a network call to a slow dependency serialises every other request that needs it for the entire duration, which converts a latency problem into a throughput ceiling. That is queueing, and it behaves exactly as described in Queueing: Why Systems Get Slow Before They Get Broken.
| Mechanism | Round trips (typical) | What it buys | Where it hurts |
|---|---|---|---|
| Local lock (single process) | 0 — memory | Mutual exclusion within one process | Nothing distributed; scales with cores, not machines |
| Leader-local write | ≈ 1 to the leader | Ordering through a single point | Leader is a throughput ceiling and a latency floor for distant clients |
| Quorum write | ≥ 1 to a majority, in parallel | Durability across failures | Bounded by the slowest node in the majority — a tail problem |
| Consensus round (e.g. Raft-style) | ≥ 1–2 in the steady state, more on leader change | Agreed, ordered, replicated decisions | Leader elections stall writes; cross-region membership is expensive |
| Distributed lock | ≥ 2 (acquire, release) plus hold time | Mutual exclusion across machines | Hold time serialises everyone; failure needs leases and fencing |
| Two-phase commit | ≥ 2 rounds to all participants | Atomicity across resources | Blocks on coordinator failure; slowest participant sets the pace |
The lock you added is now the bottleneck
A distributed lock has a cost profile that surprises people because most of it is not the acquisition. Acquiring and releasing are two round trips; the expensive part is that while one holder has the lock, every other request needing it is queued. Throughput through the locked section is therefore bounded by one over the hold time, regardless of how many machines you add.
If the hold time is 2ms, that section supports roughly 500 operations per second and you will probably never notice. If someone adds a call to an external service inside the critical section and the hold time becomes 200ms, the ceiling drops to about 5 operations per second — a 100× throughput reduction from a change that looks, in the diff, like moving one line inside a block.
The rules that keep this survivable: never perform I/O while holding a lock; keep critical sections to memory operations; use leases so a crashed holder cannot block the system forever; and question whether the lock is needed at all. Frequently the operation can be made idempotent or conflict-tolerant instead, which removes the coordination rather than optimising it (see Idempotency).
Buying less coordination
The cheapest coordination is the coordination you do not perform. Before optimising a consensus round, it is worth asking whether the operation genuinely needs global agreement, or whether it needs something weaker that is dramatically cheaper: per-key ordering rather than global ordering, eventual convergence rather than immediate agreement, or an idempotent operation that is safe to apply more than once.
Where coordination is genuinely required, the lever is usually *scope* rather than mechanism. Partitioning so that each key's authority is a single node makes most operations leader-local instead of consensus-wide. Batching several decisions into one coordination round amortises the round trips. Keeping the participant set small and physically close reduces the RTT that every round trip is multiplied by.
And the honest position: some guarantees are worth their cost. A financial ledger that must never double-spend should pay for consensus, and an engineer who removes that coordination to improve p99 has made the system faster and wrong. The goal is knowing what each guarantee costs, so the trade is made deliberately rather than discovered during an incident.
- Do you need agreement, or idempotence? An operation safe to apply twice may need no lock at all.
- Do you need global ordering, or per-key ordering? Per-key is usually enough and is vastly cheaper.
- Can the participant set be smaller or closer? Every round trip is multiplied by the RTT between participants.
- Can decisions be batched? One coordination round amortised over many operations changes the economics entirely.
- Is the critical section doing I/O? If so, the hold time — not the round trips — is your throughput ceiling.
- Is a stale read acceptable here? Local reads with bounded staleness remove coordination from the common path.
Key points
- Every coordination guarantee is paid for in round trips, and each round trip is multiplied by the RTT of your topology.
- Distributed lock cost is dominated by hold time, not acquisition: throughput through a critical section is roughly one over the hold time.
- Performing I/O inside a critical section can cut throughput by orders of magnitude from a one-line change.
- Quorum and consensus latency is set by the slowest participant in the required set, which makes it a tail problem, not an average one.
- The cheapest coordination is none: idempotence, per-key ordering and bounded-staleness reads remove it rather than optimise it.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Request 1 → lock service: acquires the lock in one round trip; nothing looks wrong yet.
- 2Request 1 → external service: makes a 200ms call *while still holding the lock*, because the call was added inside the existing critical section.
- 3Requests 2..N → lock service: block for the full 200ms hold time; their CPU is idle and their latency is climbing.
- 4Lock section → throughput: the ceiling is now roughly one operation per 200ms — about 5 per second — regardless of how many application instances are running.
- 5Operator → dashboards: every node shows low CPU, no slow queries, and rising latency, because the constraint is a queue for a lock nobody is graphing.
- • "CPU is idle on every node, so we have capacity." Everyone is waiting for the same lock; capacity is irrelevant.
- • "Adding more instances will increase throughput." Not through a serialised critical section — the ceiling is set by hold time.
- • "The lock is fast, acquisition is 2ms." Acquisition is not the cost; hold time is.
- • "We need strong consistency here." Sometimes true. Check whether idempotence or per-key ordering would give the correctness you actually need.
- • "Consensus is slow because the algorithm is slow." It is slow because it needs round trips between participants, and your participants are far apart.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Time spent waiting to acquire locks, as a distinct span or metric rather than folded into total request time.
- • Lock hold time distribution, since the tail of hold time sets the throughput ceiling for everyone else.
- • Coordination round-trip count per operation, and whether latency scales with participant count rather than data size.
- • Quorum acknowledgement latency per participant, to identify whether one slow node is setting the pace.
- • Contention rate: how often an operation had to wait at all, which distinguishes an expensive lock from a busy one.
- • Remove I/O from critical sections so hold time is memory-bounded rather than dependency-bounded.
- • Ask whether the coordination is needed: make the operation idempotent or conflict-tolerant and delete the lock entirely.
- • Narrow the scope — per-key locks and per-key authority instead of global ones, so unrelated operations stop contending.
- • Batch decisions into fewer coordination rounds where semantics allow, amortising the round trips.
- • Keep participant sets small and physically close, and use leases with fencing so a crashed holder cannot block the system indefinitely.
- • Throughput through the previously-serialised section, which should rise roughly in proportion to the hold-time reduction.
- • Lock wait time p99 and contention rate, compared against the same window before.
- • End-to-end p99 for the affected operation, to confirm the saving reached the user rather than moving to another queue.
- • A correctness check appropriate to the guarantee you weakened — removing coordination must be validated for correctness, not only for latency.
- • Removing coordination trades a correctness guarantee for latency and throughput — sometimes correct, sometimes catastrophic, never free.
- • Per-key partitioning makes cross-key operations genuinely hard and pushes complexity into the application.
- • Batching amortises round trips at the cost of latency for the first operation in each batch.
- • Leases bound the damage of a crashed holder and introduce clock assumptions and fencing requirements of their own.
- • An alert on lock hold time p99, which is the metric that predicts the throughput ceiling before it is hit.
- • A review rule prohibiting network calls inside critical sections, since this regresses through ordinary refactoring.
- • A load test that drives the coordinated path at expected peak concurrency, since contention only appears under concurrency.
- • An invariant or property test guarding any correctness guarantee that was deliberately weakened.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVERound-trip counts are typical steady-state figures for each mechanism class, not guarantees. Specific protocols and implementations differ, and failure paths cost considerably more than the steady state.
- ENVIRONMENT-SPECIFICThe absolute cost of any coordination mechanism is its round-trip count multiplied by the RTT between participants, so the same design is cheap within a rack and expensive across regions.
Misconceptions
Apply it
Where the depth lives
The round-trip counts here are steady-state costs; the theoretical results explain why no protocol can do better under the failure assumptions each one makes.