Replication

Multi-Leader Replication: Accepting Writes in More Than One Place

Two or more nodes accept writes for the same data and replicate to each other. It buys local write latency and write availability during a partition, and it buys them by making write conflicts a structural certainty rather than a rare accident.

▶ Run the lab

The question this answers

The question

What actually changes when a second node is allowed to accept writes for the same key?

The guarantee — the property claimed, and its scope

Every leader accepts writes locally and each write eventually reaches every other leader. There is no global order of writes, so the system guarantees convergence *only* if a deterministic conflict-resolution rule is defined for every writable field — and convergence to a value is not the same as convergence to the right value.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

A leader knows its own accepted writes and the writes it has received from peers. It does not know whether a peer is currently accepting a write to the same key, and it cannot: that is what "concurrent" means here. Any belief that "no conflicting write exists" is inference over a channel that may be delivering nothing precisely because it is broken.

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?What does a node know?How does it work?What can fail?How does it fail?Where is coordination?What holds under failure?How does it recover?How would you know?What is the simpler thing?
replicationmulti-leaderconflictsmulti-region

The conflict is created by the design, not by bad luck

In leader-based replication two writes to the same key are ordered because one node saw both. Remove that node and there is nothing left to order them with. Two writes that happen "at the same time" in different regions are not merely likely to conflict — they are, by the definition of concurrency in this domain, *unordered*, and no amount of clock accuracy repairs that. See Happens-Before: The Only Ordering You Actually Have and There Is No Global Clock.

So the honest way to read multi-leader replication is: you have chosen to move conflict resolution into your application. If your answer to "what happens when both regions update the same profile field?" is "that will not happen", you have not chosen a resolution rule, you have chosen last-write-wins by accident, which silently discards one of the writes.

Two writes, no order — and neither leader did anything wrongprotocol
Leader EULeader UStitle = "Draft": deliveredtitle = "Draft"title = "Final": deliveredtitle = "Final"accept title = "Draft" (write) at t=1accept title = "Draft"accept title = "Final" (write) at t=2accept title = "Final"receive "Final" — conflict (decide) at t=7receive "Final" — conflictreceive "Draft" — conflict (decide) at t=8receive "Draft" — conflictt=1time →t=8
delivereddelayed (dashed, long)duplicated (×2)dropped — stops short, never arriveswritedecide
Each leader acknowledged its client before learning of the other write. Both are correct locally. The system now has two histories and must produce one answer — and whatever rule it uses is a product decision, not a database setting.

What it actually buys, stated precisely

The gains are real and worth naming exactly, because they are narrower than the marketing suggests. Local write latency: a user in Sydney writes to Sydney, not to Virginia. Write availability under partition: when the link between regions fails, both sides keep accepting writes — a single-leader system makes the minority side read-only. Independent capacity: each leader carries its own write load.

What it does *not* buy: a stronger consistency model (it is strictly weaker), simpler operations (it is strictly harder), or a system where the application can ignore concurrency. Any pitch for multi-leader replication that does not lead with conflict handling is selling the upside without the invoice.

PropertySingle leaderMulti-leader
Write latency for a distant userprotocolRound trip to the leader regionLocal
Write availability when regions are partitionedprotocolMinority side is read-onlyBoth sides accept writes
Order of writes to one keyprotocolTotal, decided by the leaderNone — concurrent writes are genuinely unordered
Uniqueness constraints, counters, balancesassumptionEnforceable locallyNot enforceable without coordination, which defeats the purpose
Conflict handlingprotocolNot neededRequired for every writable field
Where multi-leader genuinely differs from single-leader

The invariants that cannot survive the move

Some application rules are compatible with multi-leader replication and some are structurally impossible. The test is simple: can the rule be checked by looking at one leader's state alone? "This document's body is whatever the last editor typed" — yes. "This username is globally unique" — no, because two leaders can both see it free. "This balance never goes below zero" — no, because two withdrawals each look affordable locally.

Rules of the second kind do not become impossible; they become *coordinated*, which means the write for those specific operations must go through a single point and therefore gives up exactly the local-write property you adopted multi-leader for. The mature design is a hybrid: multi-leader for the fields that can converge, single-leader or consensus for the small set of invariants that cannot. See Start From the Invariant, Not From the Architecture and Distributed Uniqueness: One Name, Many Shards.

  • Safe under multi-leader: last-value fields with an agreed resolution rule, append-style collections, counters implemented as CRDTs, per-region-owned records.
  • Unsafe: global uniqueness, non-negative balances, capacity limits, "only one active X", monotonic sequence numbers.
  • The practical pattern is per-record home region — writes for a record are normally accepted only by its home leader, so conflicts arise only during failover.

Key points

  • Removing the single writer removes the thing that ordered concurrent writes; conflicts become structural, not accidental.
  • It buys local write latency, write availability under partition, and independent write capacity — and nothing else.
  • Every writable field needs a resolution rule. Not choosing one means choosing last-write-wins, which loses data silently.
  • Invariants checkable at one leader survive the move; global invariants like uniqueness and non-negative balances do not.
  • The mature shape is hybrid: multi-leader where convergence is enough, single-leader for the few operations that need order.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works
  • Each leader accepts writes locally, logs them, and acknowledges its client without contacting peers.
  • Each leader asynchronously streams its writes to every other leader, typically in a full mesh or a configured topology.
  • A receiving leader applies foreign writes, detecting a conflict when a write it receives targets a value that has been changed concurrently — detected by version vectors, not by timestamps. See Version Vectors: Making the Conflict Visible.
  • A resolution rule produces one value per conflicting key: last-write-wins, application merge, a CRDT operation, or storing siblings for a later reader to resolve.
  • The resolved value propagates again until all leaders hold the same value, which is convergence.
What can fail at the boundary
  • Two leaders accept concurrent writes to the same key, and there is no fact of the matter about which came first.
  • The inter-leader link fails and each side accumulates hours of divergent writes.
  • Replication topology loops cause a write to be applied repeatedly or to bounce indefinitely without an origin marker.
  • A resolution rule is not commutative or associative, so leaders converge to different values depending on delivery order.
  • A schema change or constraint is applied in one region and violated by data arriving from another.
How it fails — what an operator sees
  • Silent write loss under last-write-wins: two users edit a record seconds apart in different regions, and one edit vanishes. There is no error, no log line, and the affected user is certain they saved it.
  • Divergent replicas that never converge: leaders hold different values for the same key indefinitely because the resolution rule is order-dependent. The operator finds it via a comparison job, not an alert.
  • Constraint violations that "cannot happen": two rows with the same unique key, or a balance below zero, because both leaders validated locally against consistent-looking state.
  • Reconnection storm: after a long partition, hours of accumulated writes flood across the link at once, and the conflict-resolution work saturates both leaders while user traffic is still arriving.
  • Replication loop: a write is re-applied on each pass around the topology, showing up as a value that keeps reverting or a row count that grows without new user activity.
Where coordination is required
  • None on the write path — that is the entire purchase, and everything else is the cost of it.
  • Coordination reappears anywhere a global invariant is required, and there it must be genuine coordination (a single owner, a lock with fencing, or consensus), not a cleverer merge rule. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It for the discipline of minimising this set.
  • Conflict resolution itself is not coordination — it is a deterministic local computation, which is exactly why it can only produce convergence and never a global invariant.
What still holds under failure
  • During a partition both sides remain fully writable, and each side's local reads stay consistent with that side's own writes.
  • The two sides' histories diverge for the duration, and the divergence is proportional to write volume and partition length.
  • On heal, convergence is guaranteed only if the resolution rule is commutative, associative and idempotent; otherwise the system may not converge at all.
How it recovers
  • Detect: monitor per-link replication lag and the conflict rate. A conflict rate of zero on a genuinely multi-leader system usually means conflicts are being resolved silently, not that none occur.
  • Contain: during a long partition, consider degrading one side to read-only for the record classes whose conflicts you cannot resolve well — a deliberate reduction of the design's benefit in exchange for a smaller mess.
  • Recover: on heal, throttle the backlog exchange so conflict resolution does not starve live traffic.
  • Reconcile: keep conflict losers rather than discarding them — a conflict log lets you recover a lost edit later, which last-write-wins alone never can.
  • Verify: run a periodic cross-leader comparison over a sampled key range and alert on inequality. Convergence must be checked, not assumed. See Anti-Entropy: Repairing Divergence Nobody Reported.
How you would know
  • Conflict rate per key class, and the distribution of resolution outcomes — how often each rule fires.
  • Per-link replication lag and backlog size in entries, per direction.
  • A durable conflict log with both versions and the winner, since this is the only way a silently lost edit is ever recoverable.
  • Cross-leader divergence checks over sampled keys, run continuously rather than during incidents.
  • Constraint-violation counts for invariants that are supposed to be impossible — the count is the honest measure of how well the invariant survives.
When it helps
  • Users writing from geographically distant regions where the round trip to a single leader is felt in the product.
  • Offline-capable clients, which are multi-leader systems whether or not anyone calls them that — the device is a leader.
  • Collaborative editing and similar workloads where the data type has a genuine merge semantics. See CRDTs: Deterministic Merge, Not Correct Merge.
  • Systems that must accept writes on both sides of a partition for business or regulatory reasons.
When it hurts
  • Anything with global invariants: uniqueness, balances, inventory, capacity, sequence numbers.
  • Data where a silently lost edit is unacceptable and no meaningful merge exists — most business records.
  • Teams without capacity to own conflict resolution as ongoing product work, since it is not a one-time configuration.
  • Single-region deployments, where it adds every cost and delivers none of the latency benefit.
Simpler alternatives

Letting a second node accept writes

Letting a second node accept writes
The only thing multi-leader replication adds is that two nodes may accept a write to the same key without first agreeing. Everything else follows from that, including the parts nobody wants.
invariant this key must preserve
west user’s write
accepted locally
west write latency
local
converged
never — partitioned
east / west now hold
Q3 Plan / Q3 Planning
Both leaders accept a write to the same keysimplified
User (east)Leader eastLeader westUser (west)PUT title: deliveredPUT titlePUT title: deliveredPUT titlereplicate: sent, never arrives — dropped in flightreplicatedropped — never arrivestitle = "Q3 Plan" → 200 (write) at t=1title = "Q3 Plan" → 200title = "Q3 Planning" → 200 (write) at t=2title = "Q3 Planning" → 200t=0time →t=2
delivereddelayed (dashed, long)duplicated (×2)dropped — stops short, never arriveswrite
Two writes were acknowledged. Neither user did anything wrong and neither node did. The conflict was created by the topology.
Multi-leader buys exactly three things: local write latency, write availability while the link is down, and independent write capacity per site. It buys nothing else — in particular it does not buy stronger consistency, and it does not buy a conflict-free system with a cleverer library. Removing the single writer removes the thing that ordered concurrent writes, so every writable field now needs a resolution rule; not choosing one means choosing last-write-wins. The mature shape is hybrid: multi-leader where convergence is enough, and a single writer for the handful of operations that need order.
simplifiedConflict resolution here is last-version-wins, which is what `propagate` implements and what most stores default to. The values, delays and steps are illustrative; what is faithful is that a partition removes delivery, and that a rule you did not choose still chooses.

What people believe, and what is true

Claim

Conflicts are rare, so we can ignore them.

Reality

Conflicts are proportional to concurrent writes to the same key, which is exactly what popular records receive. And their rate is highest during partitions, which is when the design was supposed to help.

Claim

Timestamps resolve conflicts correctly.

Reality

Wall-clock timestamps across machines are subject to skew, so last-write-wins can discard a genuinely later write. Version vectors detect concurrency correctly; timestamps only pretend to.

Claim

Multi-leader gives us high availability.

Reality

It gives *write* availability under partition, at the cost of the strongest guarantee the system can offer. Whether that is "high availability" depends entirely on whether a divergent answer counts as available.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

More than one node accepts writes for the same data. That removes the thing that ordered them, so conflicts are guaranteed and must be resolved by rule.

Practical

Before adopting it, list every writable field and state its resolution rule, then list every invariant and check whether it is verifiable at a single leader. The invariants that are not must be moved to a single owner or to consensus — that hybrid is the real design.

Advanced

Multi-leader replication is coordination-free writing, and the CALM theorem tells you exactly what that buys: computations expressible monotonically converge without coordination, and non-monotonic ones — anything involving a negation, a limit, or a uniqueness claim — provably require it. So the field-by-field audit above is not a checklist habit, it is the theorem applied. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It and CRDTs: Deterministic Merge, Not Correct Merge.

Apply it

Build it, then break it
  • 🔧 Design a resolution rule for a shopping cart that never loses an added item, and explain what it does to a removed one.
Interview questions
  • 💬 You enable multi-master across two regions. What is the first question you must answer for every writable column?
  • 💬 Why can a wall-clock timestamp not correctly resolve a conflict between two regions?
  • 💬 Which of these survive multi-leader replication: a document body, a unique username, an account balance, a comment thread? Justify each.