EU and US Are Partitioned. Can Both Keep Accepting Writes?

At 14:02 the transatlantic link degrades and then stops. Frankfurt continues serving European users at normal latency. Virginia continues serving American users at normal latency. Both regions’ dashboards are green for everything local. Both show the other region as unreachable. Both are correct.

Note what the operator sees: not an outage. Traffic is flowing, error rates are normal in each region, and the only anomalies are a replication lag climbing without bound and a peer health check failing. In a great many organisations this state persists for tens of minutes before anyone treats it as an incident, because none of the alerts that fire are the ones people watch.

The decision that must be made in those minutes is not "how do we restore the link" — that is somebody else’s job and it will take as long as it takes. It is: for each class of data, do we keep accepting writes on both sides? And that question has a different answer for the audit log, the user profile table, the username registry and the ticket inventory, in the same system, at the same moment.

Take five invariants from an ordinary system and ask the question of each. The answers differ, and the differences are the whole lesson.

The pattern that emerges: an invariant that is local to one record survives both regions writing, provided each record has one owner. An invariant that is a global aggregate does not, unless it can be split into per-region shares in advance. An invariant that is existence over an unbounded namespace — uniqueness — cannot be split at all, and always requires either a single owner or after-the-fact repair.

Look at what the last two rows did. They did not weaken the invariant and they did not coordinate. They pre-partitioned the resource so that each region holds an exclusive share it can enforce alone. This is the single most useful technique in the module, and it generalises: escrow for money, allocation for inventory, per-region id ranges for sequence numbers, per-region quotas for rate limits.

The price is always the same and it is worth naming precisely: you convert "sometimes wrong" into "sometimes unnecessarily refused". Overselling becomes impossible; underselling becomes possible. Overdrafts become impossible; false declines become possible. For most businesses that is an excellent trade, because a refusal is visible, explicable and recoverable while a violated invariant is silent and expensive — but it is a trade, and the product owner should make it, not the database.

When the resource cannot be split — a globally unique name over an unbounded namespace, a strict global sequence, a "at most one active session" rule — there is no escape. One side must stop, or you must repair afterwards. [[distributed-uniqueness]] is the general treatment, and [[coordination-avoidance]] is the discipline of noticing which of your invariants are actually splittable, which is more of them than people assume.

A two-region partition is symmetric, and symmetry is the problem: both sides have equal claim, so any rule of the form "the survivor keeps writing" is unimplementable, because both believe they are the survivor.

A third location breaks the symmetry cheaply. Put a witness — a small, stateless-ish voter holding no user data — in a third region. Now a partition has an asymmetric outcome: whichever side can still reach the witness holds the majority and keeps writing; the other side steps down. This is [[quorums]] applied to geography, and it converts an undecidable situation into a decidable one for the cost of one small deployment.

The honest costs. The witness is now on the availability path for promotion decisions, so its own outage matters. It adds an RTT to any decision that consults it, which is why it is used for *leadership* decisions rather than per-write ones. And it must be in a genuinely independent failure domain — a witness in the same cloud provider’s shared control plane is less independent than the diagram suggests, and a partition that takes out the provider’s global services takes the witness with it.

Three full regions do the same job while also giving you capacity, which is why the standard advice for anything needing consensus across geography is three regions rather than two. Two regions plus a witness is the budget version, and a legitimate one.

People plan for the partition and forget the merge. But nothing goes wrong *during* a partition — each side is internally consistent and locally correct the entire time. Everything goes wrong when the link returns and two internally-consistent histories have to become one.

The reconnection is also an operational event in its own right: hours of queued replication arrive at once, saturating the link, the apply path and often the database. A partition that lasted forty minutes can produce an hour of degraded performance after it ends, and teams frequently misread that as a second incident. Rate-limit the drain.

Then the actual merge. Duplicate unique values must be resolved and somebody is getting renamed. Escrow shares must be rebalanced, and any region that exhausted its share while another sat idle needs to be replenished. Counters must be summed rather than compared. Ordering-dependent state machines need to be replayed by causality rather than by timestamp. And every one of these is an *application* decision — [[application-merge]] — because the database cannot know that the correct resolution for two conflicting shipping addresses is to ask the customer.

Which yields the most practical instruction in this lesson: write the merge procedure before the partition, not after. For each dataset that both regions may write, there should be a named, tested answer to "what happens when these two histories meet?" A system where that answer does not exist has not chosen availability under partition — it has chosen to discover its merge semantics during an incident.