Physics sets a floor on coordination

Multi-Region Systems

7 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.

A Region Boundary Is a Consistency Decision▶ lab

Crossing a region boundary multiplies the cost of agreement by roughly a hundred. Nothing else about the system changes — the same consistency models are available, the same protocols work. What changes is that every one of them now has a price you can feel, and a failure domain you cannot hide.

Q · What actually changes when my system spans more than one region?

The One Number You Cannot Optimise▶ lab

London to New York is about 5,600 km. Light in fibre covers that in ~28 ms one way, ~56 ms there and back, before a single packet is routed, queued or processed. No framework, no protocol and no amount of tuning moves that number. It is the floor under every cross-region design decision.

Q · How much of my cross-region latency is a physical constant, and how much can I actually engineer away?

Three Ways to Accept a Write in More Than One Place▶ lab

Single-writer region, multi-writer, or partitioned ownership where each region is authoritative for its own slice of the keys. The first is simple and slow for distant users; the second is fast and makes every conflict real; the third is the one people forget, and is very often the right answer.

Q · Which region is allowed to accept a write, and what does each answer cost me?

Active-Passive: Simple to Reason About, Rarely Tested▶ lab

One region serves; another stands ready. The model is easy to explain and easy to be correct about — there is exactly one writer, so there are no conflicts, ever. Its two weaknesses are not conceptual: the failover is slow and multi-step, and it is almost never exercised, which means its probability of working the first time is far below what the runbook implies.

Q · A standby region exists. What actually happens when I have to use it, and how long has it been since anyone checked?

Active-Active: Every Conflict Scenario Becomes Real▶ lab

Both regions serve, both accept writes, there is no failover step to get wrong. What you buy is local latency and continuously-proven capacity. What you pay is that every concurrent-write scenario you could previously wave away is now a thing that happens, in production, at a rate you do not control — and that a bad write propagates to both regions at wire speed.

Q · Both regions are serving live traffic. Which problems did that solve, and which did it create?

EU and US Are Partitioned. Can Both Keep Accepting Writes?▶ lab

The link between two regions fails. Both are healthy, both are serving users, neither can reach the other, and neither can tell whether the other is dead or merely unreachable. Whether both may keep accepting writes has an answer — but it is a property of the invariant, not of your preference, and it is different for different data in the same system.

Q · The regions cannot see each other and both are up. Which of them is allowed to say yes?

When the Data Is Not Allowed to Leave▶ lab

Some data must legally remain inside a jurisdiction. That is not a deployment preference — it is a hard constraint that propagates upward into your partitioning key, your replication topology, your quorum placement, your indexes, your caches, your backups and your logs. It can forbid outright the design you would otherwise have chosen, and it is best discovered before you build it.

Q · A class of my data may not leave its jurisdiction. What does that forbid, and what does it force?