Reliability & Disaster Recovery

Active-Active

Both regions serve traffic at the same time. It removes failover from the critical path and replaces it with a permanent distributed-data problem: conflicts, ordering, routing and consistency, all live, all the time.

The question this answers

Infrastructure question

What does it take for two regions to serve traffic simultaneously, and what problem do you take on permanently in exchange?

Application requirement

Users on both continents need low-latency writes, and a regional failure must be a capacity event rather than an outage. The organization has the engineering capacity to operate a distributed data tier and has said so with a straight face.

What it provides

Continuous service from multiple regions with no failover step, local write latency for every user, and capacity that degrades rather than disappears when a region is lost.

Application RequirementInfrastructure RequirementComputeNetworkStorageIdentityDeploymentScalingReliabilityObservabilitySecurityCostTrade-offs

The shape, and the honest warning

Active-active means every region accepts reads and writes at all times. The failover problem disappears because there is nothing to fail over to — a lost region is a capacity reduction and a routing change, not a promotion. That is genuinely the best availability posture available, and it is why the largest consumer services run this way.

It is also, by a wide margin, the hardest thing in this domain. The moment two regions accept writes, you own a distributed data problem permanently rather than only during an incident. Two users update the same record in different regions within the replication window. Which write wins? What happens to the other one? Does the user who lost find out? Is there an invariant — an account balance, a seat allocation, a unique username — that cannot survive being decided in two places at once? These questions have no default answers, and the framework will not solve them for you.

Be honest about the prerequisites. Active-active suits workloads that are naturally partitionable (each user's data belongs to one region), or naturally commutative (appends, counters, events), or genuinely tolerant of stale reads. It suits inventory, banking balances and anything with a global uniqueness constraint very badly. If your first instinct is "we will handle conflicts in the application", write down three concrete conflict cases and their resolutions before committing — that exercise ends most active-active projects, which is a good outcome (Active-Passive Failover is usually what was actually wanted).

Both regions live. The bidirectional replication edge is where all the difficulty lives.PROVIDER-NEUTRAL
Latency-based global routingpublic— sends each user to the nearest healthy region; withdraws a failed one
Region EU — serving
Load balancerpublic
app tier (multi-zone)private— sized to absorb US traffic on failure
Data tier (writable)private
Region US — serving
Load balancerpublic
app tier (multi-zone)private— sized to absorb EU traffic on failure
Data tier (writable)private
Global identity / configinternal— anything genuinely global is a shared failure domain across both regions
Latency-based global routingLoad balancer· EU users
Latency-based global routingLoad balancer· US users
Load balancerapp tier (multi-zone)
Load balancerapp tier (multi-zone)
app tier (multi-zone)Data tier (writable)· local reads + writes
app tier (multi-zone)Data tier (writable)· local reads + writes
Data tier (writable)Data tier (writable)· bidirectional replicationcrosses boundary
Data tier (writable)Data tier (writable)· bidirectional replicationcrosses boundary
app tier (multi-zone)Global identity / config
app tier (multi-zone)Global identity / config

The conflict, concretely

Abstract talk about conflict resolution hides how ordinary the failure is. Below is a ninety-second window with a transatlantic replication lag of about 200 ms under load. Two writes to the same record, neither region aware of the other, both acknowledged to their user as successful. The system must then choose, and whatever it chooses, one user was told something untrue.

Last-write-wins is the default in many systems and is the most dangerous default there is, because it is silent. It relies on clocks that are not synchronized across regions, it discards data without telling anyone, and it produces the class of bug where a customer swears they saved something and the record says otherwise — unreproducible, unloggable, and correct according to the configuration.

The workable strategies each have a real cost. Partition by key: every entity has a home region, and writes are routed there — this is the strategy that actually works, and it is really active-active at the *fleet* level with single-writer semantics per entity. Commutative structures: counters, sets and append-only logs that merge without ambiguity, which requires modelling your domain that way from the start. Application-level merge: explicit rules per entity type, correct and expensive to write and maintain. Global consensus: a database that orders writes globally, correct and slow, since it pays a cross-region round trip on the write path.

T+0ms    EU   user A: UPDATE profile SET phone = '+49...' WHERE id = 42
T+5ms    EU   commit ok -> 200 OK returned to user A
T+40ms   US   user B: UPDATE profile SET phone = '+1...'  WHERE id = 42
T+45ms   US   commit ok -> 200 OK returned to user B
T+205ms  EU   receives US write (ts=40)  ... conflict with local (ts=0)
T+245ms  US   receives EU write (ts=0)   ... conflict with local (ts=40)

last-write-wins by timestamp:
  both regions converge on '+1...'  -> user A's change is gone.
  No error was raised. No log line says a write was discarded.
  User A saw 200 OK and will report a bug you cannot reproduce.

and if the two clocks disagree by 300ms, the regions may briefly
converge on DIFFERENT values -- which is worse than losing a write,
because now the two regions disagree about the present.
Two regions, one record, 200 ms of lag. ILLUSTRATIVE.

Strategies, and what each one demands of you

Choosing a strategy is choosing which cost to pay: latency, developer effort, domain-model constraints, or silent data loss. The matrix names the trade for each. Note that the two strategies that are actually safe — partitioning by key and global consensus — are also the two that constrain the system most.

Routing deserves as much attention as data. Latency-based routing is standard, but a user who moves between regions — travelling, a mobile network, a changed resolver — can read their own write from a replica that has not received it yet, and see their change vanish. Session affinity to a region, or read-your-writes tracking with a version token, is usually required. This is not a data-tier problem you can delegate; it shows up as "the app lost my edit" in support tickets.

Finally, capacity. Active-active only survives a region failure if the survivor can carry everything. Two regions each running at 70% utilization means losing one leaves 140% of demand on a region built for 100%. Real active-active runs each region under 50%, which means paying for more than twice the capacity you need on an ordinary day — the true cost of the design, and the one that is missing from most estimates.

StrategyHow it resolvesCostWhen it is rightWhat it forbids
Last-write-winsHighest timestamp winsSilent data loss; depends on clock syncEphemeral, low-value data — a cached preferenceAnything a user would notice losing
Partition by keyEach entity has a home region; writes route thereCross-region write latency for users away from homeThe strategy that works. Naturally partitionable data: tenants, accounts, usersEntities that many regions must write concurrently
Commutative types (CRDTs)Merges are order-independent by constructionThe domain must be modelled this way from the startCounters, sets, presence, collaborative textInvariants like "balance must not go negative"
Application-level mergeExplicit rules per entity typeExpensive to write, test and maintain foreverA small number of high-value entity typesBroad application across a large schema
Global consensus databaseWrites are globally ordered before acknowledgementA cross-region round trip on the write pathStrong invariants that must hold globallyLow-latency writes — geometry does not negotiate
Conflict strategies: what each one costs and what it forbids

Key points

  • Active-active removes failover from the critical path and replaces it with a permanent distributed-data problem.
  • Two regions accepting writes means conflicts are a steady-state condition, not an incident.
  • Last-write-wins is silent data loss that depends on unsynchronized clocks. It is the default and it is almost never the right answer.
  • Partitioning by key — each entity has a home region — is the strategy that actually works, and it is single-writer semantics wearing an active-active hat.
  • Read-your-writes across regions requires routing affinity or version tracking, or users will watch their own edits disappear.
  • Each region must run under 50% utilization to absorb the other, so the real cost is more than double.
  • If you cannot write down three concrete conflict cases and their resolutions, you want active-passive.

The loop, answered

Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.

How it works
  • Latency-based global routing sends each client to the nearest healthy region and withdraws a failed one automatically.
  • Each region runs a complete, independently writable stack with its own data tier.
  • The data tiers replicate bidirectionally; ordering across regions is not guaranteed and must be resolved by a stated strategy.
  • Conflict resolution runs either in the data tier (timestamps, vector clocks, CRDT merges) or in the application (explicit rules per entity).
  • Region-affinity or version tokens preserve read-your-writes for a user whose requests land in different regions.
  • Losing a region is handled entirely by routing: traffic shifts, capacity drops, and no promotion occurs.
What you still own
  • Own the conflict strategy per entity type and document it. Anything without a stated rule is being resolved by a default you did not choose.
  • Instrument conflicts: count them, log them with both versions, and alert on rate. A conflict rate you cannot see is data loss you cannot see.
  • Maintain capacity headroom for full absorption in every region, and verify it by shifting all traffic to one region during a drill.
  • Test region loss by withdrawing a region from routing on purpose. In active-active this is a routing change, which makes it far safer to rehearse than a promotion.
  • Keep clock synchronization monitored if any resolution depends on timestamps — clock skew is a silent correctness bug, not an operational nuisance.
How it fails
  • Silent conflict loss: two writes, one survives, nobody is told, and the bug report is unreproducible.
  • Divergence: regions converge on different values because clocks disagreed or a merge rule was not commutative, and the system now has two truths.
  • Broken read-your-writes: a user's request lands in the other region and their own change is missing.
  • Invariant violation: the same seat, username or balance is allocated in both regions because the constraint could only ever be enforced in one place.
  • Replication backlog: a partition heals and a large backlog of conflicting writes arrives at once, resolving en masse with no human in the loop.
  • Capacity collapse on region loss: the survivor was at 70% and receives 140% of demand.
  • A "global" component — identity, configuration, a licence server — turns out to be single-region and takes both regions with it.
How it scales
  • Read capacity scales cleanly with regions; write capacity does not, unless writes are partitioned by key.
  • Replication traffic and conflict probability both grow with write volume, so the hard part gets harder as you succeed.
  • Each region added multiplies the number of replication relationships and the number of ways ordering can surprise you.
  • Absorption headroom must scale with every region: N regions each need capacity for their share plus a failed peer's share.
Security
  • Every region is a full production environment with full data access. The number of places holding production data has multiplied, and each needs identical controls (Least Privilege in Infrastructure).
  • Bidirectional replication means a compromise or a bad write in one region propagates to the others — replication is a blast-radius multiplier (Failure Domains).
  • Data residency and active-active are frequently incompatible: if EU personal data may not leave the EU, it cannot be replicated to a US writer.
  • Conflict resolution logs contain both versions of records and therefore contain sensitive data; they need the same protection as the database (Audit Trails).
Cost shape
  • More than double: each region needs absorption headroom, so total provisioned capacity is roughly 2.2–2.5× a single-region design.
  • Bidirectional replication transfer is continuous and proportional to write volume, in both directions.
  • Engineering cost dominates. Conflict handling, routing affinity and testing are ongoing work, not a one-time build.
  • The one saving: no idle standby. Every provisioned instance is serving, which makes the utilization story better than active-passive even as the total is higher.
What to watch
  • Conflict rate and resolution outcomes per entity type — the highest-value active-active metric, and the one most often absent.
  • Replication lag in both directions, since the two are not symmetric and the slower one bounds your consistency.
  • Per-region capacity headroom against the absorb-a-peer threshold.
  • Read-your-writes violations, detectable by tracking a version token per session and counting stale reads.
  • Clock skew between regions, if any resolution depends on time.
  • The signal that lies: global success rate. Both regions returning 200 OK is exactly what a silent conflict looks like.
Simpler alternatives
  • Active-passive. Almost always what the team actually wants: regional survivability, one writer, no conflicts. Start here and prove it is insufficient (Active-Passive Failover).
  • Read-local, write-global: serve reads from every region, route all writes to one. Most of the latency benefit, none of the conflict problem, and the write path is a single point of failure you can name.
  • Regional partitioning with no cross-region replication: EU users live in the EU, US users in the US, neither is a failover for the other. Simple, compliant, and no conflicts by construction.
  • A CDN plus edge caching for read latency, keeping a single-region origin.
  • A managed globally-distributed database that implements the hard parts, accepting its consistency model, its pricing and its lock-in.
What adopting this costs
  • Buys the best availability posture available; costs a permanent distributed-data problem and more than double the capacity.
  • Local writes are fast and can conflict; globally ordered writes cannot conflict and are slow.
  • No failover to rehearse, in exchange for correctness questions that are live every second of every day.
  • Conflict strategies that are safe (partition by key, global consensus) are the ones that constrain the domain model most.
  • Every region is another full production environment to secure, patch, monitor and keep identical.

What people believe, and what is true

Claim

A multi-writer database makes active-active easy.

Reality

It makes replication easy. Deciding which write wins when two users edit the same record remains a domain question only your application can answer.

Claim

Conflicts are rare, so we can handle them later.

Reality

Conflict probability scales with write volume and lag. "Later" is a busy Tuesday, and by then the resolution rule has been running silently for months.

Claim

Active-active means we always have full capacity.

Reality

Only if each region can absorb the others. Two regions at 70% each means losing one delivers 140% of demand to a region built for 100%.

Apply it