ExpertReliability & Disaster Recovery← All practice

One Zone Died and the Multi-Zone Application Went Down Anyway

The report, in their words

The provider reports degraded power and network in zone A of our region. Our application runs in three zones with instances in each, exactly as designed. Two thirds of the fleet is healthy and serving. And the site is down: every request that touches the database fails, and every outbound call to the mail and payment providers times out.

The infrastructure under investigation

Exposure is shown, because half of these failures are a reachability question.

Three zones of application tier in front of two single-zone dependencies.PROVIDER-NEUTRAL
Region
Zone A — degraded
App instancesprivate
NAT gateway (single)public
Database primaryprivate
Zone B
App instancesprivate
Read replica (read-only)private
Zone C
App instancesprivate
Load balancer (all zones)public— deregistered zone A within 40s
Payment + mail providerspublic
Load balancer (all zones)App instances· removed
Load balancer (all zones)App instances
Load balancer (all zones)App instances
App instancesDatabase primary· writes — unreachable
App instancesDatabase primary· writes — unreachable
Database primaryRead replica (read-only)· async replication
App instancesNAT gateway (single)· all egress
App instancesNAT gateway (single)· all egress
NAT gateway (single)Payment + mail providerscrosses boundary

Pull evidence

One item at a time, and nothing here tells you which one matters. Deciding what is worth looking at is most of the diagnosis.

The database
Egress from the surviving zones
Application instances in zones B and C
Why there is only one NAT gateway
Load balancer behaviour
Route tables for the three private subnets
Replica lag at the moment of failure
The last failover exercise
The database endpoint

0 of 9 inspected. You are not required to open all of them — a real investigation is judged on how few you needed.

What is your diagnosis?

Commit to one. Nothing below is shown until you do.

Guessing wrong and being told exactly why is the point of this page. Reading the answer first is not practice.