AdvancedCI/CD & Deployment← All practice

The Deploy That Removed Its Own Capacity

The report, in their words

A routine deploy started at 16:05. By 16:15 the service is serving 502s for about a third of requests, the fleet is down from 12 instances to 4, and the deployment is still "in progress". A rollback was requested at 16:16 and it is also stuck. Latency on the four surviving instances is eight seconds.

The infrastructure under investigation

Exposure is shown, because half of these failures are a reachability question.

Where the requests are going, and where the capacity went.PROVIDER-NEUTRAL
Userspublic
Region
VPC
Load balancerpublic— returns 502 when it has no healthy target
4 surviving instancesprivate
8 replaced instances — deregisteredprivate
Database — 38% CPUprivate
Payment provider status endpointpublic— answers in 1.9–2.6s, and now sits inside /health
Users→Load balancer· 502 for ~1 in 3crosses boundary
Load balancer→4 surviving instances· all traffic
Load balancer→8 replaced instances — deregistered· removed: Target.Timeout
4 surviving instances→Database — 38% CPU· healthy
4 surviving instances→Payment provider status endpoint· /health calls thiscrosses boundary
8 replaced instances — deregistered→Payment provider status endpoint· /health calls thiscrosses boundary

Pull evidence

One item at a time, and nothing here tells you which one matters. Deciding what is worth looking at is most of the diagnosis.

Load balancer target health
What the release actually changed
The payment provider status endpoint
Application logs on an instance the balancer calls unhealthy
Deployment controller state
Latency and CPU on the four survivors
The database
New instance registration timing
Where the 502s are generated

0 of 9 inspected. You are not required to open all of them — a real investigation is judged on how few you needed.

What is your diagnosis?

Commit to one. Nothing below is shown until you do.

Guessing wrong and being told exactly why is the point of this page. Reading the answer first is not practice.