Infrastructure Diagnosis Labs

The checkout API started returning 500s at 09:14. Every request logs connect ETIMEDOUT 10.0.3.41:5432 after exactly 30 seconds. Nothing was deployed this morning — the last release was Thursday. The database console shows the instance available, CPU near idle, and zero connections from the API.

Between 12:00 and 12:40 the checkout service fails about one payment in six with ETIMEDOUT calling the payment provider. Every dashboard is green: instances healthy, all load balancer targets healthy, CPU 35%, database fine, error budget barely moving. The provider says the requests they receive all succeed — they just receive fewer than we say we sent.

The nightly export job failed at 02:10 with AccessDenied: not authorized to perform s3:PutObject on arn:aws:s3:::exports-prod/nightly/2026-08-24.csv. The on-call engineer attached a policy granting s3:* on that bucket to the job's role and retried. Same denial. It ran successfully every night for eight months and nothing in the job changed.

A routine deploy started at 16:05. By 16:15 the service is serving 502s for about a third of requests, the fleet is down from 12 instances to 4, and the deployment is still "in progress". A rollback was requested at 16:16 and it is also stuck. Latency on the four surviving instances is eight seconds.

A service was redeployed at 11:20 and none of its pods stay up: 0/1 Ready, status CrashLoopBackOff, restart count climbing. The image is byte-for-byte the one that has been running since yesterday. The only thing in the merge was a ConfigMap update.

Roughly half of a service's pods are Running and the other half are stuck in ImagePullBackOff. The deployment went out four hours ago and was fine. This morning the cluster gained a second node pool for capacity, and every pod that lands on the new nodes fails to pull the image.

Scaling a Deployment from 6 to 10 replicas has left four pods Pending for twenty minutes. The cluster dashboard reads 38% CPU and 41% memory used across fourteen nodes, so there is obviously room. Two more nodes were added an hour ago and the pods are still Pending.

One pod of the report-rendering service restarts most nights between 03:00 and 03:20. No alert has ever fired, because the Deployment always has other replicas ready and the restart is over in ten seconds. Someone finally looked at the pod: Restart Count: 46, last state Terminated, Exit Code: 137.

A routine terraform plan on the platform repository — no code change, just a rebase before some unrelated work — reports Plan: 1 to add, 0 to change, 0 to destroy. The module declares three application instances. The console shows two. Nobody remembers deleting anything and the service has been serving traffic normally all week.

The provider reports degraded power and network in zone A of our region. Our application runs in three zones with instances in each, exactly as designed. Two thirds of the fleet is healthy and serving. And the site is down: every request that touches the database fails, and every outbound call to the mail and payment providers times out.

A bad migration at 14:05 dropped a column and rewrote 40 million rows in the orders table. The decision to restore from last night's backup was made at 14:20. It is now 17:30, the restore is still running, the estimated finish keeps moving, and somebody has finally asked how long this is supposed to take. The runbook says "RTO: 1 hour".

Finance flags that last month's infrastructure spend is roughly three times the month before. Engineering shipped no new services, instance count is unchanged, and traffic is up about 15%. The leading theory in the thread is that the provider raised prices.

At 00:00 UTC the mobile app stopped working for every user: SSL handshake failed. The web app shows a browser interstitial about an invalid certificate. Every dashboard is green — instances healthy, all load balancer targets healthy, application error rate zero, latency normal. The first thing the on-call engineer checked, an internal monitoring endpoint, returned 200 OK.

Every weekday at 09:00 the service sheds requests for three to four minutes: latency climbs to six seconds, a few percent of requests time out, then everything recovers on its own. The autoscaling policy fires correctly every morning and adds the right number of instances. By the time they serve traffic the peak is over. The team has lowered the threshold twice and nothing changed.