Operate a Production SaaS
The capstone. Fifteen questions in a fixed order, then ten things that go wrong while you are answering them. The order matters: each answer constrains the ones below it, which is why the interesting failures in real systems are almost never in the component that failed.
You own production for a multi-tenant SaaS platform serving millions of users. Frequent deployments, no planned downtime, safe database changes, fast rollback, tenant isolation, observability, on-call, backups, disaster recovery and cost awareness are all requirements — not aspirations. Work the questions in order; each answer constrains the ones below it.
The questions
Answer each one out loud before opening it. The answer is much less useful if you have not first committed to one of your own.
- 1
How does code reach production?
- 2
What exactly is the artifact?
- 3
How is configuration injected, and how are secrets delivered?
- 4
How is infrastructure created?
- 5
How is the deployment rolled out?
- 6
How are health checks defined?
- 7
What happens on shutdown?
- 8
How are migrations deployed?
- 9
How is a bad release detected?
- 10
How is it rolled back?
- 11
What alerts page on-call?
- 12
What is the runbook?
- 13
How much capacity is reserved?
- 14
How are backups restored?
- 15
How does region failover work?
Injected failures
Each of these arrives while the system is running. Every one has a response that looks right and makes things worse or leaves the mechanism intact — read the trap even when you got the cause.
Nothing. Builds are green and the application behaves normally.
The service starts and serves, but points at the staging database.
The rollout stalls. Old instances keep serving; the new ones never take traffic.
Write latency climbs sharply; timeouts spread across every endpoint that writes.
Latency rises, then errors, then autoscaling adds instances that do not help.
Clients fail to connect. The service itself is healthy and its dashboards are green.
Database load multiplies; latency spikes; some requests fail outright.
The API is healthy. Users report stale data rather than errors.
Half of traffic is failing. The surviving region is at full utilisation.
Recovery is underway and will not meet the stated objective.