Production practice

A team has one CI job that runs for 38 minutes and fails about a third of the time for reasons unrelated to the change. People merge without waiting. Redesign the pipeline. You are told the goal is "make CI faster".

A service is deployed by pulling the main branch onto each server and running the start script. You are asked to produce a proper build artifact instead. The build currently takes two minutes and "works fine".

A newly containerized service starts and exits within a second. The platform shows it restarting repeatedly. `docker logs` prints nothing. The developer says it works when they run it locally with `npm start`.

A rate-limit value lives in a central configuration service polled by every instance every 30 seconds. An engineer edited it through the web UI and added a trailing space. The value parsed to zero, every request was rejected fleet-wide within half a minute, and the UI shows no record of the previous value. Fix the system, not the typo.

Deployment is an SSH loop: pull the main branch on each of four servers, install dependencies, restart the process. It takes eleven minutes, occasionally drops requests, and last month one server ended up on a different commit than the others for two days. Design something better without introducing an orchestrator.

A rolling deployment has replaced half the instances. Error rate is elevated on both old and new instances, not just the new ones. The rollout has stalled because health checks on the new pods are failing intermittently. Someone suggests completing the rollout so at least there is only one version.

A change is running on 5% of traffic. After eight minutes: error rate 0.19% on the canary versus 0.21% on the baseline, p50 latency slightly better, p99 marginally worse. CPU is lower. The deploy dashboard is green and the rollout is waiting for a decision.

An infrastructure change adds a subnet, retags some resources, and renames a database module from `db` to `postgres_primary` for clarity. The plan output ends with "Plan: 4 to add, 2 to change, 1 to destroy". The author says the destroy is an old security group and the change is safe to apply during business hours.

A service has one health endpoint used for both liveness and readiness. It checks the database, the cache and two downstream APIs, and returns 503 if any is unavailable. During a brief database slowdown, every instance was restarted, and the service was unavailable for far longer than the database was.

A release adds a NOT NULL column with a default to a 90-million-row table and deploys code that writes it. The migration is running as part of the deploy. Twenty minutes in, write latency on that table has climbed, replication lag is at four minutes, and half the fleet is on the new version. Someone asks whether to cancel the migration.

A rolling deploy produces a reliable burst of 502s. The entrypoint is `sh -c "node server.js"`, the termination grace period is 30 seconds, and the application has no signal handler. The team's proposed fix is to raise the grace period to 120 seconds.

After a deploy, a Deployment reports 3 of 8 replicas available. One pod is Pending, one is in CrashLoopBackOff, one is Running but not Ready, and one has restarted eleven times with the last state showing OOMKilled. The team's instinct is to delete the pods and let them reschedule.

Checkout success rate has fallen from 99.4% to 71%. The on-call engineer follows the runbook: "1. Check the dashboard. 2. If it looks bad, restart the pods. 3. If that does not work, escalate." They restart the pods. Success rate recovers to 88% for six minutes, then falls to 64%. Two other engineers have joined and are typing in the same terminal session.

A service runs 12 instances across three zones at 80% CPU during peak, autoscaling on CPU with a maximum of 16. Leadership wants to cut infrastructure spend and points at the idle capacity outside peak hours. Produce a capacity plan.

An API scales on CPU at a 70% target. During the morning peak, p99 latency goes from 120ms to 4 seconds, database connections are exhausted, and the autoscaler adds nothing because CPU never exceeds 45%. An engineer lowers the CPU target to 40%. The fleet doubles, and the outage gets worse.

A team deploys every two weeks. Each release contains around 60 merged pull requests, several product changes and usually a schema change. During the last incident it took 25 minutes to establish which version was running in production, and rollback was rejected because it would have reverted a feature announced that morning.

A pipeline builds images tagged `service:latest`; the deployment references that tag. The registry accepts pushes from three service accounts and four humans. A scanner reports 340 findings, all acknowledged. During a routine audit, someone finds that the running digest does not match any pipeline run from the last month.

A service runs active-passive across two regions. The primary region is degraded — elevated errors, not a clean failure. The passive region is provisioned at 40% of the primary's capacity because "it is only a standby". The database replicates asynchronously with lag typically under two seconds. You have to decide whether to fail over.

At 14:00 a maintenance script ran a `DELETE` without the intended `WHERE` clause against a 1.2 TB production database. It was noticed at 14:20. Nightly `pg_dump` backups have succeeded for 400 consecutive days, into a bucket in the same cloud account. The stated RTO is one hour. Nobody has restored this database since it was 40 GB.

A platform team of six owns the deployment system every service is required to use. Onboarding a new service takes three weeks and two tickets. Two product teams have quietly built their own pipelines. The platform team is asking how to enforce compliance, and has proposed blocking non-platform deployments at the cloud account level.

An organisation runs 400 services owned by 60 teams across four regions. Releases are currently coordinated in a weekly meeting with a spreadsheet. A single service's bad release last month caused a cross-service outage because a change to a shared message format was deployed before its consumers could handle it. Design the release system.

Checkout succeeds for most users and fails for some, starting at 09:03 and continuing. No deploy for six days. Error rate is 6% and rising slowly. The payment provider's status page is green. Latency to the database is normal. One engineer is convinced it is the network, one wants to restart the payment worker fleet, and one is reading application logs. There are no deploy or config annotations on any dashboard.