Deploys Are the First Suspect
The highest prior probability for a sudden change in behaviour belongs to the thing that just changed — usually yours.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Why should a deploy be the first hypothesis in almost every sudden production incident, and how do you make it cheap to check?
Something broke at 14:32. Before anyone reads code, the team needs to know what changed near 14:32 — and to be able to undo it quickly if it was theirs.
The deploy went out at 14:05 and everything was fine for half an hour, so it cannot be the deploy. Start looking at the database.
Many deploy-caused failures are delayed by construction: a leak needs time to fill memory, a cache needs time to cool, a new query needs a specific request to arrive, a rolling deploy takes minutes to reach every instance.
- Many deploy-caused failures are delayed by construction: a leak needs time to fill memory, a cache needs time to cool, a new query needs a specific request to arrive, a rolling deploy takes minutes to reach every instance.
- Ruling out the deploy on a feeling costs the whole investigation, because everything else in the suspect list is slower to check.
- "The deploy" is not only application code — config, feature flags, infrastructure changes, dependency upgrades and schema migrations all ship and all cause this.
- Without deploy markers on the graph, "what changed at 14:32" is answered from memory in a chat channel, badly.
What is actually happening
- Production is a system in equilibrium. A step change in behaviour requires a change in inputs, and there are only two sources: a change you made, or a change the world made. You deploy several times a day; the world changes your traffic profile far less often.
- That is a base-rate argument, not a moral one. It says: check the cheap, high-probability hypothesis first, because eliminating it costs minutes and eliminating the alternatives costs hours.
- Delay between deploy and symptom is normal and has specific mechanisms: rolling replacement, cache warmth, connection pool refill, memory accumulation, a code path only some requests reach, a scheduled job that runs hourly (Rolling Deployments).
- Rollback is a diagnostic as well as a mitigation. If behaviour returns on rollback, you have a cause with far more confidence than any amount of code reading gives you.
- Some changes are not rollback-safe. A migration that dropped a column, or a message written in a new format that old consumers cannot parse, makes "just roll back" a second incident (Expand and Contract Migrations).
A base-rate argument, not an accusation
The reasoning is Bayesian and simple: a system in steady state does not spontaneously change behaviour. Something moved. The set of things that moved recently is small and knowable, and the things you moved are on it far more often than the things you did not.
The practical consequence is an ordering, not a conclusion. Check the deploy first because it is the cheapest hypothesis to check and the most likely to be right, then move down the list.
| Change type | Ships how | Typical delay to symptom | On your dashboard? |
|---|---|---|---|
| Application code | Deploy | Seconds to hours, depending on which path is affected | Usually — as a deploy marker |
| Runtime config / env vars | Deploy or restart | Immediate on the restarted instance | Often not |
| Feature flag | Flip, no deploy | Immediate, fleet-wide | Rarely, and this is the common blind spot |
| Schema migration | Deploy or manual | Immediate, or at the first query that hits it | Rarely |
| Dependency upgrade | Deploy (lockfile) | Whenever the changed behaviour is reached | Only via the deploy marker |
| Secret / certificate rotation or expiry | Scheduled or manual | At a round timestamp | Almost never — needs its own alert |
| Infrastructure change | Platform | Immediate to gradual | Only if platform events are ingested |
| Traffic mix / data volume | The world | Gradual, occasionally sudden | Yes, as request rate and row counts |
Making the check take sixty seconds
The instinct is worthless without the instrumentation. "Was there a deploy near 14:32?" should be answerable by looking at the same graph that shows the symptom — not by asking in a chat channel and waiting for someone who remembers.
The pipeline below is what makes the deploy hypothesis cheap. Every step is small, and each one closes a specific way the check currently fails.
- 1Deploy marker on the graph
Vertical line at each release, on latency and error dashboards
fails by Marks the pipeline start rather than the moment new code served traffic
- 2Version on every signal
Build id as a label on logs, metrics and spans
fails by High-cardinality labels get dropped; keep it to build id, not commit message
- 3Change feed beyond deploys
Flags, config, migrations and secret rotations on the same timeline
fails by Flag systems often have no audit trail unless configured
- 4Group errors by version
Answers "is the new build failing and the old one not" during a rolling deploy
fails by Only works while both versions are running — the window is minutes
- 5Capture before rollback
One trace, one profile or one heap snapshot from an affected instance
fails by Skipped under pressure, and then the cause is unfindable afterwards
- 6Roll back
Restores service and tests the hypothesis at the same time
fails by Unsafe if the release included a non-backward-compatible migration
- 7Confirm and record
Note the recovery time and whether it matches the rollback
fails by Recovery coinciding with a traffic trough gets misread as a fix
When rollback is not available
Rollback is the default because it is fast and reversible, but a release that changed data or wire formats can make it the wrong move. Knowing in advance which category a release is in is part of shipping it.
What did this release change besides code?
when No schema change, no new message format, no data written in a new shape.
cost Effectively free. Roll back first, diagnose after.
when New nullable column or new table; old code ignores it.
cost Still safe. This is why expand-and-contract exists (Expand and Contract Migrations).
when A column or table was dropped, or a constraint tightened.
cost Rollback breaks the old code. Fix forward, or restore from backup — both slower and riskier.
when Records or messages produced by the new version cannot be read by the old one.
cost Rollback strands that data. Needs a compatibility shim or a forward fix (Commands vs Events).
when The behaviour is gated and the gate is runtime-controlled.
cost Flip the flag: faster than a rollback and far narrower. This is the argument for shipping dark (Feature Flags: Rollout, Kill Switches and Debt).
when Webhooks sent, payments captured, emails delivered.
cost Nothing to roll back. Reconciliation and compensation, not reversal (Outbound Webhooks).
How to build it
Most important first.
- Put deploy, config-change, flag-flip and migration markers on every latency and error dashboard. This one change pays for itself in the first incident.
- Tag every log line, metric and trace with the build version, so "which version is erroring" is a group-by rather than an investigation (Structured Logging).
- Make rollback fast and boring — a one-command operation rehearsed outside incidents. A rollback that takes twenty minutes will not be used when it matters.
- Keep migrations backward compatible so that rolling back application code never requires rolling back schema: expand, deploy, migrate data, contract (Schema Migrations from the Application Side).
- Deploy in a way that limits blast radius — canary or a small first wave — so the comparison between old and new is available while both are running (Canary Deployments).
- Separate the release from the deploy: ship code dark behind a flag so the risky change can be reverted without redeploying (Feature Flags: Rollout, Kill Switches and Debt).
What can go wrong
- Rolling back to a version that is also broken, because the real change was two deploys ago and only became visible now.
- A rollback that leaves the database on a newer schema the old code cannot read.
- Blaming the deploy and reverting it while the true cause was a simultaneous dependency change, then repeating the deploy later and re-triggering the incident.
- A flag flip that is not recorded anywhere, so the "nothing changed" claim is technically believed and factually wrong.
- Deploy markers that show the *start* of a rolling deploy only, so the actual moment new code began serving is still unknown.
- During a rolling deploy two application versions serve concurrently against one schema — a race between versions that only exists in the deploy window (Expand and Contract Migrations).
- A migration running while old instances are still writing can race with those writes.
- Rollback restores old code, including any vulnerability fixed in the reverted release. Track that explicitly rather than discovering it later (Dependency Security).
- Dependency upgrades ship in deploys; a compromised or malicious package version becomes a deploy-shaped incident with a security cause (Dependency Security).
- Rotating a secret is a change with a deploy-like blast radius and belongs on the same timeline (Secrets Are Not Configuration).
- Deploy pipelines have production credentials. Who can trigger a deploy is an authorization question, and an incident is not the moment to loosen it.
- "It ran fine for 30 minutes, so it is not the deploy." Delayed onset is the norm for leaks, cache effects and rarely-taken code paths.
- "Nothing changed." Config, flags, secrets, dependency versions, infrastructure and data volume all change without a commit.
- "Rollback fixed it, so the code was the bug." Rollback also reverts config bundled with the release, and it clears process state. It is strong evidence, not proof.
- "We deploy continuously, so deploys cannot be the cause." Frequency changes the size of each suspect, not the category.
- "It is the deploy" asserted without a marker to point at. Prior probability justifies checking first, not concluding first.
Operating it
- Deploy markers overlaid on latency, error rate and saturation graphs — the highest value-per-line-of-config in observability.
- Version label on error rate: a broken release shows as errors concentrated in one build id while the previous build stays clean.
- During rolling deploys, per-instance metrics grouped by version. A half-deployed fleet is a natural A/B test that expires when the deploy finishes.
- A change log that includes config and flags, not just commits.
- Schema version per instance during migration windows.
- More frequent deploys make each one a smaller suspect and the correlation sharper — the argument for continuous delivery is partly a debugging argument.
- Larger fleets lengthen rolling deploys, widening the window between "deployed" and "fully serving" and making the delay between change and symptom longer.
- With many services deploying independently, "what changed" becomes a cross-service question and needs a shared change timeline.
- Rolling back destroys the state that would let you diagnose from the running system. Capture a trace, a profile or a dump first when the incident allows it.
- Fast rollback encourages reverting without understanding, so causes go unfound and recur. The discipline is: revert to restore service, then find the cause with the artefacts you captured.
- Canary deploys add pipeline complexity and slow delivery slightly. They buy a comparison you cannot otherwise get.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALA base-rate argument that holds anywhere changes are shipped regularly.
- SCALE-SPECIFICWith a handful of deploys a month, a deploy is a very sharp suspect and the correlation is nearly conclusive; with hundreds a day it is still the first place to look, but the window narrows and per-version tagging becomes essential rather than nice.
- CLOUD-SPECIFICHow much of "what changed" you can see depends on the platform: a managed platform may replace instances, rotate certificates or change defaults on its own schedule, so a change you did not make can still be a deploy-shaped event you must be able to look up.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — change management, progressive delivery and the incident review that turns "we rolled back" into a known cause.