Deployment-Centric Debugging
The highest-signal habit in the domain: when an incident begins, ask what recently changed before asking what is wrong.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why is "what did we ship?" a better first question than "what is broken?"
A system that has been stable for weeks starts failing. The code that is failing did not write itself last night. Something moved.
Treat the incident as a bug hunt. Read the stack trace, find the offending code path, reason about how it could produce this behaviour, then work out how it got into that state.
You spend the first half hour understanding code that has been correct in production for months, because it looked like the obvious place to start.
- You spend the first half hour understanding code that has been correct in production for months, because it looked like the obvious place to start.
- The stack trace points at the place that failed, which in a system with retries, pools and timeouts is often several layers away from the thing that moved.
- Bug-hunting has no time bound and produces no mitigation. Even a correct diagnosis at minute fifty is fifty minutes of impact.
- When you eventually discover a config change at 13:58 that nobody mentioned, the entire investigation is discarded — and it would have been the first thing on a change feed.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Systems in steady state stay in steady state. The overwhelming majority of production incidents are triggered by a change, because a system with no change and no growth mostly keeps doing what it was doing.
- That makes change a very strong prior. The set of things that changed in the last few hours is small, enumerable and timestamped; the set of things that could be wrong is neither.
- Correlating a symptom onset with a change is cheap: one axis, two series. Reasoning about a code path is expensive: unbounded, and biased by whatever you looked at first.
- The prior is not a proof. Change correlation produces a ranked suspect list, and the top suspect is sometimes innocent — a deploy that coincides with a provider degradation is a real thing that happens (Change Correlation).
- It also has a decisive practical property: the top suspect usually has a reversal already built. You can act on the suspicion before you have confirmed it, which no code-reading hypothesis lets you do (Rollback: Only Useful If It Is Actually Safe).
Two openings to the same incident
This is the same incident investigated two ways. The difference is not skill or tooling; it is which question was asked first.
Read the errors. They are connection timeouts to the primary database. Look at the database: CPU is elevated but not pinned. Look for slow queries. Find several. Read the code that issues them. Discuss whether an index is missing. At 14:51 someone mentions the 13:58 release.
Open the change feed for 13:30–14:10. One release at 13:58, rollout completing 14:02. Onset at 14:03. Roll back at 14:06; error rate returns to baseline by 14:09. Then read the diff and find a query that lost its `LIMIT`.
Both paths reach the same query. One reaches it after the impact is over and with the diff to point at it; the other reaches it through the entire space of things that could make a database slow, while users are still failing.
Ranking the suspects
Not every change is equally suspicious, and the ranking is not "most recent first". What matters is the combination of how close the change is to the symptom in time, how plausibly it touches the failing path, and — decisively for what you do next — how cheaply it reverses.
The last column is the one that turns an investigation into an action. A suspect you can reverse in ninety seconds is worth acting on at much lower confidence than one that takes an hour to undo.
| Change type | Typical delay to symptom | How it hides | Reversal cost |
|---|---|---|---|
| Code deploy | Seconds to minutes, following the rollout curve | Impact ramps with traffic share, so onset looks gradual | Low — redeploy the previous artifact (Rollback: Only Useful If It Is Actually Safe) |
| Config change | Immediate, or at next restart, or at next cache expiry | Often not in the deploy feed at all; may apply per-instance over time | Low if versioned, high if edited in a console |
| Feature flag flip | Immediate for the targeted cohort only | Blast radius is a cohort, so global metrics barely move at first | Very low — flip it back (Feature Flags: Deploy Is Not Release) |
| Infrastructure change | Minutes to hours; some only surface under load | An IaC apply that recreated a resource looks like nothing to the app team | Variable — a rename that destroyed a resource may not reverse at all (Destructive Changes: What a Rename Really Does) |
| Dependency version | From the deploy that picked it up, which may be days later | The change was upstream; your repository shows only a lockfile line | Low if pinned, impossible to identify if unpinned (Dependency Pinning) |
| Managed service or provider change | Whenever they rolled it, not when you looked | Not in any feed you own; visible only as a dependency metric moving | None — you can only route around it |
| Traffic change | At the onset of the pattern, e.g. a campaign or a batch job | Nothing in your systems changed at all | Shed or throttle, not reverse (Load Shedding) |
| Data or schema change | At the migration, at the backfill, or when the row count crosses a plan boundary | A backfill saturating the primary looks like generalised slowness | Often not reversible (Destructive Migrations) |
When the feed says nothing changed
A genuinely empty change window is not a dead end. It is a strong signal that redirects the search toward the failure classes that need no trigger, and those are a short list.
Before trusting it, though, check that the feed covers all four categories. "Nothing changed" from a feed that only contains code deploys means "no code deployed", which is a much weaker statement.
What breaks a system that nobody touched?
when Gradual onset; a resource curve approaching a ceiling — disk, connections, memory, a table growing past the point where a query plan flips
cost Requires that you were graphing utilisation against the limit rather than the raw value (Headroom)
when Your error rate tracks a dependency's latency, and your own saturation is normal
cost Only visible if you measure each dependency from your side; provider status pages lag
when Onset lands on the hour, on a scheduled job, at month end, or at a certificate expiry
cost Scheduled jobs and certificate lifetimes are rarely on any dashboard (Renewal: Automating the Thing That Expires)
when Request volume or mix moved without you changing anything — a campaign, a client retrying, a crawler
cost Needs traffic broken down by client and endpoint, not just a total
when Impact is sharp and localised and none of the above fits
cost Assume this last, but check it: console edits, manual restarts and out-of-band scaling are the usual gaps (Manual Production Changes)
How to do it properly
Most important first.
- Open the change feed before the code. Deploys, config changes, infrastructure changes, dependency updates, traffic shifts, flag flips — all on one axis with the symptom (The Debugging Timeline).
- Compare the symptom onset time against the rollout window, not against the merge time. A rolling deploy takes minutes and impact often appears part-way through, as the new version takes a larger share of traffic (Rolling: Two Versions, One Database).
- Ask about changes you did not make: a managed service upgrade, a certificate renewal, a provider deploy, a DNS TTL expiring, a scheduled job that only runs on the first of the month.
- If the top suspect is reversible and the impact is user-facing, reverse it. Confirming first is a luxury purchased with user pain.
- Make "no change" a signal rather than an absence: if the change feed genuinely shows nothing in the window, that is real information and it promotes growth-against-a-limit and dependency degradation to the top.
- Record what you ruled out and when, so the next responder does not repeat it.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The habit does not cause impact; it shortens it. What it depends on is a reversal path — without one, knowing the suspect earlier buys you less.
What can go wrong
- Only code deploys are in the feed, so config, infrastructure and dependency changes are invisible and "nothing changed" is wrong (A Config Change Is a Production Change).
- The feed shows deploy start but not rollout completion, so the correlation looks weak when it is actually exact.
- Rolling back the top suspect masks a different fault that the deploy merely made visible — the incident returns hours later with no deploy to blame.
- A change that was flagged off ships days before its impact, because the flag flip is the real change and it is not in the deploy feed (Feature Flags: Deploy Is Not Release).
- Clock skew between the deploy system and the metrics system makes a correlation look like it happened in the wrong order (Clock Synchronisation).
- The habit overcorrects into blaming whatever deployed most recently, and a team that always rolls back stops learning what actually broke.
- "Most incidents follow a change, so most changes cause incidents." The base rates are wildly different; the correct conclusion is to look at changes first, not to ship less.
- "Correlation proves cause." It ranks suspects. Two things happening at 14:03 in a system that deploys hourly is weaker evidence than the same thing in a system that deploys monthly.
- "Change means the code deploy." Config, infrastructure, dependency and traffic changes are all changes, and three of them are usually missing from the feed (Change Correlation).
- "If we deploy less we will have fewer incidents." Deploying less produces larger deploys, which are harder to correlate and harder to reverse (Change Size: Why Small Changes Are Safer, and When They Are Not).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Anyone on call can list every change in a given hour, across all four categories, from one place, without asking a person.
- The gap between symptom onset and the nearest change is a number you can state, not an impression.
- Post-incident, the timeline shows the correlation was checked in the first few minutes, whether or not it was the cause.
- This is a habit rather than a deployment, but it changes what rollback means: acting on correlation requires that reversal is cheap and safe, so the habit is only as good as your rollback story.
- Where reversal is not safe — a completed destructive migration, a consumed one-way API call — correlation still narrows the search, but you must roll forward instead (Roll Forward: When Going Back Is the Harder Option).
- Automate the change feed itself: deploys, config, IaC applies, flag flips, dependency bumps and scaling events into one timestamped stream (The Audit Trail).
- Automate deploy annotations onto every operational dashboard so the correlation is visible without anyone constructing it (Deploys on the Same Timeline as the Symptom).
- Automated rollback on canary analysis is appropriate; automated attribution of cause is not. A machine can say "metrics regressed after this rollout"; only a person should say "and that is why".
- It biases you toward the recent and away from the slow-burning. Capacity exhaustion, memory growth and data-shape drift produce incidents with no change to find, and this habit is worst exactly there.
- Acting on correlation means sometimes rolling back an innocent release, which costs the team a shipped change and some credibility if it happens often.
- Building a real change feed spans several systems — CI, config, IaC, flags, dependency management — and nobody owns it by default.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALHolds anywhere software is changed continuously. The lower the deploy frequency, the stronger each individual correlation is — and the larger and harder to reverse each change becomes.
- SCALE-SPECIFICOn a system deploying many times an hour, several changes will always sit in the window and correlation alone cannot single one out; you need per-change rollout metrics (Canary Analysis: Compared Against What?). On a system deploying weekly, the correlation is nearly conclusive but the change is huge.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.