Roll Forward: When Going Back Is the Harder Option
Irreversible migrations, side effects already emitted and dependencies that moved on — the situations where the fix has to go forward, and how to ship one safely under pressure.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The change is bad and reverting it would be worse. How do I ship a fix under time pressure without making a second incident?
Rollback is the default mitigation because it returns to a known-good state. Some changes destroy the known-good state on their way past, and after that the only direction available is forward — usually at the worst possible moment to be writing code.
Always roll back first. Rolling forward means shipping untested code during an incident, which is exactly what you should not do.
Rolling back across an irreversible change is not returning to a known state — it is entering a state that has never existed, which is strictly more dangerous than a small fix.
- Rolling back across an irreversible change is not returning to a known state — it is entering a state that has never existed, which is strictly more dangerous than a small fix.
- When the previous version cannot read the current data, a rollback produces its own error rate on top of the one you were trying to stop.
- External side effects are already outside your system. No deployment reverses an email, a webhook or a settled payment; the only remedy is a compensating action, which is code you have to write (Partial and Logical Data Recovery).
- Sometimes the defect is not in your artifact at all — a dependency upgrade, a provider change, an expired certificate — and there is no previous version of yours that fixes it.
- Teams with a strict rollback-only doctrine have no rehearsed path for forward fixes, so the first one is improvised at 3am.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A change is reversible when the previous version is still a valid program for the current state. Roll-forward is what you do when that condition has been broken, and the four ways it breaks are: destroyed state, emitted side effects, moved dependencies, and a defect that was never in your code.
- The risk of rolling forward is not that the code is new — it is that it is shipping with less evidence than usual, through a process people are tempted to bypass, by someone under pressure.
- So the discipline is entirely about keeping the fix small and the path normal: the smallest possible change, through the same pipeline, with the same gates, ideally behind a flag so the fix itself can be disabled.
- A roll-forward that is a revert commit is still a roll-forward — it moves the system to a new version, and it goes through the pipeline like any other change. That is usually the cleanest form.
- Where the fix cannot be small — a full re-architecture, a data repair — the correct move is often to mitigate first by other means (disable the feature, shed the load, serve degraded) and ship the real fix at normal speed (Load Shedding).
The four reasons back is not available
These are the conditions to check *before* reaching for a rollback, not after it fails. Each one has a tell that is visible at review time, which is the point: reversibility is decided when the change is written, not when it breaks.
| Why back is unavailable | What the change did | The tell at review time | What forward looks like |
|---|---|---|---|
| Destroyed state | Dropped a column, rewrote rows in place, deleted data, changed an on-disk format irreversibly | The migration has a destructive statement, or a backfill that overwrites rather than adds | A fix against the new schema; restore only if a tested backup exists (Restore Drills) |
| Side effects emitted | Sent emails, delivered webhooks, took payments, called a partner API, published to a topic others consumed | The change writes to anything outside your system boundary | A compensating action — corrected notice, replay, refund — plus a fix to stop further emission |
| Dependencies moved on | A library, runtime, provider API version or certificate changed alongside or since | The release bundles an upgrade with a behaviour change | Fix forward against the current dependency, or roll back the whole set that moved together |
| Defect is not in your artifact | Nothing — the failure is in a provider, a config change, an expired credential, a data condition | Nothing in your diff explains the symptom, and the deploy correlates only by timing | There is no previous version of yours to return to; the fix is forward or the mitigation is elsewhere (Change Correlation) |
| Rollback would take longer than the fix | Nothing irreversible — but reversal is a long rolling redeploy and the fix is one line behind a flag | A known rollback duration much larger than a pipeline run | A minimal forward fix, flagged, through the normal pipeline |
Choosing a direction while impact is ongoing
This decision is made in the first minutes of an incident, which is why the criteria should be familiar rather than reasoned from scratch. Note that the first option is not "roll back" — it is to stop the bleeding by the fastest available means, which is often neither direction.
What stops user impact soonest without creating a second problem?
when The behaviour is behind a flag, or the affected endpoint can be disabled or degraded.
cost Users lose the feature entirely, and it needs the off path to still work (Feature Flags: Deploy Is Not Release).
when The previous version is still valid against current schema, config and data — the ordinary case.
cost A full rollout duration of continued impact, and it undoes nothing the change already did (Rollback: Only Useful If It Is Actually Safe).
when Back is unavailable for one of the four reasons, or reversal is materially slower than the fix.
cost Shipping under pressure with less evidence; needs the discipline of a tiny change through the normal pipeline.
when Neither direction is fast enough and impact must stop now: shed load, rate-limit, fail over, serve cached or degraded responses.
cost Degraded service for everyone rather than broken service for some, and the defect is still deployed (Load Shedding).
when The impact is bounded and already over — a batch that ran wrongly, a one-off emission — and the remaining work is data repair.
cost Requires being right that impact has stopped. If it has not, this is the option that lets it continue (Partial and Logical Data Recovery).
A roll-forward that stayed disciplined
The point of this reconstruction is the ordering: mitigate before fixing, keep the fix small, use the normal pipeline, and treat the residue as work rather than as an afterthought.
- T+0changeRelease ships. It includes a backfill that rewrote a column in place, and it emits a notification per processed record.
- T+6msignalSupport reports customers receiving incorrect notifications. The technical signals are all green — this is the class only a business signal catches (Canary Analysis: Compared Against What?).
- T+8mactionRollback considered and rejected in under a minute: the backfill overwrote the old values, so the previous version has nothing correct to read.
- T+9mrecoveryNotification sending disabled by flag. Emission stops. Impact is bounded even though nothing is fixed (Feature Flags: Deploy Is Not Release).
- T+12mactionScope established: how many records were rewritten, how many notifications went out, to whom.
- T+25mactionA one-line fix to the transformation is written, reviewed by a second person, and pushed through the normal pipeline — no bypass.
- T+34mchangeFix deployed behind its own flag, enabled for internal accounts first, then widened (Progressive Delivery: Exposure as a Dial).
- T+41mrecoveryCorrect behaviour verified on real records. Notifications re-enabled.
- T+50mactionResidue as planned work: a data repair for the rewritten records, and a corrective notice to affected customers with an owner and a deadline.
- Next dayrecoveryFinding recorded: the backfill should have written to a new column rather than in place, which would have made this reversible (Expand, Migrate, Contract).
Durations are illustrative. The transferable structure is: mitigate at T+9m without fixing anything, keep the fix to one line, refuse the pipeline bypass, and convert the irreversible residue into tracked work.
How to do it properly
Most important first.
- Decide reversibility before shipping, not during the incident. A change that emits side effects or contracts a schema should be labelled one-way at review time (Change Management).
- For one-way changes, plan the forward path in advance: what the likely failure is, what the fix would look like, and how you would disable the behaviour meanwhile.
- Keep the fix minimal. The smallest change that stops the impact, not the correct long-term design — that comes later, at normal speed (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Use the same pipeline and the same gates. An emergency path that skips CI is how a fix becomes a second incident; if the pipeline is too slow for emergencies, that is a pipeline problem to fix on a normal day.
- Put the fix behind a flag where you can, so the roll-forward itself has a reversal (Feature Flags: Deploy Is Not Release).
- Prefer mitigation over repair while under pressure: turning the feature off, shedding load or failing over ends impact without requiring correct code written quickly (Stop the Harm Before You Understand It).
- Write down the compensating action for side effects already emitted — the corrected email, the replayed webhook, the refund — as a task with an owner, not as something to remember.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Poorly contained by nature: a forward fix ships to the whole system under time pressure with less evidence than usual. What contains it is keeping the change tiny, putting it behind a flag so it has its own off switch, and refusing to bypass the pipeline — none of which are properties of the situation, all of which are choices.
What can go wrong
- A hurried fix that introduces a second defect, so the incident now has two causes and a confused timeline.
- An emergency bypass of CI that ships something that would not have compiled cleanly, or that was never built from the commit anyone thinks it was.
- A fix that addresses the symptom while the underlying change keeps corrupting data in the background.
- Roll-forward chosen because rollback felt like an admission, rather than because rollback was unavailable — culture producing a worse technical decision.
- Fixing forward while a destructive migration is still running, so the code and the data are both moving.
- The compensating action for side effects never written, because the incident closed once the errors stopped.
- "Rolling forward is reckless." It is the correct answer whenever the previous version is no longer valid. What is reckless is shipping a large, unreviewed change quickly — which is a property of the fix, not of the direction.
- "We rolled forward, so we did not need a rollback plan." You needed one and did not have one. That is the finding.
- "The errors stopped, so it is resolved." Side effects already emitted and data already written are still there; the incident includes them (Reconstructing What Actually Happened).
- "Emergencies justify skipping the pipeline." The pipeline is what tells you what you are running. Skipping it during an incident removes the evidence at the moment it matters most.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Impact ended, measured by the signal that detected it, and no new class of error appeared after the fix.
- The fix went through the normal pipeline and is identifiable by digest and commit, so the next responder knows what is running (The Audit Trail).
- The residue is enumerated: which records, which messages, which external effects need compensating, with an owner.
- A roll-forward needs its own reversal. Ship it behind a flag, or ensure the fix is itself trivially revertible, so a bad fix does not require a third change.
- If the fix is a data repair, the reversal is a backup and a tested restore path — verify the restore is available before running the repair, not after (Restore Drills).
- Where neither is possible, the honest answer is that this change is one-way, and it deserves a level of review that ordinary changes do not get (Destructive Migrations).
- Automate detection of one-way changes: a CI check that flags destructive migrations, and a review checklist item for anything that emits external side effects.
- Automate the fast path so it is also the safe path — an emergency deploy should be the normal pipeline running quickly, not a different pipeline with fewer checks.
- Keep the roll-back-or-forward decision human. It depends on what the change did to state and to third parties, which no tool knows.
- Rolling forward trades the certainty of a known-good state for avoiding an unknown one. That is only a good trade when the known-good state is genuinely gone.
- Keeping every change reversible constrains design — no destructive steps, no early side effects, more staging — and that constraint has a real cost in delivery speed.
- Fast emergency paths are useful and are also the paths that skip evidence. The resolution is to make the normal path fast enough that no emergency path is needed.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe four irreversibility causes — destroyed state, emitted side effects, moved dependencies, defects outside your artifact — apply to every stack. What differs is how often each occurs: a service with heavy third-party integration hits side effects constantly, an internal batch system almost never.
- ORG-SPECIFICWhether an emergency change may ship with reduced review is a governance decision, not a technical one. Regulated environments often require the same approvals under incident conditions and compensate with pre-approved runbooks; other organisations grant on-call broad authority. Both work; what does not work is having no stated answer until an incident.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.