Destructive Migrations
Dropping, renaming, truncating and narrowing are the only changes with no rollback — and during a rolling deploy they break the instances that have not been replaced yet.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why is DROP COLUMN dangerous when the column is unused, and what does a safe destructive change look like?
Every other change in this domain has an inverse. A destructive migration does not: the previous artifact cannot bring the data back, and the code still running when it lands is code that was written when the thing existed.
The new code does not use the column any more, so the migration that drops it can ship with the release that stopped using it. It is dead weight; removing it is cleanup.
The release that stopped using it has not finished rolling out when the migration runs. Every instance still on the previous artifact is querying a column that no longer exists.
- The release that stopped using it has not finished rolling out when the migration runs. Every instance still on the previous artifact is querying a column that no longer exists.
- The failure is total for that table rather than partial. ORMs generate explicit column lists from the model, so an old instance's every query against that table references the dropped column and fails — not just the queries that used the value (What an ORM Actually Does).
- Rollback makes it worse. Reverting to the previous artifact puts back code that requires the column, against a schema that no longer has it.
- The blast radius is the whole fleet at the moment the migration commits, and it shrinks only as the rollout replaces instances. This is the opposite shape from a bad deploy, which starts small.
- "Unused" was established by reading code. Scheduled jobs, admin tools, analytics readers, replicas serving a reporting workload and a paused canary are all code that was not in the repository you read (Cron Jobs in Production).
- For a drop that removes data rather than a definition, there is no version of the system that can recover it except a restore, with everything since the backup lost (RTO and RPO).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Destructive changes come in four shapes with different recoverability: dropping a definition (a column, table or index), removing values (truncate, delete), narrowing (a type or length change that cannot represent existing values) and renaming (which is a drop and an add that the engine performs atomically).
- Renames are the most under-recognised: the schema keeps the data, so it feels safe, but from the perspective of code the old name is gone. Every consequence of a drop applies.
- The recoverability of a definition drop varies by engine and by what it stored. Dropping a column in PostgreSQL marks it dropped in the catalog and leaves the bytes on disk until the rows are rewritten — but there is no supported path to get it back, so "the data is technically still there" is not a recovery plan.
- The compatibility failure has the inverse shape of a normal bad deploy. A bad artifact starts on one canary instance and spreads as you roll forward; a destructive migration starts at 100% of the fleet and shrinks as you roll forward. Canary analysis cannot see it, because the canary is the healthy version.
- The safe form is always the same: make the thing unreferenced, prove it is unreferenced by measuring rather than by reading, wait longer than the lifetime of anything that could still reference it, and only then remove it (Expand, Migrate, Contract).
A drop during a rolling deploy, minute by minute
The shape of this timeline is what makes destructive migrations distinctive: impact is maximal at T+0 and decreases as the rollout progresses, which is the opposite of every other failure in this domain and defeats the instinct to roll back.
- T+0changePipeline applies
ALTER TABLE orders DROP COLUMN legacy_status. The statement completes in milliseconds. - T+0signalEvery instance in the fleet is still the previous artifact. Its generated queries name
legacy_status. 100% of requests touchingordersbegin failing. - T+30ssignalError-rate alert fires. Dashboards show a total failure on one table, all instances, all endpoints that touch it.
- T+1mchangeRollout of the new artifact begins. Each replaced instance recovers, so the error rate starts falling — slowly, in proportion to rollout progress.
- T+2mactionOn-call, seeing errors during a deploy, initiates a rollback. This is the correct instinct and the wrong action here.
- T+3msignalThe rollback replaces new instances with old ones, which fail against the schema. The error rate stops falling and starts rising again.
- T+6mactionSomeone reads the migration in the release and identifies the coupling. The rollback is abandoned and the rollout is resumed — rolling forward is the only path (Roll Forward: When Going Back Is the Harder Option).
- T+12mrecoveryRollout completes. The last old instance is replaced. Errors stop.
Two details generalise. First, the rollback made it worse — so a runbook for this class of incident must say "check whether the release contained a schema change" before it says "roll back" (Deployment-Centric Debugging). Second, nothing here was slow or badly configured. The statement was instant and the deploy worked exactly as designed.
Four kinds of destruction, four recovery stories
Grouping these as "destructive" is useful for gating and misleading for planning: the recovery paths are genuinely different, and so is how much warning you get.
| Change | What is lost | Recovery | Compatibility impact |
|---|---|---|---|
DROP COLUMN | The definition, and access to the values | Re-add the definition (instant, restores nothing) or restore from backup | Total for that table on any code that names the column — which, with an ORM, is all of it |
DROP TABLE | Definition and all rows | Restore only. In PostgreSQL a drop inside an open transaction can still be rolled back; MySQL commits it immediately | Total for anything reading it, including consumers outside your repository |
| Rename column or table | Nothing, physically | Rename back — fast, and genuinely a rollback | Identical to a drop from the code's perspective; this is the trap |
TRUNCATE | All rows; definition survives | Restore only; not a transactional delete on either engine | Reads succeed and return nothing, which is worse than an error because it looks like a product bug |
| Narrowing a type or length | Values that no longer fit | Restore, and only for the rows that were truncated | New writes may be rejected; old code may write values the new type refuses |
| Dropping an index | No data | Rebuild it — but the rebuild is slow, and the interval is spent scanning | None functionally; potentially severe on latency, immediately (Why Is This Query Slow? Indexes) |
The procedure that makes it boring
None of these steps is difficult. The discipline is that the last one is gated on the evidence produced by the ones before it, rather than on the belief that produced the change.
- 11. Stop writing it
Ships as an ordinary code change. The thing still exists and is still read.
fails by Missing a write path — a batch importer, an admin action, a queue consumer.
evidence Write counter on that path reads zero across a full traffic cycle.
- 22. Stop reading it
Ships as a separate code change. The thing is now referenced by nothing, and still exists.
fails by Being combined with step 1, which halves the number of safe rollback targets.
evidence Read counter on that path reads zero; the column is still populated and still correct, so this step is fully reversible.
- 33. Prove disuse
Soaks for longer than your rollback horizon and longer than the slowest scheduled consumer.
fails by Proving it by grep instead of by measurement, which cannot see consumers outside the repository.
evidence Counters at zero for the full soak, covering at least one run of every periodic job (Job Scheduler Reliability).
- 44. Make it reversible first
Renames rather than drops where the engine allows —
orders_legacy_status_deprecated— or stops populating without removing.fails by Skipping to the drop because the rename feels like a half measure. It is a half measure, deliberately.
evidence Nothing breaks after the rename, which is the last cheap proof you get.
- 55. Back up and verify
Takes a backup of the affected table and restores it somewhere to prove it is readable.
fails by Treating a successful backup job as a successful backup (Restore Drills).
evidence Rows verified present in a restored copy, by query.
- 66. Drop
Removes it, under an explicit approval, with a lock timeout, in its own release.
fails by Shipping it alongside anything else, so if something breaks you cannot tell which change did it (Change Correlation).
evidence Error rate flat afterwards — and the knowledge that if it were not, you would have to roll forward.
Steps 1 and 2 are separate releases on purpose. Each additional release boundary is another artifact you can safely roll back to, and the number of safe rollback targets is the real measure of how well-sequenced a destructive change is.
How to do it properly
Most important first.
- Never ship a destructive migration in the same release as the code change that stopped using the thing. One release stops using it; a later release removes it.
- Prove disuse with a signal: a counter or log line on the old access path that has read zero for a defined soak period, covering at least one full cycle of every scheduled job (The Audit Trail).
- Make the soak period longer than your rollback horizon. If you would consider rolling back to a two-week-old artifact, the column stays for more than two weeks.
- Prefer a reversible intermediate: rename the table to
table_deprecatedand leave it, or stop writing a column before dropping it. A rename you can undo is a much better first step than a drop you cannot. - Take a verified backup of the affected table immediately before the destructive step, and verify it by restoring it somewhere, not by checking that the backup job exited zero (Restore Drills).
- Require an explicit, named approval for destructive statements, separate from ordinary code review — this is one of the few places a gate genuinely earns its cost (Change Management).
- Have a CI check that detects destructive statement forms and fails the build unless the change carries the approval marker (Policy as Code).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing, and uniquely so: this is the one change whose blast radius starts at 100% and shrinks as the rollout proceeds. The only containment is time — putting enough distance between "stopped using it" and "removed it" that no live code remembers it.
What can go wrong
- The drop lands during a stuck rollout, so the compatibility window that should have been minutes is however long the rollout stays stuck.
- A rollback attempted during the resulting incident, which restores the old artifact against the new schema and fails identically (Rollback: Only Useful If It Is Actually Safe).
- A rename treated as non-destructive because no data was lost, shipped in a single release, breaking the whole fleet on commit.
- A "temporary" backup taken before the drop, to a location nobody can restore from under time pressure, or in a format the current engine version cannot read.
- Disuse proved by a code search that misses a consumer outside the repository — a data pipeline, a BI tool, a partner integration reading a replica (Read Replicas From the Application).
- The mitigation failing: a soft-delete convention that everyone stops honouring, so the "reversible" rename accumulates until someone does a real cleanup pass and drops a table that was still in use.
- Truncating the wrong table because the statement was typed against production while intending staging — the classic, and the reason destructive access should be deliberate rather than ambient (Production Access).
- "The column is unused, so dropping it is safe." Unused by the code you are deploying. The code currently running is a different question, and it is the one that matters.
- "A rename is not destructive because no data is lost." From the code's point of view a rename is a drop plus an add. Every failure mode of a drop applies to it.
- "We have backups." A restore is an outage plus a data-loss window. It is the disaster response, not the undo button (Backup Operations).
- "The canary will catch it." The canary runs the new code, which is fine with the change. The failure is on the old version, which is everything the canary is being compared against (Canary Analysis: Compared Against What?).
- "
DROP COLUMNis instant, so it is low risk." Duration and risk are different axes. This is the fastest statement in the module and the only one with no inverse. - "We will drop it later" — said once, and then never revisited, which is how schemas end up with columns whose meaning nobody can reconstruct. Deferring is correct; forgetting is not.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Before: the disuse counter has been zero for the full soak period, and you can say what the longest-lived consumer of that table is.
- Before: a restore of the pre-change backup has been performed into a scratch environment and the affected rows verified present.
- During: error rate by application version. If old instances start failing, you will see it on one version and not the other — which is also how you tell this apart from a bad artifact.
- After: error rate flat, and the disuse counter now permanently absent rather than zero.
- After: reclaimed storage, where the engine reclaims it — and no surprise if it does not, since several engines only reclaim on rewrite.
- There is none. This is the lesson: the destructive step is the one place in this domain where "how do I get back" has the honest answer "you do not".
- Recovery, which is different, means restoring from backup and losing every write since — an outage plus data loss, chosen deliberately as the lesser harm (Partial and Logical Data Recovery).
- For a definition drop with no data loss, the recovery is to re-add the definition, which is fast and does not bring values back. That distinction matters: re-adding a dropped column takes seconds and restores nothing.
- The real rollback plan is the sequencing: because the code stopped using it a release earlier, rolling the application back to that release is safe. The rollback horizon is exactly how far back you can go before you meet an artifact that needed the thing.
- Automate detection: a linter that identifies destructive statement forms in a migration diff and requires an explicit acknowledgement to merge.
- Automate the pre-flight: take and verify the backup as a pipeline step, so it is not a thing someone remembers to do.
- Automate the disuse metric so the evidence is collected continuously rather than assembled the day someone wants to drop something.
- Keep human, emphatically: the approval to run it, and the timing. This is the canonical case where a fully automated path is worse than a gated one (Guardrails, Not Gates).
- The safe sequence means unused schema hangs around for weeks, and schemas accumulate columns nobody can justify.
- The soak period conflicts directly with the desire to finish a migration, and the pressure to skip it is highest when the sequence has already taken longer than expected.
- A gate on destructive changes is a real gate, with the queueing and the exception process that implies — justified here by the absence of a rollback, and not a precedent for gating everything else.
- Keeping a pre-drop backup for a meaningful retention window costs storage and creates a data-protection obligation for data you were trying to remove (Sensitive Data Classification).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe compatibility failure follows from rolling deploys and is engine-independent. So is the sequencing rule: stop referencing, prove disuse, wait, then remove.
- DATABASE-SPECIFICRecoverability and cost differ. PostgreSQL wraps DDL in transactions, so a
DROPinside an uncommitted transaction can be rolled back — and itsDROP COLUMNis a catalog change that does not reclaim space until rows are rewritten. MySQL commits each DDL implicitly, so there is no transaction to abort, andTRUNCATEin particular is not a transactional delete on either engine. Neither engine offers a supported way to recover a dropped column's values. - ORG-SPECIFICWhether a destructive change needs a named approval, a change record or only a code review is a policy decision, not a technical fact. What is not optional is that someone specific decided to run it and that the decision is recorded (The Audit Trail).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.