MigrationsGENERALDATABASE-SPECIFIC

Expand, Migrate, Contract

The pattern that makes schema change safe: add the new shape, move to it, and only remove the old shape in a later deploy.

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

How do I change a schema when two versions of my code are running against it at the same time?

The problem

A schema change and a code change cannot be applied simultaneously across a fleet. Whichever goes first, there is a window in which the schema and the running code disagree — and any single-step change guarantees that window contains something broken.

What teams do first

Rename the column and update the code in the same pull request. The migration runs, the deploy follows, and it is one atomic change from the team's point of view.

How it breaks

It is atomic in the repository and not in production. Between the migration and the last replaced instance, old code is querying a column that no longer exists — every request touching that table fails.

How it breaks in production
  • It is atomic in the repository and not in production. Between the migration and the last replaced instance, old code is querying a column that no longer exists — every request touching that table fails.
  • Rolling back the deploy does not help: the schema is already renamed, so the old artifact fails the same way it did during the rollout.
  • Rolling back the schema does not help either, if the new code has already written rows under the new name.
  • The window is not brief. A rolling deploy across a real fleet takes as long as it takes, and a stuck rollout extends it indefinitely (Rolling: Two Versions, One Database).
  • Deploying the code first inverts the problem rather than solving it: new code queries a column that does not exist yet, and the canary fails immediately.
CodeBuildTestArtifactReleaseDeployRunObserveOperateIncidentRecoverLearnImprove

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

  • The pattern replaces one incompatible change with a sequence of individually compatible ones. Each step leaves a schema that both the currently deployed code and the next version can operate against.
  • Expand. Add the new shape alongside the old. Nothing reads it yet. This step is additive, so no running code is affected and its rollback is "leave it there".
  • Deploy compatible code. Ship code that writes both shapes and still reads the old one. After this deploy, every write is present in both places; historical rows are not.
  • Backfill. Copy the historical rows into the new shape, batched and throttled, while dual writes keep new rows current (Backfills).
  • Switch reads. Move reads to the new shape, ideally behind a flag so the switch is reversible without a deploy (Feature Flags: Deploy Is Not Release).
  • Verify. Confirm the new shape is complete and that the switch changed nothing observable.
  • Contract. Stop writing the old shape, then remove it — in a later deploy, never during the rollout that introduced the new one.
  • The property that makes it work is that at no point does the currently running code depend on something the schema does not have, and at no point does the schema forbid something the running code does.

Six steps, and the two that must not share a deploy

The steps are unremarkable individually. The discipline is entirely in the boundaries between them: which ones may share a release and which must not.

One rule carries most of the safety: contract happens in a later deploy than the switch, never in the same rollout. Everything else is bookkeeping.

Renaming `old_name` to `new_name`, safely
  1. 1
    1. Expand

    Add new_name, nullable, no default. Nothing reads or writes it.

    fails by Adding it NOT NULL or with a volatile default, turning an additive change into a rewrite.

    evidence Column exists; no change in error rate or latency.

  2. 2
    2. Deploy dual-write code

    Every write path writes both columns; all reads still use old_name.

    fails by A write path nobody remembered — an admin tool, an importer, a consumer of a queue.

    evidence Count of rows with new_name populated rises with write volume, from every path.

  3. 3
    3. Backfill

    Copies historical rows into new_name in batches, throttled against replication lag.

    fails by One statement over the whole table; no resume point after a failure (Backfills).

    evidence Zero rows remain where old_name is set and new_name is not.

  4. 4
    4. Switch reads

    Reads move to new_name, behind a flag so the switch is reversible in seconds.

    fails by Shipping the switch as a deploy, so reverting it needs a rollout during an incident.

    evidence Endpoint error rate and latency unchanged against baseline; the flag has been flipped back and forth once, deliberately.

  5. 5
    5. Verify

    Confirms nothing still reads or depends on old_name, across every deployable and every scheduled job.

    fails by Verifying by reading code instead of by measuring the running system.

    evidence A counter on the old read path has been zero for longer than the lifetime of your longest-lived job (Cron Jobs in Production).

  6. 6
    6. Contract

    Stop writing old_name, then drop it — in a separate, later release.

    fails by Shipping it with step 4. This is the failure the whole pattern exists to prevent.

    evidence Column gone; error rate flat; and you already know nothing referenced it, because step 5 proved it (Destructive Migrations).

Steps 1–2 may ship together only if the migration runs strictly before the new code starts. Steps 4 and 6 must not, under any circumstances, share a release.

The same rename, as production experiences it

Written as a sequence of releases, the pattern looks slow. Written against a clock, it becomes obvious why: the expensive resource being managed is the time during which two code versions coexist, and that time is not yours to compress.

One rename, four releases
  1. Release 1changeMigration adds nullable new_name. No code changes. Additive, inert, reversible by ignoring it.
  2. Release 1 + rolloutsignalNothing observable happens. This is the correct outcome of an expand step.
  3. Release 2changeCode deploys that writes both columns and reads old_name. During the rollout, some instances write one column and some write two — both states are valid.
  4. Release 2 + rolloutsignalDual-write coverage metric climbs to 100% of writes as the last old instance is replaced.
  5. Backfill windowactionBatched backfill runs against live traffic, throttled on replication lag. It can be paused, resumed, and re-run; new writes are already covered by dual writes.
  6. VerificationsignalCompleteness query returns zero remaining rows. A sampled consistency check finds no divergence between the two columns.
  7. Release 3changeReads switch to new_name via a flag, ramped rather than flipped. old_name is still written and still correct, so the flag is a real revert path.
  8. SoaksignalThe old read path counter goes to zero and stays there, across at least one full cycle of every scheduled job.
  9. Release 4changeDual writes removed. old_name is now stale but still present — deliberately, for one more release.
  10. Release 5recoveryDROP COLUMN old_name. Irreversible, and by now boring.

The gap between release 3 and release 5 exists so that a rollback to release 2 remains possible for as long as anyone might want one. Contracting early does not make the change finish sooner; it makes the rollback path disappear while you still need it.

changesignalactionrecovery

What people actually ship instead

GENERALTrue of any deployment strategy that replaces instances gradually, and of blue/green, where both colours share one database. It is only false for a full stop-the-world deploy with the service offline, which is a different trade rather than an exemption (Recreate: Stop Everything, Then Start the New Thing).

Almost every migration incident is one of two collapses of this sequence: doing it in one step, or doing steps 4 and 6 together. Both look like reasonable simplifications in review, because a diff does not show deploy boundaries.

One pull request, two rollouts
The atomic rename
migration:  ALTER TABLE users RENAME old_name TO new_name;
code:       reads and writes new_name

deploy:
  t0  migration applies
  t0  every running instance is old code -> queries old_name -> ERROR
  t1  first new instance starts serving correctly
  t2  ... rollout continues, error rate proportional to remaining old instances
  t3  rollout completes, errors stop
  rollback? old artifact fails against the new schema too.
The staged rename
release 1  add new_name (nullable)          -> nothing reads it
release 2  write both, read old_name         -> both states valid
backfill   batched, throttled, resumable     -> no schema change
release 3  read new_name behind a flag       -> revert = flip flag
release 4  stop writing old_name             -> still present
release 5  drop old_name                     -> irreversible, and safe

Not because it is tidier — because in the second version there is no moment at which the deployed code and the schema disagree. The error window in the first version is exactly the duration of the rollout, and its rollback path is broken in both directions.

How to do it properly

Most important first.

  • Write the whole sequence down before starting, including the contract step and who will do it. The unfinished contract is the most common outcome of this pattern.
  • Make every step independently deployable and independently revertible. If a step needs the next one to be safe, it is not a step.
  • Put the read switch behind a flag rather than a deploy. Switching reads is the step most likely to reveal a problem, and a flag reverts in seconds (Feature Flags: Deploy Is Not Release).
  • Prefer dual writes in application code over database triggers: the application version is visible in review, deploys with the code, and rolls back with it.
  • Let the compatibility window be long. There is no prize for contracting the same day, and there is a real penalty for contracting while an old instance is still alive.
  • Before contracting, prove nothing reads the old shape — with a metric or a log line emitted at the old read path, not by reading the code (The Audit Trail).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone
One testEveryone
What contains it

The pattern itself is the containment. Each step is either additive (no effect on running code), reversible by a deploy, or reversible by a flag — except contract, which is contained only by the evidence you gathered before running it.

What can go wrong

Failure modes, including of the mitigation
  • Contract shipped in the same release as the read switch, which recreates the exact incompatibility the pattern exists to prevent.
  • The dual-write path writes only one shape on some code path — a background job, an admin tool, a batch importer — so the new column is silently incomplete (Background Jobs and Workers).
  • The backfill completes, but rows written during the backfill by a code version deployed before the dual-write release are missing. The verification query has to cover the whole table, not just the backfilled range.
  • The sequence stalls after the read switch. The old column stays forever, new engineers cannot tell which is authoritative, and both are half-maintained.
  • A rollback to a pre-dual-write version during the window, which stops populating the new shape without anyone noticing until the contract step fails its verification.
  • Using a trigger for dual writes and forgetting it exists — it survives application rollbacks, which is exactly what makes it hard to reason about.
Misreads this invites
  • "Expand/contract means adding a column and dropping the old one in the same migration." That is the incompatible change with extra statements. The pattern is about deploy boundaries, not about SQL.
  • "We use blue/green deploys, so we do not need this." Blue/green gives you two application versions against one database, which is the compatibility problem stated more sharply, not solved (Blue/Green: Paying for the Fastest Rollback There Is).
  • "The backfill is the hard part." The backfill is the slow part. The hard part is knowing when nothing reads the old shape any more.
  • "Contract is optional." Skipping it leaves a schema that lies about its own intent, and the next engineer has to reconstruct which column is authoritative from application code.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked
  • After the dual-write deploy: a count of rows where the new shape is populated rises monotonically with new writes, from every write path.
  • After the backfill: SELECT count(*) FROM t WHERE new_col IS NULL AND old_col IS NOT NULL returns zero, run to completion rather than sampled.
  • A consistency check comparing old and new shape on a sample of rows, run repeatedly during the window, not once.
  • Before contracting: the counter on the old read path has been zero for longer than your longest-lived deployable artifact, including batch jobs that run weekly.
  • After the read switch: error rate and latency for the affected endpoints unchanged against the pre-switch baseline (Canary Analysis: Compared Against What?).
How you get back
  • Expand: nothing to roll back. An unused column is inert.
  • Dual-write deploy: ordinary application rollback. The new column stops being populated, which is safe because nothing reads it yet.
  • Backfill: idempotent by construction, so a rollback means stopping it and re-running later.
  • Read switch: flip the flag. This is the reason the switch should be a flag and not a deploy.
  • Contract: no rollback. This is the one irreversible step in the sequence, which is precisely why it is last, separate, and gated on evidence rather than on confidence (Destructive Migrations).
What to automate, and what stays human
  • Automate the backfill: batching, throttling, checkpointing and resumption are exactly the kind of repeatable work that should not be a person running a loop in a terminal.
  • Automate the verification queries and put them on a dashboard, so "is the new column complete" is a lookup rather than an investigation.
  • Keep human: the decision to switch reads, and the decision to contract. Both depend on evidence about what is still running, which is a judgement about your own fleet (Manual Production Changes).
What this costs
  • A one-line rename becomes six steps across at least three deploys, and for a large table the backfill can span days.
  • The schema is in an intermediate state for the whole window: two columns meaning the same thing, and an ambiguity for anyone reading the schema without the context.
  • Dual writes cost a little write throughput and add a code path that can diverge.
  • The pattern trades a short, sharp risk for a long, low one — and long low risks are the kind organisations forget to finish.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

  • GENERALThe pattern is independent of engine and of ORM. It follows from one fact that holds everywhere: schema and code deploy separately, so every incompatible change needs a compatible intermediate state.
  • DATABASE-SPECIFICOnly the cost of each step is engine-specific. Adding the new column is metadata-only on modern PostgreSQL and MySQL for a nullable column; the backfill's cost profile differs because PostgreSQL's MVCC writes a new row version per update and needs vacuum afterwards, whereas InnoDB updates in place and grows undo instead (UPDATE, DELETE and Dead Tuples).

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.