A Migration and a Deploy Are One Event
Schema and code version separately but must be compatible continuously, which makes every schema change a two-artifact rollout with a compatibility window.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The migration is in the same pull request as the code. Why is that not the same as them changing together?
The repository presents schema and code as one change. Production applies them as two changes, at different times, to different numbers of things — and between the two there is a window where a version of the code is running against a version of the schema it was never tested with.
Run migrations as part of the deploy. The pipeline applies the migration, then rolls out the new artifact, so by the time users see anything both halves are in place.
The rollout is not instantaneous. From the moment the migration commits to the moment the last old instance is replaced, old code is running against the new schema. That interval is the compatibility window and it is measured in minutes at best.
- The rollout is not instantaneous. From the moment the migration commits to the moment the last old instance is replaced, old code is running against the new schema. That interval is the compatibility window and it is measured in minutes at best.
- The window can become unbounded. A stuck rollout, a failed health check, a paused canary or a partial deploy all extend it indefinitely (Version Coexistence: N and N+1, in Both Directions).
- Rollback reopens the window from the other side. Reverting the artifact puts old code back against a new schema — the same incompatibility, now during an incident (Rollback: Only Useful If It Is Actually Safe).
- On some platforms the migration runs from the application container's startup, so it races: N replicas try to apply it, and the one that wins may not be the one you are watching.
- If the deploy is blue/green, both colours share one database, so the window is not a rollout artefact — it is the entire cutover period by design (Blue/Green: Paying for the Fastest Rollback There Is).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Schema has one version in production. Code has as many versions as you have running instances during a rollout. A schema change is therefore a change against an unknown mixture of code versions.
- The correctness condition is not "the new code works with the new schema". It is: every code version that can be running must work with every schema version that can be present. During a rollout that is a two-by-two matrix, and three of the four cells are the interesting ones.
- Ordering determines which cells you visit. Migrate-then-deploy visits (old code, new schema). Deploy-then-migrate visits (new code, old schema). Only one of those is survivable for a given change, and which one depends on whether the change is additive or subtractive.
- This is the rule that makes additive-first work: additive changes are safe with migrate-first, subtractive changes are safe with deploy-first, and expand/migrate/contract is simply the technique of turning every change into a sequence of additive-then-subtractive steps (Expand, Migrate, Contract).
- A rollback is a deploy, so it has the same coupling. A schema change is only safely rollback-compatible if the previous artifact tolerates the new schema — which is a property you decide when you write the migration, not when you need the rollback.
One schema, many code versions
The diagram is the whole lesson. The database has one version of the schema; the fleet has as many code versions as the rollout has produced so far. Compatibility is a property of every pair, not of the pair you tested.
- (new code, new schema) — the only combination anyone tests, and the only one that is not in question.
- (old code, new schema) — what migrate-first produces, for the whole rollout. Safe only if the change is additive.
- (new code, old schema) — what deploy-first produces. Safe only if the new code does not yet depend on the change.
- (old code, old schema) — the rollback target, which only exists if the migration left the old schema usable.
A rollout with a schema change, minute by minute
Written out, the ordering question stops being abstract. Notice how much of the timeline is spent in a state nobody explicitly designed.
- T+0changePipeline runs the migration step:
ADD COLUMN, nullable, no default. Schema is now v2. - T+0signalEvery running instance is still the previous artifact. The window is open, and 100% of the fleet is in the (old code, new schema) cell.
- T+1mchangeRollout begins. First new instances pass health checks and start serving.
- T+3msignalCanary analysis compares the new version against the old on error rate and latency. Both versions are serving; both are querying the same v2 schema.
- T+9mrecoveryRollout completes. The last old instance is replaced. The window closes.
- T+14mactionAn unrelated bug is found in the new artifact. A rollback is started.
- T+14mchangeThe window reopens: the previous artifact is being placed back against the v2 schema. Because the change was additive, this is fine — and it is fine by design, not by accident.
- T+20mrecoveryRollback completes. The column added at T+0 is still there, unused, harmless, and available for the next attempt.
Had the migration been a rename or a drop, T+0 would have broken the entire fleet and T+14m would have made the rollback useless. The additive-first rule is what makes both moments boring.
Which order, and why
There is no universally correct ordering. There is a correct ordering per change direction, and a rule for changes that are both.
The release contains both a schema change and a code change. Which goes first?
when The change is purely additive: a nullable column, a new table, a new index.
cost The whole fleet runs old code against the new schema for the rollout's duration. Harmless for additive changes; fatal otherwise.
when The change removes something, and the new code has already stopped using it.
cost New code runs against the old schema until the migration lands, so the new code must tolerate the thing still existing.
when The change is both additive and subtractive — a rename, a type change, a re-model.
cost Two releases minimum, usually more, and a long intermediate state (Expand, Migrate, Contract).
when The change genuinely cannot be made compatible and downtime is acceptable.
cost Downtime for the migration's full duration, which on a large table is the worst version of this trade (Recreate: Stop Everything, Then Start the New Thing).
when A single-instance deployment where no rollout mixture exists.
cost Breaks the moment you run a second replica, and the failure is a race that reproduces intermittently.
How to do it properly
Most important first.
- Run migrations as a distinct, ordered step in the pipeline, from one place, not from application startup on every replica (The Deployment Pipeline).
- Order by direction: additive migrations before the deploy, subtractive migrations in a later deploy after the code that stopped using the thing.
- For every schema change, write down which code versions must tolerate it — at minimum, the version currently in production and the one being deployed.
- Treat the currently-deployed artifact as a hard constraint on the migration, and say so in review. "Does the version in production survive this?" is the review question that catches most of these.
- Keep the compatibility window explicitly bounded where you can: fast rollouts, no paused canaries during a migration window, and no schema change while another rollout is in flight.
- Record the schema version in the release manifest so "which schema does this artifact expect" has an answer during an incident (The Release Manifest).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Partially, by the rollout mechanism: with a canary, an incompatibility that hits new code shows up on a small percentage first. An incompatibility that hits old code has no containment at all — it affects everything not yet replaced, which at the start of a rollout is the entire fleet.
What can go wrong
- A rollback that succeeds mechanically and fails functionally, because the old artifact cannot read the new schema. The deploy system reports success and the error rate does not move.
- A long-lived canary held at a small percentage across a migration, so the compatibility window lasts hours instead of minutes (Canary: One Percent, Then Five, Then Watch).
- Migrations applied per-replica at startup, so a slow-starting replica applies a migration after the rest of the fleet has already moved on.
- A background worker fleet deployed on a different cadence from the web fleet — so "the old version" is still running somewhere long after the web rollout finished (Background Jobs and Workers).
- A scheduled job with a weekly cadence carrying the old code, which meets the new schema days later, when nobody connects the two events (Cron Jobs in Production).
- The mitigation failing: a compatibility test suite that runs the new code against the new schema only, which is the one combination that was never in doubt.
- "They are in the same commit, so they are atomic." Atomic in git, sequential in production. The repository has no notion of a rollout.
- "Rollback is always available." Rollback of the artifact is. Whether it works depends on a decision made in the migration, often weeks earlier.
- "The window is short so the risk is small." The window is short only when everything goes well. Every mechanism that makes a rollout safer — canaries, pauses, gradual ramps — makes the window longer (Progressive Delivery: Exposure as a Dial).
- "Blue/green avoids this." Blue/green gives you two full application versions against one database simultaneously. It maximises the compatibility requirement rather than removing it.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Error rate by application version during the rollout, not in aggregate. A compatibility failure shows as errors from one version while the other is clean (Canary Analysis: Compared Against What?).
- The version distribution across the fleet reaching 100% new, with a timestamp — that timestamp is when the window closed.
- A deliberate rollback rehearsal after a migration in a lower environment with a production-shaped schema, proving the previous artifact still runs.
- A test that runs the previous release's test suite against the migrated schema. This is the cell everyone skips.
- The artifact rolls back normally. The question is whether the schema permits it, and that must be true by construction rather than by luck.
- Rolling back the schema is almost never right during an incident: it is a second migration, applied under time pressure, against a mixed fleet, and destructive if the new code wrote anything.
- The reliable posture is a schema that supports both artifacts — which means additive-only within any single release, and a contract step deferred until the previous artifact is no longer a rollback target (Destructive Migrations).
- When it truly is not rollback-compatible, say so in the release record before shipping, and plan to roll forward instead (Roll Forward: When Going Back Is the Harder Option).
- Automate ordering: the pipeline decides when the migration step runs relative to the rollout, so it is not a per-engineer choice.
- Automate the check that the schema version an artifact expects matches the schema present, and fail startup loudly if it does not (Validate at Startup, Fail Loudly).
- Keep human: the decision to proceed with a change that is knowingly not rollback-compatible. That is a risk acceptance, and it should have a name attached (Change Management).
- Additive-first means carrying dead schema for a while, and a schema that is temporarily wider than the model it represents.
- Deferring the contract step keeps rollback available but leaves the sequence unfinished, and unfinished sequences accumulate.
- Running migrations as a separate pipeline step is more moving parts than a startup hook, and requires a place for the step to run with the right credentials (Production Access).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALFollows from the fact that one database serves many application instances. True on any platform, any engine and any deployment strategy that does not stop the world.
- PLATFORM-SPECIFICWhere the migration step runs differs: a Kubernetes Job or init container, a pipeline stage with database credentials, a platform release hook, or a human at a console. Init containers in particular reintroduce the per-replica race, because they run once per pod (Kubernetes Anti-Patterns).
- DATABASE-SPECIFICWhether a half-applied migration is possible depends on transactional DDL. PostgreSQL can wrap a multi-statement migration in a transaction so a failure leaves nothing behind; MySQL commits each DDL statement implicitly, so a failure in the middle leaves a partially migrated schema that the next attempt must tolerate.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.