Incremental Migration
Old and new coexist; you migrate one slice, verify it against reality, and repeat. The design work is choosing the slice and defining what "verified" means.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
If old and new have to run side by side for months, what does one increment consist of and how do I know it worked?
Move customer identity out of the monolith into a new service. Fifty-two call sites, four downstream consumers, and a table that eleven other tables have foreign keys into.
Build the new identity service, point everything at it in one release, and delete the old code. Coexistence is complexity for its own sake; the switch is one config change.
Fifty-two call sites do not behave identically, and the four or five that differ will not be found by reading. They are found by traffic, which means they are found after the switch.
- Fifty-two call sites do not behave identically, and the four or five that differ will not be found by reading. They are found by traffic, which means they are found after the switch.
- The switch is atomic in the config and not atomic in reality: in-flight requests, cached values, running jobs and replicas mid-deploy all straddle it (Partial Failure).
- Rolling back after the new service has written identity data means reconciling two sources of truth, which is a data-repair project rather than a revert (The Dual Write Problem in Backend).
- As requirements arrive during the migration — and they do, because identity is central — there is no rule about which system they land in, so they land in both and diverge.
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- There is no maintenance window; the system is live continuously.
- The team can dedicate two engineers, which sets the size of a slice: whatever two people can migrate and verify inside a sprint.
- The eleven foreign keys mean the identity table cannot simply move, so data coexistence is a hard requirement rather than a phase (Data Migration).
- Rollback must remain available for at least two weeks after each slice, which constrains how quickly the old path can be deleted.
- At any instant, every piece of behaviour has exactly one authoritative implementation. Coexistence means running both, not believing both.
- Every increment must be independently revertible. An increment that can only be undone by undoing the one before it is not an increment.
- The system is releasable at the end of every increment, including the ones in the middle where old and new both exist (Designing the Migration).
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- Each increment owns one slice of behaviour, its data, its call sites and its verification. If those four are not all in the increment, the increment is not finished.
- A single component owns which implementation is authoritative for a slice — a flag, a router, a config key — and nothing else may decide it (Feature Flags and What They Cost).
- Someone owns the divergence report between old and new, and owns triaging it before each traffic increase. An unowned report is not verification.
- Someone owns deleting the old path per slice, on a date. Deletion is part of the increment, not a cleanup epic (Deprecation).
- A slice boundary should be a behaviour with its own callers and its own data. "Read a customer's email address" is a slice; "the persistence layer" is not.
- The verification boundary is wherever both implementations can be run against the same input and compared. If no such point exists, create one before migrating anything (Seams).
- The rollback boundary is the increment: everything in one increment must revert together, and nothing outside it should have to (Reversible and Irreversible Decisions).
The four states every slice passes through
Coexistence is not one state, it is four, and treating them as one is how migrations end up with both systems authoritative. Making them explicit means the current state of every slice is a fact rather than a guess, and it makes the forbidden transitions nameable.
The two transitions that must not exist are the interesting part. Jumping straight to the new implementation skips the evidence; going backwards from a deleted old path is not a rollback, because the code is gone.
| From | On | To | Guard | Effect |
|---|---|---|---|---|
| OldOnly | new implementation deployed | Shadowing | contract suite passes against both implementations | divergence report starts collecting |
| Shadowing | traffic shift begins | Splitting | divergence below threshold for a full business cycle, including slow paths | switch set to a percentage |
| Splitting | percentage reaches 100 | NewAuthoritative | no unexplained divergence and error rates unchanged | old path idle but intact |
| Splitting | rollback | Shadowing | no data written by the new path that the old cannot read | traffic returns to old; divergence triaged |
| NewAuthoritative | rollback | Splitting | within the agreed two-week window and data is still dual-readable | switch reduced; incident review |
| NewAuthoritative | retirement date reached | Retired | two weeks with no rollback and no divergence; data fully migrated | old code, old tests and switch deleted |
- OldOnly → NewAuthoritative — The switch with no shadowing phase — no evidence that the implementations agree on real traffic, and the first divergence is discovered by a customer. This is the naive cutover, and it is forbidden because the fifty-two call sites are not identical.
- Retired → NewAuthoritative — There is nothing to roll back to; the old path was deleted. Retirement is deliberately terminal, which is why the guard on entering it is a completed data migration and a rollback window that has expired unused.
- Shadowing → Retired — Deleting the old path while it is still the authoritative one leaves the slice with no working implementation. Every path to Retired must pass through NewAuthoritative.
Every slice is in exactly one of these states, and the set of states across all slices is the migration's actual status — far more useful than a percentage (Explicit State).
One increment, and what verification means
An increment is not "some work on the migration". It is a slice taken from OldOnly to Retired, and it either got there or it did not. Increments that stop at NewAuthoritative accumulate flags and branches, which is how a migration becomes permanent.
The verification step is the one worth being precise about, because "it looks fine" is what everyone says right up until the monthly invoice job runs.
- 1Establish coexistence
Both implementations reachable, switch in place, old authoritative. Ships as its own release.
fails by Building the switch as part of the first behaviour change, so a bug in either is indistinguishable from a bug in the other.
- 2Shadow on live traffic
Run both, serve old, record divergence per call site.
fails by Shadowing only the hot path, missing the nightly job and the partner integration.
- 3Triage divergence to zero-or-explained
Every difference is either fixed or written down as intended, by a named person.
fails by Accepting a standing 2% as normal, which converts the report into wallpaper.
- 4Shift traffic in steps
1%, 10%, 50%, 100%, holding at each long enough to cover a full business cycle.
fails by Moving fast enough that the monthly job never runs under the new path before cutover.
- 5Migrate the data
Expand the schema, dual-write, backfill, verify counts and samples, move reads (Expand and Contract).
fails by Leaving the slice reading the old store, which couples the new service to the old schema indefinitely.
- 6Retire
Delete old code, old tests and the switch. Announce it.
fails by Keeping the old path "just in case", so the slice is never actually migrated and the state space never shrinks.
Two engineers, one sprint, one slice. If a slice does not fit in that box it is not a slice yet — split it (Slicing a Feature).
What the overlap costs while it lasts
The coexistence period is where migrations actually live, and it is the part no plan shows. Pricing a change during the overlap makes the trade honest: incremental migration is not cheaper, it is *bounded*, and the boundedness is what you are buying.
This also explains the sequencing advice. Two slices in overlap at once is manageable; six is a system whose behaviour nobody can describe.
Customers may have a secondary email for billing. Read paths must return both; write paths must validate both.
The change must land in both implementations and keep them equivalent, or the divergence report goes red and the traffic shift stalls. Roughly double the work, plus a decision about whether to pause the shift while it lands.
One implementation, one schema, one test suite. The change is a day.
How to build it
Most important first.
- Establish coexistence first: both implementations exist, both are reachable, the old one is authoritative, and a switch exists per slice. That state is a deliverable in its own right and should ship before any behaviour moves.
- Pick the first slice for information, not value — smallest blast radius, clearest verification — because the first increment is where you find out what your migration actually costs.
- Shadow before switching: run both on live traffic, serve the old result, record every divergence. Divergence in production traffic is the only evidence that generalises (Characterization Tests).
- Shift traffic in steps and hold at each step long enough to cover the slow paths — the nightly job, the monthly invoice, the partner that only calls on Fridays.
- Migrate the data with the slice, using expand-and-contract so both shapes are readable during the overlap (Expand and Contract).
- Delete the old path and its flag as the final step of the increment. An increment that ends with both paths alive has added a state to the system rather than removed one (Boolean Flag Explosion).
- Repeat with the next slice, and re-plan after the first: the measured cost of increment one is worth more than the estimate of increments two through nine.
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- During the overlap every change to a migrated slice costs more, not less: two implementations, one switch, and a decision about whether the change belongs in both. This is the honest cost of coexistence and it is paid for months.
- After a slice is fully migrated and its old path deleted, a change to it costs what a change in the new system costs — which for identity is one service, one schema, one test suite.
- The next increment is cheaper than the last, because coexistence infrastructure — the switch, the shadow harness, the divergence report — is built once and reused. That amortisation is the reason to sequence rather than parallelise.
- The cost that grows: every increment left half-finished adds a permanent flag and a permanent branch. Ten half-finished increments cost more than the original problem (What Technical Debt Actually Is).
- Coexistence is genuinely more complex than either end state, and it is the state the system spends the longest in. Anyone who says incremental migration is "safer" without acknowledging that has skipped the expensive part (Designing the Migration).
- It is slower to a finished migration than a single cutover, sometimes much slower, and the extra months are real money.
- Slices carved from the old system tend to inherit the old system's boundaries, so an incremental migration can faithfully reproduce a structure you were trying to leave.
What can go wrong
- Both systems become authoritative for the same slice — usually through a job or a cache that was not on anyone's list of call sites — and data diverges silently until reconciliation.
- The overlap never ends. Slices two through nine are deprioritised, the flags stay, and the system permanently carries both implementations plus a switch (The Strangler Pattern).
- Rollback is exercised for the first time during an incident, having decayed for six weeks, and does not work.
- Verification is run on synthetic traffic, passes, and misses the real inputs entirely — the customer with no email, the account created in 2013 before the field existed.
- The mitigation fails on its own terms: shadow comparison reports 2% divergence, nobody can explain it, and rather than blocking the rollout it becomes a number people are used to seeing.
- Both implementations depend on the switch, which becomes a load-bearing component that needs its own tests and its own alarm.
- The new system depends on the old one for every unmigrated slice, usually through a translation layer so the old model does not leak into the new one (Anti-Corruption Layer).
- Downstream consumers depend on whichever system is authoritative, usually without knowing it. Their expectations are a dependency you inherit (Do We Need a Package for This?).
- "Incremental means small releases." It means each increment leaves the system in a coherent, releasable, revertible state. A small release that leaves two authoritative implementations is not an increment; it is a partially applied change.
- "We can migrate the data at the end." Data is the hardest part and it is inside each slice, not after all of them. A slice whose data still lives in the old store has not moved (Data Migration).
- "Shadowing proves equivalence." It proves equivalence on the traffic you have seen. The Friday partner, the annual job and the 2013 account are all outside it, which is why holds at each traffic step have to be long enough to include them.
- "The flag is temporary so it does not need tests." The flag decides which implementation runs. It is the most load-bearing line in the migration and the least tested one (Feature Flags and What They Cost).
- boolean-flag-explosion
Testing it, and how it ages
- A shared contract test suite that both implementations must pass, run against both in CI, so "equivalent" is machine-checked rather than argued (Contract Tests).
- Divergence comparison on live traffic, with a threshold that blocks the traffic increase rather than merely reporting.
- A rollback rehearsal per increment, executed rather than documented, including the data path.
- Explicit tests for the coexistence states themselves: old authoritative, new shadowing, new authoritative, old removed. Those are four states of the system and each can be wrong (State Machines).
- The switch mechanism outlives the migration unless someone removes it, and it will be reused for the next one — which is fine if it is owned, and a liability if it becomes a general-purpose flag framework nobody maintains (Feature Flags and What They Cost).
- Slice ordering should be revised continuously. The slices you thought were easy are frequently the ones with the surprising consumers.
- When the last slice lands, the translation layer to the old model becomes dead weight and should go with it. It rarely does (Speculative Generality).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat a migration passes through a coexistence state, and that the state must be designed rather than endured, follows from the system staying live — true of a module extraction and of a datacentre move alike.
- SCALE-SPECIFICInside one process, coexistence is a branch and a flag and the overlap can last an afternoon. Across services with separate stores and separate deployments, coexistence needs dual-write, backfill, reconciliation and a rollback plan, and lasts months. The word "increment" hides a difference of two orders of magnitude in cost.
- CONTESTEDThe strongest opposing view: long coexistence is itself a major source of defects, because the overlap doubles the state space and every bug during it is a bug in a configuration that will never exist again. Practitioners who favour short, sharp cutovers argue that the effort spent making coexistence safe would have been better spent on a rehearsed cutover with a tested rollback, and that "gradual" often means "the risky part happened while nobody was watching". That is a strong case for slices small enough to cut over in days, and a weak one for a big-bang switch of fifty-two call sites at once.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — System Design — traffic splitting, shadow load and the capacity implications of running two implementations of the same behaviour against production volume.