Embedding Drift
Retraining an embedding model produces a new coordinate system. Vectors stored from the old model are incompatible with it — a version mismatch, not a quality problem — and the vocabulary and the entities drift underneath as well.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The embedding model was retrained and the stored vectors were left in place. What is now wrong, how would you detect it, and what does a safe rollout of a new embedding space look like?
A content platform re-embeds articles nightly for a related-articles feature and retrains the embedding model monthly. After the last retraining, related-article quality dropped sharply for older articles and stayed fine for new ones. Meanwhile the article vocabulary has shifted over the year — new product names, new slang — and the model tokenises the newest articles poorly.
Retrain the encoder on fresh data, deploy it, and let the nightly job embed new content with it. Old vectors are still good vectors — they were computed from the same articles — so leave them; re-embedding an archive of millions is expensive and slow.
Old vectors are good vectors in a coordinate system that no longer exists. The new encoder's query vector is compared against them by cosine and the angles are meaningless; retrieval for any old article is random with respect to content. The dimensionality matched, so nothing failed (Feature and Model Versioning).
- Old vectors are good vectors in a coordinate system that no longer exists. The new encoder's query vector is compared against them by cosine and the angles are meaningless; retrieval for any old article is random with respect to content. The dimensionality matched, so nothing failed (Feature and Model Versioning).
- The nightly job re-embeds only new and edited articles, so the index is a mixture of two spaces in proportions that shift each night, and the quality curve slopes down gradually as the mixture changes — which looks like slow drift rather than the step change it is.
- Independently of the retraining, the vocabulary moved: the fixed tokeniser fragments new terms, the encoder has never seen them, and the newest articles are embedded from partial text. Retraining the encoder on new data helps only if the tokeniser is rebuilt with it, which it was not.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Keep the related-articles retrieval working through model retraining: every stored vector must be comparable to every query vector, which means every vector in the index must come from the same model version as the query encoder.
- The surrounding system's target is click-through on related articles; a retrieval failure on old articles is a targeted loss on the long tail of the archive.
- An index of article vectors written by whichever encoder version ran the nightly job; after the retraining, new articles are embedded by the new encoder and old articles still carry vectors from the old one, with no version column.
- A tokeniser and vocabulary fixed at the original training time. Terms that emerged since map to unknown or fragmented tokens, so recent articles are encoded from a degraded view of their own text.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An embedding space is defined only up to rotation by its training objective (Embedding Training); two training runs land in unrelated orientations even on identical data, and on new data the axes carry different content as well. A vector from run A and a vector from run B share a shape and nothing else. There is no gradual degradation between them — the mismatch is total from the first query.
- Within one space, the world still moves. New entities arrive with no training pairs (Cold Start), existing entities change their behaviour so their trained rows describe who they were, and the vocabulary the encoder tokenises with falls behind the text it is asked to encode. These are the embedding forms of Data Drift and Concept Drift, and they motivate retraining — which then causes the version break.
- Detection has two sides. Version mismatch is detected structurally: every stored vector carries the encoder version and the query path checks it. Drift within a version is detected by the same signals as any model: neighbour overlap between versions for a fixed anchor set, the fraction of unknown tokens, and the downstream metric (Model Monitoring).
- Rollout has two honest options. Re-embed everything with the new encoder into a new index and cut over atomically, so the old and new spaces never mix; or learn a linear alignment from the old space to the new one on entities present in both, and transform stored vectors — approximate, cheaper, and only as good as the alignment's residual on held-out entities.
A different map, not a worse one
The instinct that "old vectors are still good" is right and irrelevant. Each version defines its own coordinate system, and cosine between a vector from one and a vector from the other is a number with no meaning. The failure is not degradation; it is a category error that the type system cannot catch because both are arrays of the same length.
The index therefore needs the version as data, and the query path needs to check it. That turns an invisible geometric mismatch into an ordinary version mismatch with an ordinary error.
1type StoredVector = { id: string; encoder: string; v: Float32Array }2 3function related(query: Float32Array, queryEncoder: string, index: StoredVector[], k: number) {4 const same = index.filter((s) => s.encoder === queryEncoder)5 const stale = index.length - same.length6 if (stale > 0) metrics.gauge('embedding.version_mismatch', stale / index.length)7 if (same.length === 0) throw new Error(`no vectors for encoder ${queryEncoder}`)8 // cosine only among vectors from the same coordinate system9 return same10 .map((s) => ({ id: s.id, sim: cosine(query, s.v) }))11 .sort((a, b) => b.sim - a.sim)12 .slice(0, k)13}The gauge is the number the archive-quality investigation needed. A filter like this is too slow for a real index — the real version is a per-index tag and a check at cut-over — but the contract is the same: never compute a similarity across encoder versions.
Rolling out a new space
Because the old and new spaces cannot coexist in one index, the rollout is a cut-over, and the decision before it is whether the new space is better rather than merely newer.
- 1Retrain encoder and tokeniser together
Rebuild the vocabulary from recent text and train the encoder against it; produce one artifact containing both, versioned.
fails by The tokeniser is left fixed and the encoder is retrained against fragmented tokens for new terms.
- 2Compare spaces on the anchor set
Neighbour overlap at small k between old and new for a fixed anchor set; expect low overlap, and inspect where it is lowest.
fails by Low overlap is read as breakage, or as improvement; it is neither on its own.
- 3Re-embed the full corpus into a new index
A batch job embeds every entity with the new encoder, tagging the version; the old index keeps serving.
fails by Only new content is re-embedded and the live index becomes a mixture.
- 4Shadow the new index
Serve the downstream feature from both indexes for a traffic sample; compare the downstream metric, sliced by content age.
fails by The comparison is made on the proxy loss or on overlap instead of the downstream metric.
- 5Cut over atomically, keep the old index
Switch the query path to the new encoder and new index in one step; flush caches of query vectors; keep the old pair for rollback.
fails by Caches of old query vectors hit the new index for their TTL; the old index is deleted before the first week of metrics is in.
Every step has a failure that produces the mixed-version index from the problem. The pipeline is not sophisticated; it is a refusal to let two coordinate systems share a serving path.
The drift underneath
Once versions are handled, the ordinary drift remains: new entities without training pairs, entities whose behaviour has moved away from their trained rows, and a vocabulary that has fallen behind the text. These are the reasons to retrain, and the version discipline above is what makes retraining safe rather than a reason to avoid it.
The assumption to monitor is that the deployed space still describes the entities it is asked about — the same assumption every model makes, with the unknown-token rate as the embedding-specific signal.
The vectors in the serving index were produced by the serving encoder, and the entities and vocabulary they describe have not moved far from what the encoder was trained on.
holds when Version-match fraction is one; the unknown-token rate on recent text is within the training range; neighbour quality on the downstream metric is stable by content age; new entities are covered by a fallback (Cold Start).
breaks when A retraining is rolled out incrementally; the tokeniser lags a shifting vocabulary; a wave of new entities arrives without pairs; the corpus's topic mix changes so that the trained axes carry different content than they did.
respond A version mismatch is fixed by re-embedding, not by retraining. Vocabulary or entity drift is a reason to retrain the encoder and tokeniser together and to roll the new space out through the cut-over pipeline — after a shadow comparison shows it is better, not merely newer (Drift Is Not Failure).
How to build it
Most important first.
- Store the encoder version with every vector and check it on every query; refuse or route around mismatches rather than computing a cosine between incompatible spaces (Serving Contract Tests).
- Treat an encoder retraining as a full re-embedding with a blue/green index cut-over, scheduled and costed as a batch job, never as a rolling update of the live index (Re-embedding, Batch Inference).
- Rebuild the tokeniser and vocabulary with the encoder when the unknown-token rate on recent text crosses a threshold, and version the tokeniser as part of the same artifact (What a Model Artifact Contains).
- Before cutting over, compare the old and new spaces on a fixed anchor set — neighbour overlap and the downstream metric on a shadow — and use the comparison to decide whether the retraining is an improvement, not just a refresh (Shadow Deployment, Drift Is Not Failure).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The fraction of index vectors whose encoder version matches the serving encoder. Anything below one is a live mismatch, and it is the number that explains the archive's quality drop entirely.
- Neighbour overlap between old and new spaces for a fixed anchor set at a small k, and the related-articles click-through on a shadow of the new index before cut-over. Low overlap is expected and not itself a failure; a worse downstream metric is.
- Unknown-token rate on the most recent articles, as a trend. This is the vocabulary-drift signal and it moves independently of the model version.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every vector in the serving index was produced by the encoder version currently serving queries, and the index records that version per vector so a mismatch is detectable before it is served.
- The tokeniser in production is the one the encoder was trained with, and the unknown-token rate on current text is within the range training saw.
- A retraining is promoted only after a shadow comparison of the new space against the old on the downstream metric, not on the basis that fresher data must be better.
- Offline: on every encoder retraining, compute neighbour overlap on the anchor set and the shadow downstream metric against the current space; record both with the artifact.
- At cut-over: assert the new index contains a vector for every entity the old one did, all with the new version, before switching; keep the old index for rollback (Rollback & Fallback).
- Over time: version-match fraction, unknown-token rate on recent text, and the downstream metric by article age — the slice that separates a version mismatch from ordinary drift.
What can go wrong
- The alignment approach is used to avoid re-embedding and its residual is large for exactly the entities that changed most — which are the ones the retraining was meant to fix.
- The cut-over is atomic for the index and not for the caches: query vectors cached from the old encoder keep hitting the new index for their TTL.
- Version checks are added to the query path and the batch job that populates the index is not updated to write the version, so every vector reads as "unknown" and the check is disabled to stop the alerts.
- Full re-embedding on every retraining is a batch job over the entire corpus, with a cost that grows with the archive and a cut-over that needs two indexes live at once.
- Alignment is cheaper and approximate, and its error concentrates on the entities whose representation changed most — which may be the point of the retraining.
- Rebuilding the tokeniser with the encoder makes every previous artifact incompatible with the new text pipeline; the cost is versioning the tokeniser, the encoder and the index together.
- "Old vectors were computed from the same articles, so they are still valid." They are valid in the old space. The new encoder does not share it; comparing across the two is comparing coordinates from different maps.
- "Retrieval quality is declining slowly, so it is ordinary drift." A slow decline is what a mixed-version index looks like as the mixture shifts nightly. Check the version-match fraction before reasoning about the world.
- "The neighbour overlap between versions is low, so the retraining broke something." Low overlap is expected from a new coordinate system and from genuinely changed content; only the downstream metric says whether the new space is worse.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALRotation freedom between training runs and therefore incompatibility of stored vectors across versions holds for any learned embedding, from word vectors to document encoders; the only exceptions are spaces explicitly aligned or anchored during training.
- SCALE-SPECIFICOn a small corpus a full re-embedding is minutes and the blue/green cut-over is trivial; on an archive of hundreds of millions of items it is a scheduled batch job with its own cost, and alignment becomes a serious alternative despite its approximation.
Where the depth lives
This domain teaches the model and hands the rest off by name.