Embedding Training
Lookup tables as parameters, a contrastive signal from observed pairs against sampled negatives, and the consequence: rare entities get noise, and the table is part of the model artifact and must be versioned with it.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does the training signal for an embedding actually consist of, why does a rare entity end up with a meaningless vector, and what has to ship with the weights for the vectors to be usable?
A retailer's search team trains a two-tower model — a query tower and a product tower — and ships the product vectors to a vector index. New products get vectors that return nonsense neighbours; and after a retraining, the query tower was redeployed while the index still held the old product vectors, and search relevance collapsed for a day.
Train the two towers together, export the product vectors to the index, and serve the query tower behind the search API. Retrain on a schedule and re-export. The vectors are data; the towers are the model.
A product with no purchase pairs is only ever a sampled negative, so the only gradient it receives pushes it away from random queries. Its vector ends up in a region no query points to, which is not "unknown" — it is confidently irrelevant, for all queries (Cold Start).
- A product with no purchase pairs is only ever a sampled negative, so the only gradient it receives pushes it away from random queries. Its vector ends up in a region no query points to, which is not "unknown" — it is confidently irrelevant, for all queries (Cold Start).
- The retraining produced a new query tower and a new product table in a new coordinate system: same dimensionality, unrelated axes, because nothing in the loss anchors the axes between runs. The new query tower was deployed against the old index, which is a version mismatch that no schema check catches — the vectors have the right shape (Embedding Drift).
- The table was treated as data, exported as a file with a date in its name, and the model registry held only the towers. Nobody could say which table went with which tower after the fact (What a Model Artifact Contains).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Train a query encoder and a product embedding table such that a query and a product that led to a purchase score higher than that query and a product that did not. The label is the purchase pair; it means "this query was satisfied by this product at least once".
- The downstream decision is a ranked list of products per query; the training target is a pairwise preference, not that ranking (Ranking).
- Query–purchase pairs from search logs. One example is (query text, purchased product id, several product ids sampled from the catalogue as negatives). Products with no purchases in the window appear only as negatives, which is not a signal about where they belong.
- Query text passes through an encoder with shared parameters; product ids pass through a lookup table with one row per product. The two sides are updated by the same loss and are meaningless without each other.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The lookup table is a parameter matrix; the encoder is a network. A pair is scored by the dot product (or cosine) of the two sides' outputs. The loss is contrastive: for an observed pair, push the score up; for the same query against sampled negatives, push it down. Skip-gram word vectors, two-tower recommenders and text retrievers all reduce to this shape, with different definitions of "observed pair" (Self-Supervised Learning).
- Negatives are sampled because the true set of non-matches is the whole catalogue and cannot be scored per step. How they are sampled shapes the geometry: uniform sampling teaches the model to distinguish a purchase from a random product, which is easy; sampling popular or "hard" negatives that resemble the positive teaches finer distinctions and is where most of the quality comes from (The Learning Signal).
- A rare entity's row receives gradient only when it appears. As a positive it is pulled towards the queries that bought it — a few noisy pulls. As a sampled negative it is pushed away from unrelated queries. With few or no positives, the row is dominated by the pushes and the initialiser, and its position carries no information the downstream task can use.
- Because the loss constrains only relative scores, any rotation of the whole space is an equally good solution. Two training runs land in different rotations, so a query tower from one run and a product table from another do not agree on anything. The two halves are one artifact.
The signal is a contrast
Nothing in the data says where a product should be. What the data says is that this query led to this product and, by sampling, that it did not lead to those. The loss turns that into a relative constraint on scores, and the geometry is whatever satisfies many such constraints at once. The choice of negatives is therefore part of the signal, not an implementation detail.
1import numpy as np2 3def contrastive_loss(q, p):4 # q: [B, d] query vectors from the tower; p: [B, d] product rows from the table5 # every other product in the batch is a negative for each query6 logits = q @ p.T # [B, B] scores7 logits -= logits.max(axis=1, keepdims=True) # numerical stability8 log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))9 return -np.mean(np.diag(log_probs)) # the matching pair should win its row10 11# gradient flows into the query tower AND into the B rows of the table that appeared.12# a product absent from every batch as a positive is only ever a column here —13# pushed away from queries it does not match, never pulled towards one it does.In-batch negatives make every batch a small retrieval problem, which is why the technique works, and they make popular products the most frequent negatives, which is a sampling policy the run has to record.
Two halves, one artifact
The retailer's outage was a versioning failure that looked like a model failure. The tower and the table are two halves of one function; the loss constrained only their agreement with each other, so each half alone is a set of numbers with no meaning. Exporting the table as "data" severed the dependency that the training run had created.
The query tower is promoted through the registry; the product vectors are exported to object storage with a date and loaded into the index by a separate job. Nothing records which table matches which tower.
The tower, the table, the negative-sampling config and the corpus snapshot form one registry entry. The index build records the entry's version; the serving path reads the version from the index and refuses to serve a tower against a table from a different run.
The two halves share a coordinate system that exists only because they were trained together. A version check on the pair is the only thing that can distinguish "compatible" from "same shape", and the shape is all a schema check sees (Feature and Model Versioning).
Rare rows and what to serve instead
The coverage problem cannot be trained away with the same data, because the data is the problem: a product nobody has bought has produced no positive pairs. The engineering decision is what to serve for such products, and the honest options are a content-derived vector in the same space or nothing.
The assumption that follows the model into production is that the served set is covered — by training pairs or by a fallback — and that assumption moves every day as the catalogue changes.
Each product vector in the index either received enough positive pairs in training for its row to be meaningful, or was replaced by a content-derived vector produced in the same coordinate system.
holds when The index build applies the coverage threshold from the run record and the fallback encoder is the one trained jointly with this table; the fraction of fallback rows is tracked.
breaks when A catalogue import adds many new products between retrainings; the fallback encoder is upgraded independently; the coverage threshold is lowered to make more products "searchable".
respond Raise coverage with a content fallback in the trained space, or exclude the rows; do not retrain more often on the same pairs, which reproduces the same gap.
How to build it
Most important first.
- Version the product table, the query tower and the negative-sampling configuration as one artifact in the registry, and make the index carry the artifact version so a query tower can refuse to serve against a table it was not trained with (What a Model Artifact Contains, Feature and Model Versioning).
- Handle rare entities explicitly: a coverage threshold below which the id embedding is replaced by, or blended with, a content-based vector from the product's text and attributes, so a new product has a position derived from what it is rather than from noise (Content-Based Recommendation).
- Choose negatives to match the serving problem — in-batch and hard negatives for retrieval over a large catalogue — and log the sampling policy with the run, because changing it changes the geometry as much as changing the data.
- Evaluate retrieval on held-out purchase pairs by recall at a cutoff over the full catalogue, not by the training loss, which depends on the negatives sampled (Evaluation Slices by product popularity, because the tail is where it fails).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Recall at a cutoff over the full catalogue on held-out query–purchase pairs, sliced by product popularity. The tail slice is the number that says whether the cold-start handling works; the head slice is the number that always looks fine.
- The fraction of served products below the coverage threshold, and the fraction of queries whose top results come from that set.
- Do not report the contrastive training loss as retrieval quality. Its value depends on the negatives, and easier negatives make it lower without making retrieval better.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The query tower serving traffic and the product table in the index came from the same training run, and the index records which run; a mismatch is detectable before it is served.
- Every product served from the index has either enough purchase pairs for its trained row to be meaningful or a content-derived vector in the same space.
- The negative-sampling policy and the corpus that produced the current geometry are recorded with the artifact, so a retraining can be compared with the previous one rather than just replacing it.
- Offline: recall at a cutoff on held-out pairs by popularity slice; a test that scores a fixed set of (query, product) pairs with the tower and the table and asserts the scores match those recorded at training time — the cheapest possible version-mismatch check.
- At deploy: the serving path compares the tower's artifact version against the index's and refuses to start on a mismatch (Serving Contract Tests).
- Over time: neighbour-overlap between the old and new table for a fixed anchor set at every retraining, and the online search metric during rollout (Embedding Drift).
What can go wrong
- Hard-negative mining pulls in false negatives — products the query would happily have matched but that were not purchased — and the model learns to push genuinely relevant products away.
- The content-based fallback for rare products is computed by a different encoder version than the trained table, and the blended vector lives in neither coordinate system.
- The index is rebuilt from the new table but a cache of query vectors from the old tower keeps serving for its TTL, so the mismatch reappears for a subset of traffic after the rollout looked complete.
- Versioning the table with the model means re-indexing the whole catalogue on every retraining, which is a large batch job with a cut-over problem of its own (Re-embedding).
- Content-based fallbacks need a second encoder, a blending rule and a definition of "rare", and they make the space less pure — a deliberate trade of consistency for coverage.
- Hard negatives improve quality and raise the false-negative risk; the mining policy becomes a hyperparameter that needs its own evaluation.
- "New products have vectors, so they are searchable." They have rows. A row with no positive pairs is positioned by pushes away from random queries and returns nothing useful; it must be treated as missing.
- "The vectors are 128-dimensional in both versions, so the index is compatible." The shape matches; the axes do not. Two runs share no coordinate system unless something explicitly aligned them.
- "The training loss went down after we changed the negatives, so retrieval improved." Easier negatives lower the loss. Recall over the full catalogue is the only number that can say whether retrieval improved.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALLookup tables as parameters, contrastive losses with sampled negatives, rotation freedom between runs and the rare-entity problem are shared by word vectors, recommender towers and text retrievers; only the definition of a positive pair changes.
- MODEL-SPECIFICTwo-tower models with an id table have the cold-start and rotation problems in full; a single pretrained text encoder used for both sides has no id table, no cold-start gap for new text, and still changes coordinate system when the encoder is retrained or swapped.
Where the depth lives
This domain teaches the model and hands the rest off by name.