Embeddings
A discrete entity — word, user, product, document — becomes a dense vector that is a parameter of some model, learned on a proxy task. The geometry encodes what that task rewarded, not "meaning".
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Two products sit close together in embedding space and a stakeholder asks whether that means they are "similar". Similar according to what, and who decided?
A marketplace team built a "similar items" module from product embeddings and it works well in a demo. Merchandising complains that a premium espresso machine's neighbours are cheap kettles and a phone case, and asks the team to "fix the similarity". The team is not sure what the vectors actually represent.
Train an embedding model on the interaction data, take the vectors, and find nearest neighbours by cosine similarity. Embeddings capture semantics — that is what they are for — so the nearest neighbours are the semantically similar items.
The neighbours are items viewed in the same sessions as the espresso machine. People who look at a premium machine also look at kettles and, on the way out, a phone case in the "you might also like" strip that the previous recommender put there. The geometry faithfully encodes co-browsing, including the previous recommender's influence (Feedback Loops).
- The neighbours are items viewed in the same sessions as the espresso machine. People who look at a premium machine also look at kettles and, on the way out, a phone case in the "you might also like" strip that the previous recommender put there. The geometry faithfully encodes co-browsing, including the previous recommender's influence (Feedback Loops).
- The demo used popular items, whose vectors are trained on thousands of pairs. Long-tail items have vectors that barely moved from initialisation, so their neighbours are effectively random, and the tail is most of the catalogue (Cold Start).
- "Fix the similarity" has no target to fix towards. The model did what it was trained to do; the complaint is that the proxy task is not the product question, and no amount of retraining on the same task changes that.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The embedding itself has no target. It is a by-product of a model trained on a proxy task — here, predicting which item a user views next in a session — and its target is that task's label: the next item viewed.
- The similarity module's implicit target is "a buyer of A would consider B", which is a different thing from "A and B are viewed in the same sessions", and nothing in training optimised for it.
- Session logs: sequences of product views. One training example is a pair (item viewed, item viewed next) with a set of sampled non-next items as negatives. Items that never co-occur in sessions with anything get a random, untrained vector.
- The corpus is browsing, not buying. Comparison shopping, distraction and accidental clicks are all co-occurrence; the embedding does not know the difference because the label did not encode it.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An embedding layer is a lookup table: a matrix with one row per entity id and one column per dimension. Looking up an entity returns its row; the row is a parameter vector and the gradient of the loss with respect to it is computed and applied like any other weight. Entities that appear in a training example get their rows updated; entities that do not appear keep whatever the initialiser gave them.
- The geometry is a consequence of the loss. A model that scores a pair by the dot product of two rows and is trained to score observed pairs higher than sampled ones will move co-occurring entities together and non-co-occurring ones apart. "Close" means "the proxy task treated these alike", and every regularity the plot shows — clusters, directions, analogies — is a regularity of the training pairs.
- The same mechanism trains the token embeddings of a language model, where the proxy task is next-token prediction, and the user and item towers of a recommender, where it is next interaction (Collaborative Filtering, Self-Supervised Learning). What differs is the proxy task, and therefore what the geometry means.
- Agentic Engineering uses embeddings as a retrieval primitive, taking a pretrained encoder as given and asking how to index and query its outputs. This lesson is about where that encoder's geometry came from and what it can and cannot be expected to encode; the two views meet at the vector index (Vector Storage, Cosine Similarity).
A row in a table
Strip away the word "embedding" and what remains is a matrix. Each entity has an integer id, the id indexes a row, and the row is a vector of learned parameters. Nothing about the row is fixed by the entity; it is fixed by the gradients that flowed into it, and those came from the examples the entity appeared in and the loss those examples produced.
That is why an entity that never appears in training has a vector that is pure initialisation noise, why an entity that appears only with one other entity sits wherever the gradient from that single relation pushed it, and why the popular entities — the ones in every demo — look so well behaved.
1import numpy as np2 3n_items, dim = 200_000, 644table = np.random.normal(0, 0.1, size=(n_items, dim)) # every row starts as noise5pairs_seen = np.zeros(n_items, dtype=int)6 7def score(a, b):8 return table[a] @ table[b] # dot product of two rows9 10def sgd_pair(a, pos, negs, lr=0.05):11 # push the observed pair together, the sampled negatives apart12 for b, y in [(pos, 1.0), *[(n, 0.0) for n in negs]]:13 p = 1 / (1 + np.exp(-score(a, b)))14 g = (p - y) # d(log loss)/d(score)15 table[a] -= lr * g * table[b] # only rows a and b move16 table[b] -= lr * g * table[a]17 pairs_seen[a] += 118 pairs_seen[pos] += 119 20# after training: rows with pairs_seen == 0 are exactly their initial noiseOnly the rows that appear in an example receive gradient. "Learned representation" means this table after enough of these updates, and pairs_seen is the honest measure of how learned each row is.
What "close" meant to the loss
The offline picture was consistent and wrong in the way that matters. The proxy task improved, the demo looked convincing, and the product question — what a buyer would consider — was never measured, because the number that was available was the loss the model optimised.
Next-view prediction loss on held-out sessions improved over the previous model; a hand-picked demo of popular anchors showed plausible neighbours.
Merchandising rejects the neighbours of premium items; long-tail anchors return apparently random lists; the module's click-through is no better than the popularity strip it replaced.
- 1The proxy task rewards co-viewing, and premium items are co-viewed with cheap comparison items and with whatever the previous recommender displayed alongside them.
- 2Long-tail items received too few training pairs for their rows to leave initialisation; their neighbour lists are noise that the demo never sampled.
- 3Hub items with very many co-occurrences sit near everything and crowd out specific neighbours.
The relation must stay the right one
An embedding table is trained once for one relation and then reused wherever a vector is convenient. The assumption that travels with it — silently — is that the relation it encodes is the one each downstream consumer needs. The table cannot enforce it, and the consumers rarely know what it was.
The relation the embedding was trained to encode is the relation the downstream use requires, and the entities it is used for are well covered by training pairs.
holds when The training signal was chosen from the product question — purchase pairs for "would consider", textual context for "about the same thing" — and the judged relevance set confirms it; served entities are above the coverage threshold.
breaks when The table is reused for a new question; the interaction data shifts towards what a recommender shows rather than what users seek; the catalogue grows and the low-coverage fraction rises; a hub item forms.
respond Change the training signal or the serving policy. Retraining on the same pairs reproduces the same relation; document the relation with the table so the next consumer can check it.
How to build it
Most important first.
- Name the proxy task in the documentation of every embedding table, and state what "close" means in its terms — "viewed in the same session", "purchased by the same users", "appear in similar text" — so downstream users can judge whether that is the relation they need.
- If the product question is different from the proxy task, change the training signal: pairs from purchases or add-to-cart rather than views, or a contrastive objective built from the relation the product needs (Embedding Training, The Learning Signal).
- Treat coverage as a first-class property. Count how many training pairs each entity received and refuse to serve neighbours for entities below a threshold, falling back to content features or popularity (Content-Based Recommendation).
- Evaluate the neighbours on the product question, with a labelled set of "would a buyer of A consider B" judgements, not on the proxy task's loss (Business Metrics vs Model Metrics).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- A judged relevance set for the similarity module: for a sample of anchor items, which neighbours a merchandiser accepts. This is the number the complaint is about; the proxy-task loss is the number the model optimised and they are not the same.
- Training-pair count per entity, as a distribution. The fraction of the catalogue below the coverage threshold is the fraction whose vectors are noise.
- Do not report "embedding quality" as the proxy-task validation loss. It measures how well the model predicts next views, which is what it was trained for and not what it is used for.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The relation the proxy task rewarded — co-viewing, co-purchase, textual context — is the relation the downstream use needs, and this has been checked on a judged set rather than assumed from the word "similar".
- Every entity served has enough training pairs for its vector to have moved meaningfully from initialisation; entities below the coverage threshold are handled by a fallback.
- The interaction data the embedding was trained on was not shaped so strongly by a previous model that the geometry encodes that model's policy rather than user behaviour.
- Offline: score the similarity module on the judged relevance set, and separately on the proxy-task loss; if the second improves and the first does not, the proxy task is the wrong one.
- Online: an interleaving or A/B comparison of the similarity module against the previous one on the product metric (A/B Testing Models).
- Over time: track training-pair coverage per entity as the catalogue grows, and the hubness of the neighbour lists — the number of distinct items that appear as neighbours, which collapses when hubs form.
What can go wrong
- The previous recommender shaped the sessions the embedding is trained on, so the geometry encodes that recommender's choices; the new module reproduces them and the loop closes (Feedback Loops).
- A popular item co-occurs with everything and becomes close to everything — a hub — and appears in the neighbour list of half the catalogue for reasons that have nothing to do with the anchor.
- The embedding table is exported and reused by another team for a different question, where the proxy task's notion of closeness is actively misleading, and nobody documented what it was.
- A proxy task closer to the product question — purchases instead of views — has far fewer examples, so the geometry is better defined and covers fewer entities.
- Refusing to serve neighbours for low-coverage entities is honest and leaves most of a long-tail catalogue without the feature; the fallback has to be designed, not just declared.
- A judged relevance set costs merchandiser time and goes stale as the catalogue changes; without it, the only available number is the wrong one.
- "The embedding captures the meaning of the item." It captures the item's role in the proxy task. For a next-view task, "meaning" is "what people look at around it", which includes what the last recommender showed them.
- "These two items are close, so they are similar." They are close under the training relation. Whether that relation is the one the stakeholder means is the question, and the geometry cannot answer it.
- "Retrain the embedding and the bad neighbours will go away." Retraining on the same task reproduces the same relation with fresh noise. Change the signal, or change what is served.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat an embedding table is a parameter matrix trained on a proxy task holds for word vectors, recommender towers, token embeddings in a language model and document encoders alike; the proxy task differs and so does what "close" means.
- TASK-SPECIFICRecommender embeddings trained on interactions inherit the previous policy's influence and cold-start gaps; text embeddings trained on large corpora are more stable across entities but encode textual context, which is not topical relevance either.
- DOMAIN-SPECIFICOn a marketplace with a long-tail catalogue the coverage problem dominates; on a vocabulary of common words nearly every entity is well covered and the proxy-task mismatch is the main hazard instead.
Where the depth lives
This domain teaches the model and hands the rest off by name.