Cosine Similarity
The dot product divided by the norms: the angle between two vectors, ignoring their length. Right when magnitude is noise, wrong when magnitude is signal — and at scale, nearest neighbours are an index problem, not a formula.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The team switched the similarity function from cosine to Euclidean distance and the neighbours changed. Which one is right, and what does each throw away?
A support team's ticket-routing system finds the most similar past tickets by embedding and recommends the team that resolved them. It works for typical tickets and routes long, detailed tickets badly. Someone noticed the long tickets have larger vectors and proposed switching to cosine; someone else pointed out that for the user-behaviour model next door, cosine made things worse.
Similarity is distance in the embedding space, so use Euclidean distance and take the closest. Cosine is the same thing, roughly, and libraries default to one or the other; the choice does not matter much.
Under Euclidean distance, a long ticket is far from every short ticket regardless of topic, because the length difference dominates. Its nearest neighbours are other long tickets, which share verbosity rather than a problem, and the routing follows the verbosity.
- Under Euclidean distance, a long ticket is far from every short ticket regardless of topic, because the length difference dominates. Its nearest neighbours are other long tickets, which share verbosity rather than a problem, and the routing follows the verbosity.
- Under cosine, the user-behaviour model lost the activity signal: a user with three interactions and a user with three thousand in the same proportions became identical, and the downstream model that relied on activity level as a feature degraded.
- "The choice does not matter much" was true on the short, typical tickets that dominated the evaluation set and false on the long ones that dominated the complaints (Evaluation Slices).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Rank past tickets by how likely their resolving team is the right one for the new ticket. The similarity function is a proxy for that; its target is that the top neighbours share the new ticket's correct routing.
- The decision is a single team suggestion per ticket; the similarity is only as good as the routing agreement of the neighbours it ranks first.
- Ticket text encoded by a text model into vectors whose norm grows with ticket length — a property of how the encoder pools tokens, not a property of the ticket's topic. Two tickets about the same problem have similar direction and different lengths.
- In the neighbouring user-behaviour model, vectors are built from interaction counts, and the norm encodes how active a user is — which for that model is signal.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The dot product
a·b = |a||b|cos θmixes two things: the angle between the vectors and the product of their lengths. Cosine similarity divides the lengths out and keeps onlycos θ, from −1 (opposite) through 0 (orthogonal) to 1 (same direction). It is unchanged if either vector is scaled by a positive constant. - Euclidean distance
|a − b|² = |a|² + |b|² − 2a·bdepends on both lengths and the angle. On vectors normalised to unit length it reduces to2 − 2cos θ, so the two agree exactly — and disagree whenever lengths differ. The question "cosine or Euclidean" is therefore "is the norm signal or noise for this use", not a question about the formulas. - For encoders whose norm tracks length, frequency or confidence rather than content, the norm is noise for topical similarity and cosine is right. For count-based or magnitude-meaningful vectors — activity, spend, a calibrated intensity — the norm is signal and cosine discards it. A dot product without normalisation keeps both and is what a two-tower retrieval model is typically trained to use, so the similarity used at serving time should be the one the loss was trained with (Embedding Training).
- Finding the top neighbours of one query among millions of vectors is a scan of the whole table unless there is an index, and approximate nearest-neighbour indexes trade exactness for speed in ways that depend on the metric; a cosine index on unnormalised vectors and a Euclidean index on normalised ones are different structures. That problem is a database problem, and the same geometry underlies k-Nearest Neighbours as a model.
Three formulas, one question
Written out, the three similarities differ only in what they do with the lengths. The dot product keeps them, cosine divides them out, Euclidean distance folds them in with the angle. Deciding between them is deciding whether length means something here.
1import numpy as np2 3def dot(a, b): return float(a @ b)4def cosine(a, b): return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))5def euclid(a, b): return float(np.linalg.norm(a - b))6 7short = np.array([1.0, 2.0, 0.5])8long_ = 4.0 * short # same direction, four times the length (a longer ticket)9other = np.array([2.0, 0.5, 1.0]) # a different topic, similar length to short10 11print(cosine(short, long_), cosine(short, other)) # 1.0 vs 0.55: cosine sees topic12print(euclid(short, long_), euclid(short, other)) # 6.9 vs 1.9: euclid sees length13print(dot(short, long_), dot(short, other)) # 21.0 vs 3.5: dot rewards length14 15# normalise both sides and all three agree on the ranking:16u = lambda v: v / np.linalg.norm(v)17print(euclid(u(short), u(long_)), euclid(u(short), u(other))) # 0.0 vs 0.95Under Euclidean distance the long ticket on the same topic is the *furthest* of the three. That is the routing failure in one line, and normalisation is the one-line fix — if, and only if, length is noise.
Choosing by what the norm encodes
The decision has a small number of cases and each has a reason. The two systems in the problem sit in different rows of this table, which is why the same change helped one and hurt the other.
What does the vector's length mean for this encoder and this use?
when Text encoders whose norm tracks length or frequency; any use where "about the same thing" is the question regardless of how much was said.
cost Magnitude is gone; if a later use needs it, keep the raw vectors and build a second index.
when Count- or intensity-based vectors where activity, spend or confidence is encoded in the norm and the downstream use needs it.
cost Entities with large norms dominate neighbour lists; a hub problem that needs its own handling.
when A two-tower model trained to score with a dot product; the serving similarity should be that dot product, normalised only if training normalised.
cost May force a dot-product index, which some index structures support less well than cosine or Euclidean.
Nearest neighbours at scale are an index
Once the metric is right, finding the top neighbours among millions of vectors is a data-structure problem. An exact scan is linear in the table size per query; the approximate indexes that make it fast partition or graph the space in ways that assume a metric, and they return neighbours that are usually, not always, the true ones.
This domain's interest stops at the boundary: the metric, the normalisation and the recall the downstream task needs. The structures themselves belong to the database and retrieval domains, and the same nearest-neighbour geometry is what k-Nearest Neighbours uses as a model.
The approximate index, queried with the chosen metric on vectors normalised as at build time, returns the neighbours the exact metric would, with a recall the task can absorb.
holds when The index was built for this metric; normalisation is on one code path for build and query; recall against an exact scan was measured at the serving cutoff on current data.
breaks when A library accepts the wrong metric silently; a refactor drops query-time normalisation; the table grows past the size the build parameters were tuned for; the encoder changes and the norm means something new.
respond Rebuild the index for the right metric or re-tune its parameters; fix normalisation on the serving path before touching the model or the encoder.
How to build it
Most important first.
- Choose the metric from what the norm encodes for this encoder and this use, and write the reason down next to the choice. If the norm is noise, normalise the vectors once at index time and at query time, and then cosine and Euclidean agree and either index type works.
- Match the serving-time similarity to the training-time scoring function. A model trained with a dot product and served with cosine is being used at a different operating point than it was optimised for.
- Evaluate neighbour quality on the downstream task by slice — here, by ticket length — because the metric choice fails on the slices where norms vary most.
- At scale, pick the index structure with the metric and test recall against an exact scan on a sample; an approximate index has a recall number and it is not one (Vector Search: Embeddings, Similarity and ANN, Vector Storage).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Routing agreement of the top neighbours with the correct team, sliced by ticket length. The long-ticket slice is the number the complaint is about; the overall number hides it.
- Recall of the approximate index against an exact scan for a sample of queries at the serving cutoff — the number that says whether the index is returning the neighbours the metric defines.
- Do not compare metrics by average similarity score. The scales are different and a higher average means nothing about ranking quality.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- What the norm encodes for the deployed encoder is what it encoded when the metric was chosen — noise for the ticket encoder, activity for the behaviour model — and an encoder change re-opens the decision.
- Normalisation, when used, is applied identically at index time and query time, on the same code path or with an equivalence test between them.
- The approximate index's recall against an exact scan at the serving cutoff has been measured on current data and is within the tolerance the downstream task can absorb.
- Offline: neighbour quality by slice under each candidate metric on a labelled set; a unit test that a scaled copy of a vector has cosine one and Euclidean distance proportional to the scale, run on the serving path.
- At index build: recall of the approximate index against an exact scan on a sample, per metric, recorded with the build.
- Over time: the norm distribution of indexed and query vectors, so a shift in what the norm means — a new encoder, a change in ticket length — shows before routing quality does.
What can go wrong
- Vectors are normalised at index time but not at query time after a refactor, so the query norm silently re-enters the ranking for every request.
- The approximate index was built for one metric and queried with another by a library that accepts either; results are returned without error and are wrong.
- The encoder is upgraded to one whose norm carries a different meaning, and the metric choice, written down for the old one, is now wrong for the new one (Embedding Drift).
- Normalising throws away magnitude permanently for that index; if a later use needs it, the raw vectors have to be kept and a second index built.
- Matching the serving metric to the training score is correct and may force a dot-product index, which is less natively supported by some index structures than cosine or Euclidean.
- Exact nearest-neighbour search is the honest baseline and does not scale; the approximate index is a quality trade whose recall must be measured and re-measured as the data grows.
- "Cosine is the right similarity for embeddings." It is right when the norm is noise. For count-based, intensity-based or dot-product-trained vectors, it discards the signal the model was built on.
- "Cosine and Euclidean give the same neighbours." Only on normalised vectors. Anywhere norms vary, they rank differently, and the difference is concentrated on the entities whose norms are unusual.
- "We use an approximate index, so nearest-neighbour search is solved." An approximate index has a recall against exact search that depends on the metric, the build parameters and the data; it is a number to measure, not a solved problem.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe relation between dot product, cosine and Euclidean distance is algebra and holds for any vectors; which of them is appropriate is a property of what the norm encodes, which varies by encoder and by use.
- DATA-SPECIFICFor text encoders whose norm tracks length or token frequency, cosine is usually right; for interaction-count vectors and for models trained with a raw dot product, the norm is signal and cosine is a loss of information.
Where the depth lives
This domain teaches the model and hands the rest off by name.