Dimensionality Reduction
PCA keeps variance; UMAP and t-SNE keep neighbourhoods, approximately. Neither keeps meaning, and a 2D picture of a 300-dimensional space is a drawing, not a map.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
You projected a high-dimensional space to two dimensions and it looks structured. What did the projection keep, what did it throw away, and which of the things you see are artefacts?
A search team has product embeddings and a stakeholder who says: "Show me the embedding space. If similar products are near each other, we can trust it for recommendations."
Run UMAP to two dimensions, colour by category, and look. If the colours form islands the embedding is good; if two islands are close the categories are related; if a point sits in the wrong island the embedding is wrong there.
UMAP and t-SNE produce islands from nearly anything, including noise. Island separation is a property of the algorithm's parameters — perplexity, number of neighbours, minimum distance — as much as of the data.
- UMAP and t-SNE produce islands from nearly anything, including noise. Island separation is a property of the algorithm's parameters — perplexity, number of neighbours, minimum distance — as much as of the data.
- Distances between islands mean little: both algorithms preserve local neighbourhoods and are free to place clusters anywhere. Two categories that look adjacent may be far apart in the full space.
- The nearest neighbours in the plot are not the nearest neighbours in the full space. The retrieval system uses the full space, so the plot cannot show what it will retrieve (Cosine Similarity).
- The stakeholder trusts the embedding on the strength of the picture, and the recommendation system built on it surfaces neighbours the picture never showed.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- There is no target. PCA finds the directions of maximum variance; t-SNE and UMAP find a low-dimensional layout whose local neighbourhoods resemble the high-dimensional ones, under a specific and lossy definition of resemblance.
- The decision is whether to trust the embedding for nearest-neighbour retrieval — a decision about the full space — and the picture is a decision aid for a space it cannot show (Embedding Projection Caveats).
- One example is one product's embedding vector, produced by a model trained for a different objective, plus its category label for colouring the plot.
- The vectors are high-dimensional and roughly isotropic; most pairwise distances are similar, and the interesting structure is in a small fraction of the variance.
- The category labels used for colouring are the catalogue taxonomy, which was not what the embedding was trained on.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- PCA rotates the space so the first axes carry the most variance and keeps the first few. It is linear, deterministic, and preserves global distances as well as any projection of that rank can. What it throws away is everything in the dropped axes — which is where the low-variance, high-meaning directions often live. PCA keeps variance, not meaning.
- t-SNE and UMAP build a neighbourhood graph in the high-dimensional space and lay it out in two dimensions so that neighbours stay near. The objective cares about local structure and is nearly indifferent to distances between distant groups; cluster sizes, inter-cluster distances and the overall shape are largely artefacts of the layout.
- In high dimensions, distances concentrate: most points are roughly equidistant. A 2D layout must break that symmetry to draw anything, so it manufactures contrast. The picture therefore shows structure the space has and structure the layout invented, with no visual difference between them.
Variance is not meaning
PCA is a rotation followed by truncation. The first component is the direction along which the data varies most, the second the most-varying direction orthogonal to it, and so on. Keeping the first few keeps the most variance any linear projection of that rank could keep. It is optimal for exactly that criterion and for nothing else.
If the signal the downstream task needs is a low-variance direction — a subtle distinction between two product lines that differ in one attribute — PCA discards it first. The retained-variance number will be high and the downstream metric will drop.
1import numpy as np2 3def fit_pca(X_train, rank):4 mu = X_train.mean(0)5 _, S, Vt = np.linalg.svd(X_train - mu, full_matrices=False)6 explained = (S[:rank] ** 2).sum() / (S ** 2).sum() # variance kept, not meaning kept7 return {"mu": mu, "W": Vt[:rank].T, "explained": float(explained)}8 9def apply_pca(pca, X):10 return (X - pca["mu"]) @ pca["W"] # the same rotation, every time11 12pca = fit_pca(X_train_embeddings, rank=32) # fitted once, on the training fold only13Z_train = apply_pca(pca, X_train_embeddings)14Z_valid = apply_pca(pca, X_valid_embeddings) # never refit on validation or in serving15# the check that matters is downstream: task_metric(Z_valid) per rank, not pca["explained"]Two things to notice. explained is a property of the inputs and says nothing about the task. And pca is part of the artifact: refitting it on serving data rotates the coordinate system under the model that consumes Z.
The picture invents contrast
t-SNE and UMAP take a neighbourhood graph and lay it out so that neighbours stay near. That is a local objective. It has no reason to keep the distance between two far-apart groups, the relative size of groups, or the overall shape, and in practice it does not. The layout is a plausible drawing of the local structure, embedded in an invented global arrangement.
Because high-dimensional distances concentrate, the drawing has to create contrast to show anything. Islands appear. Some correspond to dense regions in the full space; some are artefacts of the parameters. The plot looks the same either way.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Two islands appear close together | The categories are declared related | Inter-cluster distance is not preserved by the objective; placement is largely arbitrary | Measure the distance between category centroids in the full space instead |
| One island is much larger than another | The category is declared more diverse | Cluster size in the layout depends on local density parameters, not on full-space spread | Compare within-category distance distributions in the full space |
| A point sits inside the wrong island | The embedding is declared wrong for that item | Its 2D neighbours may not be its full-space neighbours; the projection can misplace a point with no full-space error | Retrieve its full-space nearest neighbours and look at those |
| Clean islands on the first run | The embedding is signed off | Neighbour embeddings produce islands from near-noise at typical parameter settings | Rerun under several seeds and neighbour counts; compute projection-versus-full-space neighbour overlap |
The picture is never the retrieval system
Recommendations are served from the full space: nearest neighbours by cosine or dot product over the original vectors. The plot shows a different neighbourhood structure, and the overlap between the two is the only number that connects them. Where it is low, the plot is misleading about exactly the thing the stakeholder wanted to check.
So the assumption is not about the picture; it is about the space. Trusting the embedding for retrieval is an assumption checked with a retrieval evaluation, and the plot is at most a way to spot gross breakage.
For the points a decision is being made about, the neighbours shown in the projection overlap substantially with the full-space neighbours the retrieval system will use.
holds when A stratified neighbour-overlap check has been run for this encoder version and this projection, and the overlap is attached to the plot; PCA components in serving match the training artifact by hash.
breaks when The encoder is updated and the old plot is reused; the projection parameters are tuned until the picture looks clean; PCA is refitted in a serving batch job.
respond Retire the plot with the encoder version it was made for. If PCA rotated, restore the frozen components and treat the refit as a new model requiring re-evaluation downstream.
How to build it
Most important first.
- Decide what question the reduction is for. For compression before a downstream model, PCA with a variance target and a check on downstream metrics. For visual inspection, a neighbour-embedding with the caveat attached to the plot. For retrieval, no reduction at all — evaluate in the full space (Embeddings).
- Validate the picture against the space: sample points, compare their k nearest neighbours in the projection with their k nearest neighbours in the full space, and report the overlap. That number is what the plot is worth.
- Vary the layout parameters and the seed and show more than one plot. Structure that survives is more likely real; islands that move or merge are artefacts.
- When PCA is used as a preprocessing step, fit it on the training fold only and treat the components as part of the artifact, because a re-fit on new data rotates the space under the downstream model (Preprocessing Leakage, Preprocessing Lives in the Artifact).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Neighbour-overlap between projection and full space at small k, on a sample. That is the number that says whether the plot describes what retrieval will do.
- For PCA as compression: the downstream task metric at each retained rank, not the retained variance. Variance retained is a fact about the input; the task decides what mattered.
- Island separation, cluster tightness and inter-cluster distance in a t-SNE or UMAP plot are not measurements of anything and should not appear in a report as evidence.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The projection used for any decision has a measured neighbour-overlap with the full space on the population it is shown for, and that measurement is attached to the picture.
- PCA components used as a preprocessing step are frozen in the artifact and applied unchanged at serving time; a refit is a new model.
- The embedding the projection describes is the version in production; a new encoder produces a new space and a new plot (Embedding Drift).
- Offline: neighbour-overlap between projected and full-space k-NN on stratified samples per category; downstream metric versus retained PCA rank; the same plot under several seeds and parameter settings.
- Online: nothing about the plot can be verified online. Verify the retrieval system in the full space with an interleaving or A/B test, and never on the strength of the picture.
- Over time: re-run the overlap check on every encoder version and assert the PCA components in the serving artifact match the training ones by hash.
What can go wrong
- PCA is re-fitted on each batch of new embeddings, so the components rotate and the downstream model receives a different coordinate system every week without any schema change.
- A plot with a caveat in the footnote is screenshotted into a deck without the footnote, and the picture becomes the argument.
- The neighbour-overlap check is run on a random sample that misses the sparse categories, which are exactly where the projection is worst.
- Attaching an overlap number and several seeds to every plot makes the presentation slower and less persuasive, which is the point.
- Refusing to reduce for retrieval keeps the full-space cost — memory and compute for nearest-neighbour search — that the reduction would have saved (Vector Search: Embeddings, Similarity and ANN).
- Fitting PCA on the training fold only and freezing it means the compression is stale relative to new data, and a refit is a migration.
- "The categories form clean islands, so the embedding is good." UMAP forms islands from noise. The islands say the layout ran; the overlap check says whether they reflect the space.
- "These two clusters are close in the plot, so the categories are related." Inter-cluster distance in t-SNE and UMAP is mostly an artefact of the layout. It carries almost no information about the full space.
- "PCA kept ninety-five percent of the variance, so almost nothing was lost." It kept ninety-five percent of the variance. The task may have lived in the other five.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat PCA preserves variance rather than meaning, that neighbour embeddings preserve local structure and manufacture the rest, and that projected neighbours differ from full-space neighbours, hold for any high-dimensional data, not only learned embeddings.
- SIMULATEDThe embedding explorer this lesson links to computes neighbour disagreement between a full space and its 2D projection on synthetic vectors from a seeded generator; the disagreement rates it shows are for the shape of the argument and are not measurements of any real embedding.
- CONTESTEDA serious position holds that neighbour embeddings are indispensable and that the caveats are overstated: an experienced analyst reads a UMAP plot with the artefacts in mind, discovers real structure — duplicate clusters, mislabelled regions, a broken encoder — and no overlap number replaces that. The counter is that the same plot persuades the inexperienced reader of things that are not there, and the discipline is to ship the overlap number with the picture, not to withhold the picture.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Data visualisation — a plot that carries its own uncertainty and its own caveat is a visual-communication discipline; this lesson assumes the reader will attach the overlap number to the picture rather than the footnote.