Self-Supervised Learning
The data labels itself: hide part of it and predict it back. The signal is free and abundant, which is why it works — and why the model learns whatever the corpus contains, including what you did not want.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
If the model is trained to reconstruct its own input, what has it learned, and why would that help with a task the pretext never mentioned?
A marketplace has two hundred million product listings, a few thousand labelled for "counterfeit", and a search team that says the listing text is too varied for hand features. They want a representation that works for counterfeit detection, search and recommendation without labelling each one.
Pretrain an encoder on all two hundred million listings with a masked-token objective, freeze it, and train a small classifier on the few thousand counterfeit labels. The representation captures "what a listing means" and every downstream task inherits it.
The encoder learned that a certain seller boilerplate predicts the surrounding tokens very well, and gives it a large share of the representation. Downstream, the counterfeit head keys on boilerplate, which is a seller feature, not a counterfeit feature.
- The encoder learned that a certain seller boilerplate predicts the surrounding tokens very well, and gives it a large share of the representation. Downstream, the counterfeit head keys on boilerplate, which is a seller feature, not a counterfeit feature.
- Duplicated listings — the same product posted ten thousand times — dominate the pretraining loss. The representation is excellent at those and coarse everywhere else.
- A pretext that predicts masked tokens learns tokens; it is not told that price plausibility or image-text mismatch matters for counterfeits, and it captures those only incidentally.
- The downstream validation on takedown-notice labels looks strong because the encoder has seen those brand names millions of times. Counterfeits of unlitigated brands score as clean.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- There is no downstream target during pretraining. The pretext target is derived from the input: a masked token to predict, the next token, or whether two views come from the same listing. The label is manufactured, not observed.
- The real targets — counterfeit, relevance, click — come later, on top of the learned representation, each with its own supervised head and its own small labelled set (Transfer Learning, Fine-Tuning).
- One pretraining example is one listing with part of it hidden: fifteen percent of tokens masked, or the title and the description as two views of the same item. Every listing yields many examples for free.
- The corpus is whatever is in the catalogue: duplicates, seller boilerplate, machine-translated descriptions, and the counterfeits themselves in unknown proportion.
- The labelled counterfeit set was collected from takedown notices, so it covers the brands whose lawyers are most active (Selection Bias).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A self-supervised objective is a supervised loss whose target is computed from the input:
L(f(x_masked), x_hidden). The gradient is as informative as the pretext is hard. Predicting a masked token requires modelling syntax, co-occurrence and long-range structure; predicting whether two crops match requires invariance to the crop. What the representation encodes is whatever the pretext rewards. - Contrastive variants pull representations of two views of the same item together and push different items apart. The choice of "view" is the label: it says which differences the model should ignore. A view that hides the seller name teaches seller-invariance; one that does not teaches seller-identity.
- Because the signal is derived from the corpus, the representation inherits the corpus's frequencies. Common patterns are encoded finely; rare ones coarsely. Nothing in the loss knows which patterns the downstream task will need.
The label is manufactured from the input
A masked-token objective hides part of the input and asks the model to predict it. The target exists for every listing without an annotator, which is why the corpus can be the whole catalogue. The gradient is a supervised gradient; only the source of the target has changed.
A contrastive objective manufactures its label differently: two views of the same listing are a positive pair, views of different listings are negatives. The construction of a view is a decision about what the representation should ignore.
1def make_views(listing, drop_seller=True):2 # the view defines the invariance: whatever both views share is what the3 # representation is allowed to depend on4 a = augment(listing, drop=["seller_name", "seller_bio"] if drop_seller else [])5 b = augment(listing, drop=["seller_name", "seller_bio"] if drop_seller else [])6 return a, b7 8def info_nce(z_a, z_b, tau=0.07):9 # z_*: (n, d) L2-normalised embeddings; row i of z_a pairs with row i of z_b10 logits = (z_a @ z_b.T) / tau # (n, n): all pairs in the batch11 targets = torch.arange(len(z_a)) # the positive is on the diagonal12 return cross_entropy(logits, targets) # pull the pair together, push the rest apartThe loss is unremarkable. make_views is the modelling decision: with drop_seller=False the cheapest way to match the pair is to read the seller name, and the representation becomes a seller detector.
The representation inherits the corpus
Whatever appears often in the corpus is encoded finely, because the pretext rewards it often. Boilerplate, duplicated listings and the dominant category are what the encoder is best at. Rare patterns — the unlitigated brand, the unusual product — get the leftover capacity.
The downstream evaluation on takedown-notice labels cannot see this, because those labels are drawn from the well-represented brands. The gap opens on the slice the labels never covered.
Strong recall on the held-out takedown-notice set; a linear probe on the frozen encoder alone gets most of the way there, which was taken as evidence the representation is good.
Manual review finds that flags cluster on a few litigated brands; counterfeits of smaller brands pass; several flags are genuine listings by one seller whose boilerplate resembles a known counterfeiter's.
- 1The encoder allocated representation capacity to seller boilerplate and the most-duplicated brands, which the labelled set happened to be about.
- 2Seller fields were left in the views, so the representation carries seller identity and the head learned a seller blacklist.
- 3The takedown labels are a biased sample of counterfeits, and the offline metric is a fact about that sample.
Every head depends on the encoder staying put
Once search, recommendation and counterfeit detection all sit on one encoder, that encoder is shared infrastructure. A change to it — a new corpus snapshot, a new pretext — moves the representation under every head at once.
The assumption each head makes is that the geometry it was trained against is the geometry it is served against. That is a versioning and monitoring problem, not a modelling one.
Each downstream head is served with embeddings from the exact encoder version it was trained against, and that encoder's representation still matches live listings.
holds when Encoder version is part of every head's artifact and checked at load; embedding distribution on live listings stays close to the pretraining snapshot.
breaks when The encoder is retrained and heads are not; the catalogue shifts toward categories the corpus under-represented; counterfeiters adapt their text to the representation.
respond Treat an encoder change as a migration: re-evaluate every head on its gold set, re-run the probes, and roll heads and encoder together.
How to build it
Most important first.
- Design the pretext around what the downstream tasks must be invariant to and sensitive to. If counterfeit detection must not depend on seller identity, mask or drop the seller fields when constructing views, and check the representation cannot predict seller (Model Invariant Tests).
- Deduplicate and cap per-seller contribution to the pretraining corpus, so the representation is not an encoding of ten thousand copies of the same listing (Dataset Construction).
- Evaluate the representation on a probe for each downstream task with a small gold set sampled from the population that task will serve — not the takedown-notice set — before spending on fine-tuning (Evaluation Slices).
- Keep the pretrained encoder versioned as an artifact with the corpus snapshot and pretext definition, because every downstream model inherits a change to it (Model Lineage, Embedding Drift).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Downstream metric on a gold set sampled from the serving population, per brand tier — that is what the counterfeit decision depends on. Litigated-brand recall is a different number and will look better.
- Probe accuracy for things the representation should not know: can a linear probe recover seller id from the embedding? If yes, the downstream head can too.
- The pretraining loss itself is a diagnostic for whether training ran, not a quality metric. A lower masked-token loss on a deduplicated corpus does not imply better counterfeit detection.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The pretraining corpus is distributed like the listings the downstream heads will score, and per-seller and duplicate caps keep it from being dominated by a few sources.
- The representation is invariant to the things the downstream tasks must not depend on — seller identity, brand litigation history — as measured by a probe on each new encoder version.
- Every downstream head is pinned to a specific encoder version, and a new encoder triggers re-evaluation of every head before it is served.
- Offline: linear and shallow non-linear probes for forbidden attributes; downstream gold-set metrics per slice; nearest-neighbour inspection of a sample of embeddings to see what the model thinks "similar" means (Embeddings).
- Online: shadow the counterfeit head on live listings and compare flags with the manual review team's decisions per brand tier before any takedown is automated.
- Over time: re-run the probes and the downstream gold evaluation on every encoder version; monitor embedding distribution drift between the pretraining snapshot and live listings.
What can go wrong
- Seller invariance is enforced in the view construction, and the model recovers seller identity from writing style anyway; the probe was linear and the leak is not.
- The encoder is retrained on a fresh corpus snapshot and every downstream head silently degrades, because the representation moved under them (Embedding Drift).
- Counterfeiters copy the exact text of genuine listings. The representation, being about text, places them on top of the genuine item (Adversarial Inputs).
- Pretraining is the most expensive training run the team will do, and its quality metric is indirect; the budget is spent before anyone knows whether the counterfeit head improved (Training Cost).
- Enforcing invariances in the pretext removes information that some other downstream task might have wanted, so the encoder serves several tasks slightly worse than a specialist for each.
- Versioning the encoder with every head pinned to it means a representation improvement is a migration across every consumer, not a swap.
- "The model learned what a listing means." It learned what predicts masked tokens in this corpus. That overlaps with meaning and also with boilerplate, duplication and seller style.
- "Self-supervised learning removes the need for labels." It removes the need for labels to learn a representation. Every decision built on it still needs labelled evaluation on the population it serves.
- "Lower pretraining loss means a better encoder." It means the pretext is better fitted. On a corpus with many duplicates that can mean the encoder memorised them.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a self-supervised representation encodes what the pretext rewards at the frequencies the corpus contains holds for masked-token, next-token and contrastive objectives on text, images and audio alike.
- SCALE-SPECIFICA team with two hundred million listings can pretrain its own encoder; a team with a few hundred thousand is usually better served by adapting a public pretrained encoder and spending its effort on the probe and gold-set evaluation, because the corpus is too small for the pretext to learn much the public model does not already know (Foundation Models).
- CONTESTEDA serious position holds that domain-specific pretraining is rarely worth it now that public encoders are large, and that the marginal gain from in-domain pretext training is smaller than the gain from a clean gold set and a well-designed head. The counter is that marketplace text — SKUs, seller conventions, category jargon — is far from public web text, and that invariances such as seller-blindness can only be designed into a pretext you control.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a probe that asserts the representation cannot recover a forbidden attribute is an invariant test on an artifact, and it belongs in the promotion pipeline of every encoder version.