Transfer Learning
Pretrained model + task data → adapted model. The early layers carry structure that transfers; the late layers carry the old task. Freeze what transfers, train what does not, and expect the labelled-data requirement — and the split arithmetic — to change.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
You have a model pretrained on a large general dataset and a few thousand labelled examples of your task. What transfers, what must be relearned, and when is starting from scratch the better choice?
A dermatology clinic wants a triage model that flags lesion photos for a specialist review. They have three thousand labelled photos. A general image model trained on millions of everyday photographs is available. Someone has asked whether three thousand is enough, and someone else whether a model that learned to recognise dogs has anything to say about skin.
Three thousand images is too few to train a deep network, so either collect a hundred thousand or use a small classical model on hand-crafted colour and shape features. The pretrained model is about dogs and cars; it is irrelevant.
The classical features underfit: the texture and border patterns that drive escalation are exactly the things hand-crafted features describe poorly and a convolutional stack describes well (CNN Concepts). The offline number is mediocre and honest.
- The classical features underfit: the texture and border patterns that drive escalation are exactly the things hand-crafted features describe poorly and a convolutional stack describes well (CNN Concepts). The offline number is mediocre and honest.
- Training a deep network from scratch on three thousand images memorises them; validation collapses (Overfitting). The offline number is excellent on a random split because photos of the same patient landed on both sides, and useless on a patient-held-out split.
- Collecting a hundred thousand labelled lesion photos is years of specialist time. The clinic does not have it, and the option was never real.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a lesion photo should be escalated to a specialist; the label is the specialist's decision on historical cases; the action is queue ordering, not diagnosis (Human Oversight).
- A false negative is a delayed specialist review; a false positive is specialist time on a benign case. The threshold, not the model, sets the trade (Thresholding).
- Three thousand photos, each one lesion, labelled escalate/not by the specialist who saw the case, with the class balance around one in five escalated.
- The pretrained model saw millions of natural photographs with a thousand-class objective; no skin images to speak of, but edges, textures, colour gradients and shapes in abundance.
- Photos come from three clinic cameras with different colour profiles; the camera is recorded per photo (Group Split).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A deep network's early layers learn features that are general to the modality — edges, colour blobs, textures — and the late layers learn features specific to the training task. The general layers transfer across tasks because the low-level statistics of natural images are shared; the task-specific layers do not, and the final classification head is entirely about the old thousand classes.
- Transfer learning reuses the pretrained weights as the starting point and adapts them with the task data. At one extreme, freeze everything and train a new head on the fixed representation — cheap, safe, limited. At the other, unfreeze everything and train with a small learning rate — most expressive, most able to forget. In between, unfreeze the last few blocks. The right point depends on how far the task's images are from the pretraining images and how much labelled data there is.
- Because most of the parameters start near a good solution, the task data has to move them only a little, so far fewer labelled examples are needed than from scratch. That same fact is the risk: a large learning rate on the pretrained layers destroys the transferred structure in a few steps — catastrophic forgetting — and the model is now a randomly initialised network with three thousand images (Fine-Tuning).
What transfers and what does not
Visualise the pretrained network as a stack. The bottom layers respond to edges, colour gradients and simple textures; the middle to parts and patterns; the top to the thousand things the old task cared about; the head to the thousand class scores. A lesion photo is made of the bottom and middle. The top and the head are about the old task and are where adaptation has to happen.
That is the whole mechanism of transfer: reuse the layers whose job is the same in the new task, and retrain the ones whose job is different. The engineering decisions — which layers, what learning rate, how much data — all follow from asking where in the stack the two tasks diverge.
1# stage 1: frozen backbone, train only the head2for p in backbone.parameters():3 p.requires_grad = False4opt = Adam(head.parameters(), lr=1e-3)5# ... train until patient-held-out recall stops improving6 7# stage 2: unfreeze the last two blocks, ten times smaller LR for them8for p in backbone.blocks[-2:].parameters():9 p.requires_grad = True10opt = Adam([11 {"params": head.parameters(), "lr": 1e-3},12 {"params": backbone.blocks[-2:].parameters(), "lr": 1e-4},13])14# the pretrained weights are a good starting point; a large step destroys15# them in a few updates — that is catastrophic forgettingThe two learning rates are the lesson. The head is random and can take large steps; the transferred blocks are already close to useful and should be nudged. One learning rate for everything is the most common way to turn a pretrained network into a randomly initialised one.
Why the split arithmetic changes
With a million examples, holding out ten percent for validation leaves a hundred thousand to estimate a metric — the interval is narrow and the training set barely notices. With three thousand, ten percent is three hundred; at one-in-five escalation that is sixty positive cases, and recall on sixty cases has an interval wide enough to swallow the difference between two models.
Transfer learning makes the small dataset usable for training. It does nothing for evaluation — the metric is still estimated from the labels you have. So the validation fraction goes up, the split is by patient, and the confidence interval is reported next to the number rather than after it (Metric Uncertainty, Cross-Validation for squeezing more out of a small set).
looks like A clean random split of three thousand photos into train, validation and test with matching class balance.
why it leaks Two photos of the same lesion are near-duplicates. The model memorises the training photo and "recognises" the validation one; the transferred features make this easier, not harder, because they are excellent at similarity.
fix Split by patient id, and within a patient keep all photos on one side; stratify by camera so each split sees each colour profile (Group Split).
When starting from scratch is right
Transfer earns its cost when the pretrained model's low-level structure is the task's low-level structure. Lesion photos are natural images, so it does. A spectrogram is an image in format only, and the transfer is weaker but often still positive. A table of insurance claims shares nothing with photographs, and there is no representation to transfer; the model that wins there is a tree ensemble on the raw columns, trained from scratch, in minutes (Which Model Should We Use?).
The other case is domain distance within a modality. A model pretrained on natural photographs transfers poorly to satellite radar or electron microscopy, whose low-level statistics are different; with enough labelled data, from-scratch or domain-specific pretraining wins. The way to know is the frozen-head baseline: if the fixed representation is no better than hand-crafted features, the transfer is not there.
The clinic's photos, after the pretrained model's own normalisation, have edge, texture and colour statistics within the range the frozen early layers were trained on.
holds when The three known cameras are represented in training and their colour profiles are stable; preprocessing is the pretrained model's, shipped with the artifact.
breaks when A new camera or a firmware update changes the colour profile; a clinic starts submitting phone photos; a preprocessing re-implementation resizes differently.
respond Add the new camera's photos to the adaptation set and re-adapt with the same frozen/unfrozen configuration; do not lower the threshold to recover recall.
Does a pretrained model's structure match the task's inputs, and how much labelled data is there?
when Inputs are close to the pretraining distribution and labelled data is scarce; the baseline for every other option.
cost Ceiling set by the fixed representation; cannot adapt to domain-specific low-level features.
when The frozen head leaves recall on the table and there is enough data — thousands of examples — to adapt without forgetting.
cost Learning-rate schedule must be chosen carefully; a wrong one destroys the transfer; more compute per experiment.
when Tabular data with no pretrained representation, or a modality whose low-level statistics differ from anything pretrained, with enough data to learn them.
cost Needs orders of magnitude more labelled data on images or text; on tabular data it is simply the normal path.
when Lots of unlabelled in-domain data and a domain far from general pretraining — medical imaging at scale, scientific instruments.
cost A pretraining run of your own, with all of its cost and all of the decisions this module says you inherit (Self-Supervised Learning).
How to build it
Most important first.
- Start frozen: extract the pretrained representation and train a linear head on it. That is the baseline everything else must beat, and on three thousand images it is often close to the best (The Linear Baseline).
- Then unfreeze progressively from the top, with a learning rate an order of magnitude below the pretraining rate for the transferred layers, and watch the patient-held-out validation curve, not the training curve (Learning Curves).
- Split by patient and stratify by camera, and reserve a larger fraction for validation and test than you would with a large dataset: with three thousand examples the metric's confidence interval is wide and the validation set is doing more work per example (Train / Validation / Test, Metric Uncertainty).
- Normalise inputs exactly as the pretrained model expects — its mean, its standard deviation, its resolution. A pretrained network is a contract with its preprocessing (Preprocessing Lives in the Artifact).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Recall of escalated cases at the operating threshold on patient-held-out validation, with its confidence interval — the number that maps to the queue decision, and its width is the honest statement of how little data there is.
- The frozen-head baseline's recall on the same split; unfreezing is justified only by beating it outside the interval.
- Do not measure on a random photo split, and do not compare training accuracy across freezing strategies: a fully unfrozen network wins that contest by memorising.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The low-level statistics of the clinic's photos — edges, textures, colour distributions after normalisation — remain within the range the pretrained layers learned; a new camera or lighting setup is a shift in exactly those statistics.
- The preprocessing at serving time is the pretrained model's preprocessing, byte for byte, and is shipped with the artifact.
- The specialist's escalation criterion, which is the label, does not change; a new guideline changes the target and no amount of transferred structure helps (Concept Drift).
- Offline: compare frozen-head, partial-unfreeze and full-unfreeze on patient-held-out validation with confidence intervals; check that the chosen configuration also holds on each camera slice.
- Online: log the camera id with each prediction and monitor recall per camera as specialist decisions arrive; a new camera shows up as a per-slice drop before it shows in aggregate (Evaluation Slices).
- Over time: keep a fixed regression set of photos with specialist labels and re-score it on every retrain; a transferred model that is re-adapted on new data can forget the old cases (Model Regression Tests).
What can go wrong
- Unfrozen with the default learning rate: the training loss drops quickly, the transferred features are destroyed, and the model overfits like a from-scratch network. The training curve looks best of all.
- The pretrained model's preprocessing was not reproduced — images fed at a different resolution or unnormalised — and the transferred features receive inputs they never saw; accuracy is poor and the cause is invisible in the training code.
- A fourth camera is installed. Its colour profile is outside the three the model adapted to; the frozen early layers are fine and the adapted late layers are not (Data Drift).
- Transfer buys data efficiency at the price of inheriting the pretrained model's input contract, size and biases; the clinic's model is now as large as an internet-scale image model for a four-way decision.
- Freezing is safe and limits quality; unfreezing raises the ceiling and the risk of forgetting, and needs a learning-rate schedule chosen with care.
- A larger validation fraction gives an honest interval and leaves fewer examples to train on — with three thousand images, both numbers hurt.
- "The pretrained model has never seen skin, so it cannot help." Its early layers have seen every edge, texture and colour gradient a camera produces. That is most of what a lesion photo is made of. Only the head is about dogs.
- "We fine-tuned everything and the training loss is far lower, so it is better." A fully unfrozen network on three thousand images memorises them and reports a low training loss for exactly that reason. The patient-held-out number is the one to read.
- "Transfer learning means we do not need much data." It means you need much less. Three thousand examples still gives a wide confidence interval on recall, and a decision threshold set on that interval is a decision made under real uncertainty.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DATA-SPECIFICThe transfer argument is specific to modalities with shared low-level structure — images, audio, natural-language text — where a pretrained model's early layers are general. On tabular data there is no pretrained representation to transfer, and a gradient-boosted model on the raw columns is the stronger default.
- SIMPLIFIEDLayer-wise freezing is presented as a single dial from "all frozen" to "all trainable"; real schedules use discriminative learning rates per layer group, warm-up, and re-initialisation of normalisation statistics, which change the numbers without changing the argument.
- CONTESTEDA serious position holds that with modern self-supervised pretraining the representation is good enough that a frozen backbone plus a linear or small head is nearly always sufficient, and that unfreezing on small datasets is a net risk — it buys a point or two of accuracy in exchange for forgetting, instability and a much more expensive training loop. That is well supported for tasks close to the pretraining distribution; the argument for unfreezing is strongest when the domain is far from it, as medical imaging is, and the honest answer is to measure both on the held-out split.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Clinical validation — a triage model that orders a specialist's queue is a medical device in many jurisdictions, and the evidence standard for that is a regulatory question this lesson's patient-held-out recall only begins to answer.