Learning Curves
Error against training-set size, for training and validation together. The shape says whether more data, more capacity or better features is the fix — before any of them is tried.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
You can spend the next quarter labelling more data, or building a bigger model, or engineering features. Which one will move the validation number, and how can you know before spending it?
A medical-imaging start-up has a model that classifies scans as needing specialist review. Validation performance is short of the target. The CEO is choosing between a labelling contract for twenty thousand more scans and a larger architecture, and wants to know which is not a waste of money.
Performance is short, so get more data. Every ML story says more data helps, and labelled data is the thing the team can buy.
The team buys twenty thousand labels. The validation number barely moves, because the model was already bias-limited: its training and validation curves had converged early and flat, and more rows of the same shape do not add capacity the model does not have.
- The team buys twenty thousand labels. The validation number barely moves, because the model was already bias-limited: its training and validation curves had converged early and flat, and more rows of the same shape do not add capacity the model does not have.
- Or the reverse: the team builds a larger architecture on twelve thousand scans, training error goes to zero, validation gets worse, and a quarter is spent on a model that needed the labels instead.
- Both mistakes were predictable from a plot that takes an afternoon: train on nested subsets of the data and record both errors at each size.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a scan will be flagged by a specialist. The label is the radiologist's decision, which costs a specialist's time to produce — so each additional label has a real price.
- The decision is which scans a specialist sees first; a missed flag is a delayed diagnosis, so recall at a fixed review capacity is the number that maps to the outcome.
- One example is one scan with a binary specialist decision. Twelve thousand labelled scans from three hospitals, with the hospital id retained. Unlabelled scans are abundant; labelled ones are the bottleneck.
- The labelling process has known noise — two radiologists disagree on a meaningful fraction of borderline scans — which sets a floor that no model reaches.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A learning curve is validation error and training error as a function of the number of training rows, at fixed model settings. With few rows, training error is near zero (easy to fit) and validation error is high (nothing learned) — the gap is large. As rows are added, training error rises toward the achievable floor and validation error falls toward it.
- Where the curves end up describes the model. If they are still converging at the full dataset size — a gap that is closing but open — the model is variance-limited and more data will continue to help; extrapolate the trend to estimate how much. If they have already met and plateaued, high, the model is bias-limited: it has extracted everything its capacity allows, and more data cannot lower the plateau.
- The level of the plateau relative to the noise floor is the third reading. Curves that meet near the label-disagreement rate are at the ceiling; nothing short of better labels helps. Curves that meet well above it have headroom that capacity or features can claim.
Two shapes, two budgets
Plot validation error against rows, and training error on the same axes. In the variance-limited case the training curve rises, the validation curve falls, and at the full dataset they are still approaching each other. In the bias-limited case they met a long time ago and have run flat, together, at a level the model cannot get below.
The first shape is the argument for the labelling contract; the second is the argument against it. The explorer's size panel at /ml/curves draws the first shape for a well-chosen polynomial degree and the second for a degree too low — the same data, two diagnoses.
Variance-limited (get data) Bias-limited (get capacity / features)
error error
|\ |
| \ validation |______ validation
| \_____ |______ training
| ___/ training |
|____/___________ rows |________________ rows
gap still closing met early, plateau highPlotting it honestly
The curve is only as honest as its subsets. Draw them nested — each larger subset contains the smaller — so the points describe one growing dataset rather than several unrelated ones. Keep the validation set fixed and grouped the same way the real evaluation is grouped, or the small-subset points will show a variance regime the model does not actually have.
Repeat each size with several random subsets and show the spread. The tail of the validation curve is the part that carries the decision, and it is also the part with the fewest points and the most leverage on the extrapolation.
1import numpy as np2 3def learning_curve(train_fn, eval_fn, X, y, groups, X_val, y_val,4 fractions=(0.1, 0.2, 0.4, 0.7, 1.0), repeats=5, seed=0):5 rng = np.random.default_rng(seed)6 uniq = np.unique(groups)7 out = []8 for f in fractions:9 tr, va = [], []10 for _ in range(repeats):11 # sample whole groups so a small subset never contains12 # part of a hospital that the validation set also contains13 keep = rng.choice(uniq, size=max(1, int(f * len(uniq))), replace=False)14 m = np.isin(groups, keep)15 model = train_fn(X[m], y[m])16 tr.append(eval_fn(model, X[m], y[m]))17 va.append(eval_fn(model, X_val, y_val))18 out.append((f, np.mean(tr), np.mean(va), np.std(va)))19 return out # (fraction, train_err, val_err, val_spread)The group-level sampling is the part that gets skipped. Row-level subsets of grouped data produce a curve that says "data" when the honest curve says "capacity", because small row-level subsets already contain a piece of every group.
The decision the curve makes
The imaging team's curve had met and flattened by half the data, well above the radiologist-disagreement rate. That is the bias signature: the labelling contract would have bought nothing. A larger architecture pulled the plateau down and re-opened the gap — and *that* curve, at the full dataset, was still closing, which is when the labels became worth buying.
The order matters and the plot is what orders it. Capacity first when the plateau is high; data when the gap is open; better labels when the plateau is at the floor. Each intervention changes the curve, and the next decision reads off the new one.
Data added after the curve was plotted comes from the same distribution as the data on the curve, so the extrapolated trend applies to it.
holds when New scans come from the same hospitals, scanners and protocols as the existing set, and the labelling pool is unchanged.
breaks when The contract sources scans from a new hospital with different equipment — the new rows are a distribution shift, not more of the curve, and may raise validation error before lowering it.
respond Treat the new source as its own slice, plot its own curve, and decide whether the model is one model or two.
Where does the next quarter's budget go?
when The gap between training and validation is still closing at the full dataset size; extrapolate the tail to size the purchase.
cost Labelling spend, and a re-plot afterwards to find the next plateau.
when The curves have met and plateaued well above the label noise floor.
cost Variance to manage; the re-opened gap is the next data argument.
when Curves plateau high and a larger model of the same kind does not lower them; the structure is not reachable from these inputs.
cost Feature work or a different model family, and new serving-time transforms.
when The plateau sits at the inter-annotator disagreement rate; the model is at the ceiling of the label process.
cost A second annotator per scan, adjudication, and a smaller but cleaner dataset.
How to build it
Most important first.
- Plot the learning curve before committing budget. Train at ten, twenty, forty, eighty per cent of the data with the same validation set and settings; the shape decides between data, capacity and features (Bias and Variance).
- Read the gap and the plateau separately: an open gap says data; a high closed plateau says capacity or representation; a plateau at the noise floor says labels (Label Quality).
- When the curve says data, estimate how much by fitting the tail of the validation curve; a power law is the usual shape and the extrapolation is honest enough to decide between twenty thousand and two hundred thousand labels.
- Repeat the plot after each intervention. A capacity increase should move the plateau down and re-open the gap; that re-opened gap is now the data argument.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Validation error at each subset size, with confidence bands from repeated subsampling. The trend and its uncertainty are what the decision rests on (Metric Uncertainty).
- The gap at full size, and its slope over the last few sizes. A positive slope on the gap means it is still closing.
- Do not measure the training error at full size in isolation — it is one endpoint of a curve, and it is the curve that carries the information.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The additional data comes from the same distribution as the data the curve was plotted on; scans from a fourth hospital extend the curve only if they look like the first three.
- The model settings under which the curve was plotted are the ones used for the final training — a change in capacity invalidates the curve and requires a new one.
- The label noise floor is stable; if the radiologist pool changes and disagreement rises, the ceiling moves and the plotted headroom shrinks.
- Offline: the learning curve itself, with bands, at the current settings; then again after any change in capacity or features, to confirm the plateau moved the way the diagnosis predicted.
- Online: after the labelling contract delivers, check that the validation number moved by the amount the extrapolation predicted. A miss by a large margin means the curve was plotted on the wrong grouping or the floor was reached.
- Over time: re-plot the curve each time the dataset has grown by a meaningful fraction. The regime changes as data accumulates, and a model that was data-limited last year may be capacity-limited now.
What can go wrong
- The subsets are drawn without respecting hospital groups, so small subsets contain scans from all three hospitals and the curve reports a variance regime that the grouped truth does not have (Group Split).
- The curve is plotted at one model setting and read as a property of the model family. A larger model has a different curve; the decision "data will not help" was about the small model only.
- The extrapolation is trusted too far: the power-law tail predicts the target at a hundred thousand labels, the labels are bought, and the curve bends flat at forty thousand because the label noise floor arrived first.
- Plotting a learning curve costs several training runs at increasing sizes, which for a large network is real compute. The alternative is guessing with a quarter's budget.
- Nested subsets share a validation set, so the points on the curve are correlated and the confidence bands are narrower than they should be. Repeated random subsampling helps and multiplies the cost.
- The curve is a diagnosis at one model setting. Reading the decision off it requires also asking what a bigger model's curve would look like — which is one more set of runs.
- "More data always helps." It helps when the gap is open. Two curves that plateau together high say the model cannot use more data of this shape, and the money goes to capacity or features.
- "The plateau is the noise floor." Only if it sits at the label-disagreement rate. A plateau well above that is bias, and a larger model or a better representation will lower it.
- "The curve says data, so buy as much as possible." The curve says data *until the next plateau*. Extrapolate the tail, buy that much, re-plot.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALError against sample size with the gap and plateau readings applies to every supervised model; the shape of the tail (how fast validation error falls with data) is what varies by family and task.
- SIMULATEDThe training-set-size panel in the explorer fits polynomials to synthetic data at increasing sample sizes; the closing gap it shows is the mechanism, not a measurement on any imaging dataset.
- SCALE-SPECIFICFor models where a single training run costs days, the full curve is unaffordable and the practice becomes fitting a scaling law from a few small runs; the reading is the same, but the extrapolation carries much more of the weight.
Where the depth lives
This domain teaches the model and hands the rest off by name.