Bias and Variance
Every model is wrong in two ways at once: too simple to represent the pattern, or too flexible to ignore the noise. The gap between training and validation error tells you which.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Two models both miss the target. One is systematically off in the same direction on every dataset, the other is right on average but swings wildly between training runs. Which one do you have, and what does that decide?
A pricing team fits a demand curve for a delivery product. The analyst's first version is a straight line and under-predicts every weekend. The second version, a high-degree polynomial, fits last quarter's data almost perfectly and then forecasts negative demand for the first week of the new quarter.
Pick the model with the lowest training error. If the polynomial fits the data better than the line, it has learned more of what is going on.
Training error falls monotonically with model flexibility no matter what the data is. A model with as many parameters as rows fits any labels, including random ones, so a low training number cannot distinguish learned structure from memorised noise.
- Training error falls monotonically with model flexibility no matter what the data is. A model with as many parameters as rows fits any labels, including random ones, so a low training number cannot distinguish learned structure from memorised noise.
- The polynomial's forecast for next week is dominated by whichever noisy points happened to sit near the end of the training window. Refit on a slightly different quarter and the forecast changes sign. Nothing in the offline number warned of this because the offline number was computed on the same points.
- The line's error is stable across refits and across quarters — and always wrong the same way, because a line cannot bend on weekends. Its problem is visible offline, which is why it was abandoned first.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict units ordered per region per day, from price, day of week, weather and recent order history. The label is the count in the orders table, which is known exactly and only after the day ends.
- The decision downstream is a price and a stock level for tomorrow, so what matters is error on days the model has not seen — every day from here on is one of those.
- One example is one region-day. Roughly a year of history per region, so the sample is small relative to the number of interacting effects — holidays, promotions, weather — that shape it.
- Demand has structure the analyst can see by eye (weekly cycle, price sensitivity) and noise nobody can predict (a football match, a delivery van breaking down). Both are in the label.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Decompose the expected squared error at a point into three terms: bias² (how far the average prediction over all possible training sets is from the truth), variance (how much the prediction moves from one training set to another), and irreducible noise. Bias is the model class being unable to represent the function; variance is the model class being able to represent too many.
- Flexibility trades one for the other. A constant predictor has zero variance and maximum bias. An interpolating polynomial has near-zero bias on the training points and enormous variance everywhere between them. Every model class sits somewhere on that path, and where it sits depends on the amount of training data as much as on the model.
- You never observe bias and variance directly — you observe training error and validation error on one split. High bias shows as both errors high and close together. High variance shows as training error far below validation error. That reading is the practical content of the decomposition.
Two ways to be wrong
Imagine drawing many training sets from the same process and fitting the same model class to each. The straight line lands in almost the same place every time and misses the weekend bump every time: low variance, high bias. The degree-fifteen polynomial threads through each training set's particular noise and lands somewhere different every time: near-zero bias at the training points, high variance everywhere else.
The error on a new day is the sum of both, plus the noise no model can remove. Reducing one usually raises the other, and the best model class for *this* amount of data is the one that minimises the sum — not the one that minimises either term.
Reading the diagnosis from two numbers
Bias and variance are not observable on one dataset, but their signatures are. Sweep a flexibility knob and record training and validation error at each setting: training error falls throughout, validation error falls and then rises. Left of the minimum the model is bias-limited; right of it, variance-limited. The explorer at /ml/curves draws exactly this for polynomial degree.
The §143 reading is the whole diagnostic in one row each: training strong and validation weak means overfitting — the model has capacity the data cannot pin down. Training and validation both weak means underfitting — the model cannot represent the pattern regardless of how much data it sees.
| Training error | Validation error | Gap | Diagnosis | Fix that applies |
|---|---|---|---|---|
| High | High, close to training | Small | Bias-limited (underfitting) | More capacity, better features, less regularisation |
| Low | Much higher than training | Large | Variance-limited (overfitting) | More data, regularisation, ensembling, fewer features |
| Low | Low, close to training | Small | Fitting well — or leaking | Audit the split before celebrating |
| High | Lower than training | Negative | Split or measurement bug | Check the pipeline, not the model |
1import numpy as np2 3def prediction_spread(fit, X, y, x_query, n_resamples=50, seed=0):4 # Refit the same model class on bootstrap resamples of the training set5 # and record what each fit predicts at x_query. The spread IS the variance.6 rng = np.random.default_rng(seed)7 preds = []8 for _ in range(n_resamples):9 idx = rng.integers(0, len(X), size=len(X))10 model = fit(X[idx], y[idx])11 preds.append(model(x_query))12 preds = np.array(preds)13 return preds.mean(axis=0), preds.std(axis=0) # centre and spread per query pointThe mean of the resampled predictions is what bias is measured against; the standard deviation is variance made visible. For a small tabular problem fifty refits is seconds, and it tells you more than one validation number does.
The balance is a property of the dataset, not the model
The same polynomial that overfits a year of region-days would be well-behaved on ten years of them. Variance falls as training data grows; bias does not move. So the diagnosis is made for a model class *at a sample size*, and it expires when the sample size changes.
That is the assumption that must keep holding after deployment: the model was chosen to sit at the validation minimum for this much data with this much noise. Both quantities drift, and when they do the model is no longer at the minimum, even though nothing about the model changed.
The model class and its flexibility setting sit at or near the validation-error minimum for the current training-set size and the current label noise level.
holds when The training set is refreshed at a comparable size, the labelling process is stable, and the flexibility sweep was re-run the last time either changed materially.
breaks when The dataset doubles and the chosen model is now bias-limited, leaving quality on the table; or the label process gets noisier and the same model now fits noise it previously could not see.
respond Re-run the flexibility sweep, not just the retraining. A retrain at the old setting rebuilds the old balance on data that no longer matches it.
How to build it
Most important first.
- Always hold out data before fitting, and read two numbers, not one: the training error and the held-out error. The pair is diagnostic; either alone is not (Train / Validation / Test).
- Sweep flexibility deliberately — polynomial degree, tree depth, regularisation strength, network width — and plot both errors against it. The validation curve has a minimum; training error does not (Hyperparameters).
- Choose the fix from the diagnosis. High bias: more capacity, better features, a less restrictive model class. High variance: more data, regularisation, ensembling, fewer features (Learning Curves).
- Refit on resampled training sets and look at how much the predictions move. Variance is literally that spread, and for a small tabular problem it is cheap to measure (Cross-Validation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The gap between training and validation error, and the level of the validation error. Those two quantities map to the two diagnoses; training error alone maps to nothing.
- The spread of validation error across folds or across bootstrap refits. A model whose validation error swings widely between folds is high-variance even if its mean is acceptable (Metric Uncertainty).
- Do not measure the fit of the curve by eye on the training points. A smooth-looking curve through every point is the picture of variance.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The validation rows are drawn from the same distribution the model will face in production — otherwise the validation error is not an estimate of anything the business will see.
- The noise level in the labels is roughly stable over time; if the irreducible term grows, a model tuned to the old noise floor looks like it started overfitting.
- The amount of training data at the next refit is comparable to now, because the bias/variance balance was chosen for this sample size and moves with it.
- Offline: plot training and validation error against a flexibility knob; confirm the validation curve has an interior minimum and that the chosen setting sits at it, not on the training curve's descent.
- Online: compare the error on the first weeks of production against the validation error. A gap that opens immediately is a distribution difference, not variance.
- Over time: re-run the flexibility sweep when the dataset has grown substantially. The minimum moves right as data accumulates, and a model class that was too flexible can become the right one.
What can go wrong
- The validation set is small, so its error has its own variance, and the "minimum" of the validation curve is a noisy point the team then tunes toward (Evaluation Leakage).
- The diagnosis was right at training time and wrong a year later: more data arrived, the high-variance model would now generalise, but the team remembers "the polynomial overfits" as a fact about the model rather than about the dataset it had then.
- Both errors are low and close, the model ships, and production is poor — because the split leaked, and neither number was measuring generalisation (Data Leakage).
- Holding out data to measure the gap means training on less of it, which raises variance for the model you actually ship. Cross-validation recovers most of the data at the price of k training runs.
- Sweeping flexibility is a search, and a search on the validation set makes the validation number optimistic. The test set exists so there is one number that was never searched on (Never Tune on the Test Set).
- The decomposition is exact for squared error and only a metaphor for most classification losses. The reading — gap versus level — survives; the arithmetic does not.
- "The model with the lowest training error learned the most." It fit the most. Learning is measured on rows the model never saw.
- "A high-variance model is a bad model." It is a model with too little data for its flexibility. Double the data and the same model may be the best available.
- "Both errors are high, so we need more data." Two high curves that have already flattened together are the signature of bias; more data moves them nowhere. The fix is capacity or features.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe decomposition of error into a systematic part and a data-dependent part holds for every model family and every supervised task; only the arithmetic form (exact for squared error) is specific to regression.
- SIMULATEDThe curves in the Bias / Variance Explorer are produced by fitting polynomials to a seeded synthetic function with Gaussian noise. They show the shape of the argument — training error falling, validation error U-shaped — not a measurement on any real dataset.
- CONTESTEDThe classical U-shaped picture is challenged by the "double descent" observation: for heavily over-parameterised models, validation error can fall again past the interpolation point, and practitioners in deep learning argue the bias/variance trade-off is a small-model phenomenon. The strongest form of that view is that with enough data and implicit regularisation from the optimiser, bigger is simply better. It is a real effect; it does not remove the need to measure held-out error, which is the only claim this lesson makes.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Programming Languages & Runtime Internals — the polynomial explorer's numerical instability at high degree is a floating-point conditioning problem, not a statistical one; a reader who sees wild coefficients should suspect the solver before the theory.