SIMULATED

Bias / Variance Explorer

Polynomial degree against training and validation error, then training-set size against the gap between them. Two claims, both computed: training error only ever falls as the model gets more flexible, and validation error falls then rises. A third about data: more of it narrows the gap.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

A model that memorised the training set looks perfect until it meets new data. That sentence is repeated so often it has stopped meaning anything, so here it is as a measurement: the training curve and the validation curve are on the same axes, and the distance between them is the amount of what the model learned that was noise. The shape tells you which of two different problems you have — and they have opposite fixes. A gap that closes with more data was variance; a floor that more data does not lower was bias, and no amount of collecting will move it.

Noise in the data · 0.30

Ridge penalty λ

training MSEvalidation MSEselected degree

Degree 3 fitted to 30 points

fitted polynomialtrue functiontraining points

Degree · 3

training MSE

0.098

validation MSE

0.113

gap

0.014

best degree on validation

Left of the minimum — bias-limited

Both errors are high and close together. The model cannot represent the function — a straight line asked for a sine wave — and validation error is high for the same reason training error is. More data will not help; more capacity will. This is underfitting, and it is regularly misdiagnosed as “not enough data”.

Right of the minimum — variance-limited

Training error keeps falling — it must, since a higher degree can do everything a lower one could — and validation error climbs away from it. The extra capacity is spent fitting the noise in these 30 points. Ridge pulls the curve back: it trades a little bias for a lot of variance, and the degree where validation turns up moves right as λ grows.

How to read this page honestly

What the model is, and what it deliberately refuses to be.

SIMULATEDOne-dimensional data, y = sin(2πx) + 0.5x + noise, 30 training points and a separate draw of 200 validation points from the same function. The fits are real least squares — normal equations solved by Gaussian elimination with partial pivoting, ridge added to the diagonal — in a Chebyshev basis so that degree 12 is numerically meaningful. A degree-12 fit that wiggles is wiggling because that is what least squares does with 30 noisy points.
SIMPLIFIEDPolynomial degree is the cleanest knob for capacity, but real models do not have one knob. A tree's depth, a network's width and its training time, a boosting round count all trace the same U on validation error — and in a real project the x-axis is usually "how long we kept tuning".
DATA-SPECIFICThe degree where validation error turns up depends on noise and on how many points there are. Lower the noise and the optimum moves right; add data on the learning-curve tab and a degree that overfitted 30 points stops overfitting 320. Nothing about "degree 3 is best" transfers.

Take it further

Bias and Variance →

The decomposition, and why the two halves have opposite fixes.

Learning Curves →

How to decide whether more data is worth collecting before collecting it.

Regularisation →

Ridge, lasso, early stopping — the same trade made three ways.