Bias, Variance & Generalisation
Underfitting, overfitting, learning curves, regularisation and early stopping — the mechanics of why a model that memorised the training set looks perfect until it meets new data.
Every model is wrong in two ways at once: too simple to represent the pattern, or too flexible to ignore the noise. The gap between training and validation error tells you which.
A model that memorises the noise in its training set scores perfectly on that set and poorly on the next one. Every route to a better training number is also a route to this.
A model with too little capacity, or the wrong representation, misses structure that is plainly in the data. It is the honest failure — visible offline — and still the one most often fixed with the wrong tool.
Error against training-set size, for training and validation together. The shape says whether more data, more capacity or better features is the fix — before any of them is tried.
Every way of refusing part of the training fit: L1, L2, dropout, depth limits, shrinkage, early stopping. The strength is a hyperparameter, it is tuned on validation, and for the penalty forms the features must be on one scale.
Stop training when validation loss stops improving, keep the best checkpoint, and accept that the validation set you stopped on is no longer an unbiased estimate of anything.