Evaluation
Business metrics against model metrics, offline against online, cross-validation and its temporal variant, slices — and the rule that the test set is touched once.
A model metric describes the model; a business metric describes what happened when the model's output was acted on. The map between them is the operating point and the action, and a better model metric can produce a worse business outcome.
Offline evaluation scores a model on a historical dataset produced by the previous policy. Online evaluation measures what happens when the model acts on live traffic. Strong offline numbers are a reason to run the online test, not a substitute for it.
k-fold cross-validation trades k trainings for a lower-variance estimate and a spread. It is the right tool for small data and model selection, the wrong tool for temporal or grouped data unless the folds respect the structure, and it is not an evaluation of the model you will ship.
When the model will predict the future, validate on the future: forward-chaining folds, a gap between training end and validation start equal to the label delay, and never a shuffle. A random split on temporal data is a leakage simulator with a nicer name.
An aggregate metric is a weighted average over subgroups, and the weights are the dataset's, not the business's. A model can improve on average and regress on the segment that matters, and only a sliced evaluation can see it.
A validation metric is a sample statistic with an interval around it. Two models compared on the same holdout need a paired comparison; a small test set cannot distinguish small improvements; and every comparison made against one holdout erodes it a little.
The test set is touched once. Every look costs information; hyperparameter search, feature selection, early stopping and model selection all happen on validation. A team that picks the best of forty runs on the test set has shipped noise with a certificate.