How do you tune hyperparameters without contaminating the test set — and how many configurations is too many?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A team ran 1,200 configurations of a gradient-boosted model through random search, selected the best on the validation set, and reported its validation AUC as the expected production AUC. Six months later the model underperforms that number by a margin no drift monitor explains.
React to this
Say what you would question, what you would trust, and what you would need to know first.
Tuning report (illustrative) search: random, 1,200 trials, 14 hyperparameters selection: best validation AUC best trial: AUC 0.871 (validation) reported: "expected production AUC 0.87" test set: exists (20% holdout), never evaluated production: AUC 0.84 on labels arriving over six months, stable from month one
What it is really testing
Whether the candidate understands that selection on a set makes that set's score optimistic, in proportion to how many things were selected among, and can describe a protocol — nested validation, a single untouched test set, a budget — that produces an honest estimate.