TuningGENERALSCALE-SPECIFICCONTESTED

Grid Search and Random Search

Grid search is exhaustive and exponential. Random search covers each important setting better per trial, because most settings turn out not to matter. Neither is allowed anywhere near the test set.

Target & dataWhat to measureWhat must stay true▶ Gradient Descent Visualizer

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

Given a budget of trials and a validation set, how should the trials be placed — and what does the best trial's score actually estimate?

The problem

A demand-planning team has a gradient-boosted model with seven settings and a four-hour training run. They ran a grid over three values of each, found the best cell, reported its validation error, and shipped it. Production error is noticeably worse than the reported number, and the next grid will take a week.

The obvious approach

Pick three sensible values for every setting and try every combination. It is systematic, nothing is missed, the result is a table anyone can read, and the best cell is the best model.

Why it breaks

Seven settings at three values each is 2,187 training runs. The team ran a coarse subset and called it a grid; the "best cell" is the best of whichever cells finished by the deadline.

How it breaks — usually after the offline metric looked fine
  • Seven settings at three values each is 2,187 training runs. The team ran a coarse subset and called it a grid; the "best cell" is the best of whichever cells finished by the deadline.
  • Of the seven settings, two — the learning rate and the number of rounds — move the metric; the other five barely register. The grid spent most of its trials varying settings that do not matter, at three distinct values of the two that do.
  • Two thousand trials against one validation quarter: the winning cell is partly the one that best fits that quarter's noise. The reported number is the maximum of two thousand noisy draws, and production, which is a different draw, comes in worse (Metric Uncertainty).
  • The grid minimised symmetric error. The order policy pays more for under-ordering than over-ordering, so the cell that minimised RMSE is not the cell that minimises cost (Business Metrics vs Model Metrics).
Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target
  • Predict next week's unit demand per store and product. The label is the realised sales figure from the point-of-sale feed, with stock-outs censoring the true demand (Label Construction).
  • The decision downstream is an order quantity, so the metric that matters is an asymmetric cost of over- and under-ordering, not the symmetric error the grid was minimising.
Data
  • One example is one store-product-week: lagged sales, rolling means, promotion flags, calendar features and price. Ten million rows over three years.
  • The validation set is the most recent quarter, chosen by time (Time-Based Split). The same quarter judged all two thousand grid cells and then reported the winner.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

  • Grid search enumerates the Cartesian product of candidate values. Its cost is the product of the per-setting counts, so it is exponential in the number of settings, and its resolution on any one setting is limited to the handful of values you could afford to list.
  • Random search draws each trial's settings independently from a distribution per setting. With the same budget of trials, every trial has a distinct value for every setting. If only two of seven settings matter, a grid of three values gives you three distinct values of each important setting; random search gives you as many distinct values of each as you have trials. That is the whole argument, and it holds because in most problems the effective dimensionality of the search is low.
  • Both methods judge each trial on the same validation set. The best trial's score is a maximum over noisy estimates, and a maximum over more draws is more optimistic. Neither method knows this; the search returns the best number it saw, which is exactly the number most contaminated by selection.

Why random beats grid when few settings matter

Picture two settings and nine trials. A 3×3 grid gives three distinct learning rates and three distinct depths. Nine random draws give nine distinct learning rates and nine distinct depths. If depth turns out not to matter, the grid has tested the learning rate at three points and the random search at nine, for the same cost.

This is not a trick. In most problems a handful of settings dominate and the rest are nearly flat, and nobody knows in advance which is which. A grid pays full price to resolve the flat ones; random search pays nothing extra to resolve the important ones. Sequential methods go further by learning which settings matter as they go.

Nine trials, two settings, one of them flat
Grid, 3 × 3
Three distinct values of the learning rate, each tested three times at depths that make no difference. The search saw the learning rate at exactly three points.
Random, nine draws
Nine distinct learning rates, each paired with a depth that made no difference. The search saw the learning rate at nine points across its range.

Each random trial carries a fresh value of every setting, so the coverage of any one setting equals the trial count rather than the per-setting grid resolution. The advantage grows with the number of settings that turn out to be flat.

The best of many is an optimist

Each trial's validation score is the true quality of that configuration plus noise from the finite validation set. The search returns the trial with the highest score, and a high score is more likely to have positive noise. The more trials, the more the winner's score reflects lucky noise rather than true quality.

This is evaluation leakage in slow motion. No trial read the labels directly, but the selection did, two thousand times. The only number that escapes it is one computed on data that no trial and no selection ever touched.

leakageThe validation score used to choose the winnerSelection against the validation set

looks like A clean protocol — train on the training period, score on the validation period, pick the best configuration. No column from the future, no target in the features.

why it leaks The choice of winner is a function of the validation labels. Across many trials, the configuration that maximises the score is partly the one whose errors happen to line up with that set's noise. The score then reports the fit to the noise as if it were quality.

offline
The reported validation score of the winner is optimistic, and increasingly so with the number of trials. The search itself cannot see this: it returns its maximum.
production
Production is a fresh draw. The model regresses toward its true quality, and the gap between the reported number and the production number is read as drift or a serving bug when it is selection optimism.

fix Freeze all settings after the search and evaluate once on a set no trial touched — the test set, or the final held-out period on a time-based split. Report both numbers and the trial count.

when this feature is fine When the number of trials is small relative to the validation set — a handful of configurations against tens of thousands of examples — the optimism is within noise and the validation score is a fair estimate. The leak is proportional to trials divided by validation size.

A search that cannot touch the test set

The protocol is mechanical once the splits are right: the search loop sees the training and validation periods and nothing else; the test period is loaded in a different function that runs once, after the winner is frozen. The discipline is in the code structure, not in good intentions.

On time series the same structure applies with periods instead of random folds (Time-Series Validation). The training period ends before the validation period begins, which ends before the test period begins, and the search loop's data loader physically cannot reach the last one.

Random search with a test set the loop cannot reach
1import math, random
2 
3def sample_config(rng):
4    return {
5        "learning_rate": 10 ** rng.uniform(-3, -0.5),   # log-uniform
6        "max_depth":     rng.randint(3, 10),
7        "rounds":        rng.randint(200, 2000),
8        "l2":            10 ** rng.uniform(-2, 2),        # log-uniform
9    }
10 
11def search(train, valid, n_trials, seed):
12    rng, trials = random.Random(seed), []
13    for i in range(n_trials):
14        cfg = sample_config(rng)
15        model = train_model(train, cfg, seed=seed + i)   # seed per trial
16        trials.append((score(model, valid), cfg))
17    trials.sort(key=lambda t: t[0])
18    return trials[0][1], trials            # winner and the full log
19 
20# --- separate function, run once, after the config is frozen ---
21def final_report(train, valid, test, cfg):
22    model = train_model(train + valid, cfg)  # refit on both, settings frozen
23    return {"config": cfg, "test_score": score(model, test)}

The test set is a parameter of final_report and not of search. That is the only guarantee worth having: a reviewer can see that the search loop has no path to it without reading the loop's body.

How to build it

Most important first.

  • Search on the validation set, freeze every setting, then evaluate once on the test set — or, with a time-based split, on the held-out final period that no trial ever touched (Never Tune on the Test Set).
  • Use random search or a sequential method by default; use a grid only when there are one or two settings and you want a readable curve. Draw learning rates and regularisation strengths on a log scale — the interesting range spans orders of magnitude.
  • Tune on the metric the decision uses, or as close to it as is differentiable and stable. A search that minimises the wrong metric is a precise answer to the wrong question.
  • Record every trial, not only the winner: the configs, the seeds, the validation scores and the data version (Experiment Tracking). The number of trials is part of what the winner's score means.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

  • The test-set (or final-period) score of the frozen winner, reported once, alongside the validation score it was selected on. The gap between the two is the selection optimism, and it should be visible.
  • Per-setting sensitivity from the trial log — how much the validation score moves with each setting across trials. This tells you which two settings deserve the next budget.
  • Do not report the best validation score as the model's expected performance. It is the maximum of a noisy sample, and it is biased upward by construction.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions
  • The validation period the search judged on resembles the period the model will be deployed into; a winner chosen on a quarter with an unusual promotion calendar is optimised for that calendar.
  • The settings that mattered during the search — typically the learning rate and the capacity setting — keep mattering at the data volume the model is retrained on; the effective dimensionality of the search is a property of the data and can change.
  • No trial ever read the test period, and the reported final number was computed exactly once after all settings were frozen.
How to verify — offline, online, and over time
  • Offline: compare the winner's validation score to its score on the untouched final period. A large gap means the search overfitted the validation set; more trials will make it worse, not better.
  • Online: shadow the winner against the incumbent for a period before it takes the order decision (Shadow Deployment); the search's metric and the order cost are not the same number.
  • Over time: keep the trial log. When the next search runs, check that the sensitive settings are the same ones — a change in which settings matter is a change in the data worth understanding.

What can go wrong

Failure modes in production
  • The random search draws the learning rate uniformly on a linear scale and spends most trials in the useless upper range; a log-uniform draw was needed.
  • The search is re-run when a new quarter of data arrives, against the new most-recent quarter, and the new winner is reported without a held-out period, so the optimism is rebuilt each quarter.
  • Trials share a seed, so every trial sees the same minibatch order, and a difference that is really seed noise is read as a difference between settings (Random Seeds).
What the recommended approach costs
  • Random search gives up the readable table. You cannot point at a cell; you can only look at a scatter of trials, which is honest and less satisfying.
  • Holding out a final period costs data that the search and training could have used, and on a short series that cost is real; the alternative is a number that cannot be trusted.
  • Recording every trial is storage and discipline; it is also the only way to know whether the winner is the best of twenty or of two thousand.
Misreads
  • "Grid search is exhaustive, so it finds the optimum." It finds the best of the values you listed, which is a coarse lattice; the optimum is almost never on it.
  • "Use the test set to tune — we have plenty of data." Plenty of data does not change what the test set is for. Once it has judged a trial it is a validation set, and you no longer have a test set.
  • "The best trial scored well on validation, so that is what production will do." It is the best of many draws against one set. Production is a fresh draw and will regress toward the true value.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

  • GENERALThat the maximum of many noisy validation scores is biased upward is a statistical fact independent of model family or search method; the more trials against one set, the larger the bias.
  • SCALE-SPECIFICWith two or three cheap settings a grid is fine and readable; random search wins as settings multiply and trials get expensive. When each trial costs GPU-hours, both are too wasteful and a sequential method (Bayesian Optimisation, Successive Halving and Early Termination) or early termination pays for itself.
  • CONTESTEDA strong position holds that on well-understood model families, informed manual tuning — a practitioner who knows that learning rate and rounds dominate gradient boosting and sets them from a learning curve — beats any automated search per hour spent, and that random search is mostly a way for people without that knowledge to stumble into it. That is fair; the reply is that the manual path leaves no trial log, and the winner's optimism is just as real and less visible.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringcompute-waste
Observability & Performancebenchmark-fallacies