Hyperparameters
Parameters are learned from the data. Hyperparameters are chosen before training, judged on validation, and belong in the experiment record — because they decide what the learning is allowed to do.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Which knobs does the training procedure not set for itself, who sets them, on what evidence, and where does that decision get written down?
A credit team has two gradient-boosted models trained on the same table by two engineers. One is clearly better on validation. Nobody can say which of the dozen settings that differ between them is responsible, and the better one cannot be reproduced from its notebook.
Use the library defaults. Defaults were chosen by people who know the algorithm; if the model is weak, the data is the problem, not the settings.
The defaults for tree depth and learning rate were chosen for a benchmark that does not resemble a sparse, imbalanced credit table. The default model underfits the minority class and nobody knows that is why.
- The defaults for tree depth and learning rate were chosen for a benchmark that does not resemble a sparse, imbalanced credit table. The default model underfits the minority class and nobody knows that is why.
- The better model was found by changing five settings at once. When it is retrained next quarter on new data, one engineer copies four of them from memory, and the quarter-on-quarter comparison measures the missing setting, not the new data.
- The number of boosting rounds was chosen by watching the validation curve — which is fine — but the validation set then also picked the threshold, the depth and the regularisation. Its score is now an optimistic estimate of everything at once (Metric Uncertainty).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an applicant will default within twelve months. The label is a delinquency flag joined from the servicing system a year after origination.
- The decision is an approve / refer / decline tier, so the model's output is a probability that a policy layer turns into a tier — the thresholds themselves are one more setting chosen outside training.
- One example is one application: bureau attributes at application time, declared income, product, channel, and aggregates over the applicant's prior accounts.
- Two hundred thousand rows over three years. Trained by two people on two branches; the settings live in notebook cells, some of which were edited after the run that produced the reported number.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A learned parameter is anything the optimiser moves to reduce the loss: linear coefficients, split thresholds inside a tree, weight matrices. A hyperparameter is anything that shapes that optimisation without being moved by it: learning rate, tree depth, the regularisation strength λ, batch size, number of layers, number of trees, k in k-NN, the class weight, the early-stopping patience.
- Hyperparameters cannot be fitted on the training loss because most of them control capacity, and training loss always prefers more capacity: a deeper tree, a smaller λ, more rounds. They have to be judged on data the fit did not see — the validation set — which is why validation exists as a separate split (Train / Validation / Test).
- Some settings are hyperparameters in one framing and part of the model in another. The number of boosting rounds is a hyperparameter if you fix it in advance and a learned quantity if early stopping picks it on validation (Early Stopping). What matters is not the label but the rule: whatever validation chose, validation can no longer estimate honestly.
Two kinds of number in one model
Open a trained gradient-boosted model and you find thousands of split thresholds and leaf values. None of them were chosen by a person. Open its config and you find a dozen settings — depth, learning rate, λ, rounds, minimum leaf size — every one of which was chosen by a person or defaulted by a library, before the first split was ever considered.
The distinction is not academic. The first kind is recovered by re-running training on the same data with the same settings. The second kind is recovered only by having written it down. A model whose config lives in a notebook cell that was edited after the run is a model that cannot be rebuilt.
| Learned parameters | Hyperparameters | |
|---|---|---|
| Set by | The optimiser, during training | A person, a default, or a search — before training |
| Judged on | Training loss | Validation metric, never training loss, never the test set |
| Examples | Coefficients, split thresholds, leaf values, weight matrices | Learning rate, depth, λ, batch size, layers, number of trees, k, class weight, patience |
| Reproduced by | Re-running training with the same data and settings | Reading the experiment record — there is no other source |
| Failure when lost | Retrain | Cannot retrain the same model; the comparison to it is meaningless |
Why capacity settings need a set they never trained on
Ask the training loss whether the tree should be deeper and it always says yes. Ask it whether λ should be smaller and it always says yes. Every capacity setting has a direction that reduces training loss monotonically, and that direction ends in a model that memorised the training set (Overfitting).
So a capacity setting is chosen by a number computed on data the fit did not see. That is the whole reason a validation set is a separate split, and it is also the reason the validation number is no longer an unbiased estimate after the choice is made: the setting that looked best on validation was selected precisely because validation happened to favour it.
1import numpy as np2 3def fit_ridge(X, y, lam):4 # closed-form ridge: (X'X + lam I)^-1 X'y — lam is the hyperparameter5 d = X.shape[1]6 return np.linalg.solve(X.T @ X + lam * np.eye(d), X.T @ y)7 8def mse(X, y, w):9 return float(np.mean((X @ w - y) ** 2))10 11lams = [0.0, 0.01, 0.1, 1.0, 10.0, 100.0]12train_err = {lam: mse(X_tr, y_tr, fit_ridge(X_tr, y_tr, lam)) for lam in lams}13valid_err = {lam: mse(X_va, y_va, fit_ridge(X_tr, y_tr, lam)) for lam in lams}14 15# training error is minimised at lam == 0, always — it is a fact about the16# objective, not about the data. Validation error is minimised somewhere else.17best_lam = min(valid_err, key=valid_err.get)18record = {"lam": best_lam, "candidates": lams, "chosen_on": "validation"}The record logs the candidates as well as the winner. A validation score for the best of six values means something different from a validation score for the best of six hundred, and only the record can tell them apart.
The record is part of the model
A deployed artifact is weights plus everything that decided the weights: the data version, the feature definitions, and the resolved hyperparameters including the defaults nobody touched. Leave the last out and the artifact can be served but not rebuilt, compared or debugged.
The assumption that has to hold after deployment is not about the model's behaviour — it is about the record's completeness. It breaks when a library upgrade changes a default, when a retraining job hard-codes a value the record disagrees with, or when the person who knew the settings leaves.
The recorded hyperparameter config, applied to the recorded data version, reproduces the deployed model to within run-to-run noise.
holds when Every resolved value — set or defaulted — is logged at run time from the library's own view of its settings, and the retraining job reads the config from the record rather than from code.
breaks when A dependency upgrade changes a default; a setting is passed on the command line and not logged; the retraining job carries a hard-coded value that drifted from the record.
respond Fix the record before touching the model. A model that cannot be rebuilt cannot be compared to its successor, so no retraining decision is trustworthy until it can.
How to build it
Most important first.
- Write every hyperparameter into a config that is versioned with the run, and log the resolved values — including the defaults you did not touch — in the experiment record (Experiment Tracking). A default is a choice; a library upgrade can change it silently.
- Separate the settings into those that control capacity (depth, λ, layers, rounds), those that control optimisation (learning rate, batch size, optimizer) and those that control the decision (threshold, class weight). They are tuned in different orders and judged by different numbers.
- Judge capacity settings on validation, never on training loss, and never on the test set (Never Tune on the Test Set). The test set is opened once, after every setting including the threshold is frozen.
- Prefer a small number of settings you understand over a large number you do not. Each setting you tune is one more degree of freedom the validation set has to pay for.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The validation metric at the operating point, reported alongside the resolved hyperparameter config and the data version, so the number can be attributed to a specific run.
- The gap between training and validation loss as a function of the capacity settings — this is what tells you whether depth or λ is the lever (Learning Curves).
- Do not measure "which model is better" across two runs whose settings differ in a dozen places. That number attributes to nothing.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The resolved hyperparameter values recorded for the deployed model are exactly the values that produced it, including every library default, and a retraining job reads them from the record rather than from a notebook.
- The data the hyperparameters were chosen on resembles the data the model will be retrained on; a capacity setting tuned on two hundred thousand rows is not automatically right for two million.
- The validation set used to choose the settings was not also the set used to report the final number.
- Offline: retrain from the recorded config and data version and assert the validation metric matches the recorded one within run-to-run noise (Reproducibility). If it does not, the record is incomplete.
- Online: the deployed artifact carries its config hash; the serving system logs it with every prediction so an incident can be traced to a specific set of settings (Model Lineage).
- Over time: when retraining on new data, hold the config fixed and change only the data first. A drop attributable to the data is a different problem from one attributable to a setting that no longer fits.
What can go wrong
- A library upgrade changes a default (a regularisation term, a histogram bin count) and the retrained model differs from the recorded one; the config recorded only the settings that were set explicitly.
- The config is versioned but the search that produced it is not, so nobody knows the winning config was the best of four hundred trials against one validation set (Evaluation Leakage).
- Two settings interact — learning rate and number of rounds, batch size and learning rate (Batch Size and Learning Rate) — and a "small" change to one in production retraining moves the other out of its working range.
- Logging every resolved default makes the record long and mostly boring; the alternative is a record that silently depends on a library version.
- Separating capacity, optimisation and decision settings is a discipline that costs a little structure in the config and saves a lot of confusion about which number is allowed to judge which knob.
- Treating the threshold as a hyperparameter means it is chosen on validation like the others, which is correct and also means the validation set is spent one more time.
- "The defaults are fine, the algorithm authors chose them." They chose them for a benchmark. The default depth of a tree library is a fact about that library's tests, not about your data.
- "We tuned on validation, so the validation number is our estimate of production." It is an upper bound. The validation set chose the winner and cannot judge it impartially; that is what the test set is for.
- "Learned parameters and hyperparameters are the same thing at different levels." At the level of the record, they are opposites: parameters are reproduced by re-running training, hyperparameters must be written down or they are lost.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat capacity settings cannot be judged on training loss follows from what training loss is — it always prefers more capacity — so the validation-set rule applies to every model family from ridge regression to a transformer.
- MODEL-SPECIFICWhich settings matter differs sharply by family: for gradient boosting the learning rate, depth and number of rounds dominate; for a neural network the learning rate schedule and batch size; for k-NN essentially only k and the distance metric. A search that treats them all equally wastes most of its budget.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Programming Languages & Runtime Internals — a "default" is a value the library resolves at call time, and resolving it explicitly at run time (rather than reading the source) is the only record that survives an upgrade.