OptimisationGENERALSIMULATEDSIMPLIFIED

Gradient Descent

θ ← θ − η∇L. One update rule, one number that decides whether it crawls, converges or explodes. The learning rate is the single most important hyperparameter in deep learning.

Target & data What to measure What must stay true ▶ Gradient Descent Visualizer

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

The loss curve is flat, or it jumps to NaN after a few hundred steps. Before touching the architecture or the data, what does the update rule itself predict about this behaviour?

The problem

A team fine-tuning a document classifier reports that "the model does not learn": the training loss barely moves over a day of GPU time. A second run with a configuration copied from a blog post drives the loss to NaN in twenty minutes. They want to know whether the data is broken.

The obvious approach

Pick a learning rate that worked elsewhere, run for a fixed number of epochs, and judge the result by the final loss. Optimisation is a solved problem inside the framework; the interesting decisions are the architecture and the data.

Why it breaks

The copied learning rate was chosen for a different loss scale, batch size and model width. On this model each step overshoots the minimum along the steepest direction, the loss grows, the gradient grows with it, and within a few hundred steps a float overflows: the NaN is the update rule diverging, not the data.

How it breaks — usually after the offline metric looked fine

The copied learning rate was chosen for a different loss scale, batch size and model width. On this model each step overshoots the minimum along the steepest direction, the loss grows, the gradient grows with it, and within a few hundred steps a float overflows: the NaN is the update rule diverging, not the data.
The "does not learn" run used a learning rate so small that the loss falls by an amount invisible on the plot. A day of compute moved the weights a tiny distance in the right direction. The data was fine; the step was the problem.
Both failures look like data or model problems in the dashboard. Neither is. A day was spent auditing the corpus for the flat run and rewriting the tokeniser for the NaN run, and the fix was one number.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Minimise a scalar loss L(θ) — cross-entropy over the training examples — as a function of the parameter vector θ. The target of training is not "a good model" but a point in parameter space where the loss is low and the gradient is near zero.
The surrounding system predicts a document category; that is unchanged by anything in this lesson. Optimisation decides only whether the weights that would make the prediction good are ever reached.

Data

Training examples enter only through the gradient: at each step the optimiser sees a vector ∇L(θ) computed from a batch, never the documents themselves. Any property of the data that matters for optimisation matters through the scale and direction of that vector.
Feature scale is therefore an optimisation fact, not a preprocessing nicety. A feature in the thousands next to one in the hundredths gives the loss surface a long narrow valley, and no single step size suits both directions (Bucketing & Normalisation).

How it actually works

Precisely enough to predict its behaviour — not a framework API.

For a differentiable loss, the gradient ∇L(θ) points in the direction of steepest local increase. The update θ_new = θ_old − η · ∇L(θ_old) moves against it by a distance proportional to the learning rate η times the gradient magnitude. That is the whole algorithm; everything else in this module is a modification of what multiplies the gradient.
Along a direction with curvature c (the second derivative), a step of η is stable only when η < 2/c. Below that the iterate approaches the minimum geometrically; above it each step lands further away than it started, and the loss oscillates outward. The largest curvature anywhere in the surface therefore caps the learning rate for the whole model, even if most directions are gentle.
Stochastic gradient descent replaces ∇L with an estimate from a batch, so each step carries noise whose size scales with η and shrinks with batch size (Batch Size and Learning Rate). The noise is not only a nuisance: it is what lets the iterate leave sharp minima and saddle points that full-batch descent would settle into.
The visualiser at /ml/descent shows this on three surfaces: a bowl, where any reasonable rate converges; a badly-conditioned valley, where the stable rate is set by the steep wall and the flat floor is crossed in tiny steps; and a non-convex ravine, where the starting point decides which minimum is found.

One rule, one number

Every deep learning framework hides the same line: subtract the learning rate times the gradient from the weights. Momentum, Adam, schedules and clipping all change what multiplies the gradient, but the shape of the rule does not change, and nor does its main sensitivity: too small and nothing visible happens, too large and the iterate bounces out of the valley it was in.

The interesting fact is that "too large" is set by the *sharpest* direction of the loss surface and "too small" by the *flattest*. When those differ by orders of magnitude — a valley — there is no good single rate, and that is the situation input normalisation, initialisation and adaptive optimisers exist to avoid.

The update, written out

1def sgd_step(params, grads, lr):
2    # params, grads: same shape; lr: the learning rate
3    # nothing else happens in vanilla gradient descent
4    return [p - lr * g for p, g in zip(params, grads)]
5 
6# stability along a direction with curvature c (second derivative):
7#   step of lr moves the error from e to (1 - lr * c) * e
8#   |1 - lr * c| < 1   <=>   0 < lr < 2 / c
9# the steepest direction anywhere sets the ceiling for the whole model

The comment is the lesson. A learning rate is not "how fast to learn" but a fraction of the local curvature, and the largest curvature in the surface — often from one badly scaled input — decides the ceiling for every parameter.

Reading a loss curve as a diagnosis

The same three curves recur in every training run and each has a first hypothesis. A curve that is flat from the start with a healthy gradient norm is a step that is too small. A curve that falls, then oscillates with growing amplitude, is a step that is too large for the region the iterate has reached. A curve that drops to NaN is divergence that was allowed to overflow.

None of these say anything about the data or the architecture. They are properties of the update rule on this surface, and the visualiser lets you produce all three on a two-parameter bowl by moving one slider.

Loss-curve shapes and the learning-rate reading

Trigger	Symptom	Cause	Response
Flat loss, gradient norm healthy	Loss barely moves across an epoch; the plot looks like "no signal"	Learning rate several orders of magnitude too small for this loss scale	Log-scale sweep; expect the right rate to be far from the current one, not a small adjustment
Loss falls then oscillates with growing swings	Sawtooth curve; validation loss unstable between checkpoints	Step exceeds 2/curvature along the sharpest direction of the region reached	Lower the rate or add decay so the late, sharp region is entered with a smaller step
Loss rises steadily then NaN	Gradient norm grows every step before the overflow	Divergence from the first steps; often the warm-up is missing	Halve the rate and add warm-up; do not paper over with clipping alone
Loss plateaus high after a fast early drop	Curve looks converged but the model is poor	Iterate stuck in a flat region or saddle; the rate is fine, the geometry is not	Check the gradient norm; if near zero with high loss, see the vanishing-gradient lesson

What the tuned rate assumes

A learning rate is tuned against a surface, and the surface is a function of the data scale, the loss reduction, the batch size and the architecture. It is recorded as a single number in a config file, with none of those dependencies attached, and it is copied to the next project as though it were a constant of nature.

The assumption to make explicit is that the surface has not changed. It changes whenever the data changes scale, whenever a preprocessing step moves, whenever the batch size is raised to fill a bigger GPU, and whenever a layer is widened. Every one of those is a routine edit that nobody thinks of as an optimisation change.

must stay trueThe surface the rate was tuned on

The loss surface encountered during training has the same curvature scale as the one the learning-rate sweep was run on.

holds when Input normalisation, loss reduction, batch size, model width and the range of the training data are unchanged since the sweep.

breaks when A retraining run ingests new extreme examples; someone doubles the batch size to use a larger GPU without touching the rate; a normaliser is refitted on data with a different range; the loss switches from mean to sum reduction.

how you would know Gradient-norm and loss logging over the first thousand steps of every run, with an alert on a monotonically rising norm or on a loss that has not fallen by the end of warm-up; a training smoke test that asserts the loss decreases over a short run (Training Smoke Tests).

respond Re-run the sweep, not the full training. Record the rate together with the batch size and the data version so the dependency is visible the next time one of them changes.

How to build it

Most important first.

Sweep the learning rate first and on a log scale — over several orders of magnitude, with a short run per value — before any other hyperparameter. The loss after a few hundred steps against the rate has a characteristic shape: flat, falling, falling fastest, then diverging. Pick from the falling-fastest region, slightly below its edge (Hyperparameters).
Normalise inputs and use an initialisation that keeps activations at a sensible scale, so the loss surface has roughly similar curvature in every direction and one learning rate can serve all of them (Initialisation and Convergence).
Warm the rate up from a small value over the first steps and decay it towards the end; the early, large-gradient steps are the ones that diverge, and the late, small-gradient steps benefit from a finer step (Initialisation and Convergence).
Treat the learning rate as a function of the batch size and the model, not as a constant to copy. A configuration from elsewhere is a starting point for the sweep, never its result.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Training loss against step, on a log-scale y axis, over the first few hundred steps at each candidate rate. The shape of that curve — not its final value — is the diagnostic: flat means too small, oscillating or rising means too large.
The gradient norm per step. A norm that grows across steps is divergence in progress and predicts a NaN before it happens; a norm that collapses to near zero with the loss still high is a vanishing-gradient problem, not a learning-rate one (Vanishing and Exploding Gradients).
Do not judge a learning rate by validation accuracy at the end of a long run. That number folds together the rate, the schedule, the early-stopping point and the seed, and cannot say which of them was wrong.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

The loss surface the learning rate was tuned on is the one training runs on: the same input normalisation, the same loss scale, the same batch size and the same architecture width. Changing any of these silently moves the stable range.
The gradient the optimiser receives is the true gradient of the stated loss, not corrupted by a mixed-precision overflow, a mis-scaled loss reduction (sum versus mean over the batch) or a frozen layer that should be training.
The maximum curvature encountered during a run is close to what the sweep saw. A retraining run on data with new extreme examples can violate this without any change to the code.

How to verify — offline, online, and over time

Offline: before a long run, run the log-scale rate sweep for a few hundred steps each and plot loss against rate. Assert that the chosen rate sits in the falling region, not at the diverging edge.
Online in the run: log the gradient norm and the loss every step for the first thousand steps and alert on a norm that grows monotonically or a loss that increases for more than a handful of consecutive steps.
Over time: keep the sweep result with the experiment record (Experiment Tracking) and re-run it when the data, the batch size or the model width changes, because the stable range moved with them.

What can go wrong

Failure modes in production

A rate that was stable on the small pilot dataset diverges on the full one, because the full data contains a few examples with much larger gradients — long documents, extreme label values — that raise the maximum curvature.
The sweep is run for too few steps and picks a rate that falls fastest early but oscillates later, when the iterate reaches a sharper region of the surface.
Gradient clipping is added to stop the NaN and hides a rate that is fundamentally too large: training now proceeds, but every step is clipped, the effective rate is uncontrolled and the final model is worse than a properly tuned one.

What the recommended approach costs

A learning-rate sweep costs several short runs before the real one. On a model that takes days to train, that is hours of GPU time spent on runs that are thrown away — and it is still cheaper than one diverged full run.
Choosing a rate below the edge of stability is safe but slow; choosing at the edge is fast on the sweep and fragile on the full data. The margin is a judgement, not a formula.
Adaptive optimisers (Optimisers: SGD, Momentum, Adam) widen the range of rates that work, at the price of a second set of hyperparameters and a different set of minima.

Misreads

"The loss is flat, so the model has no signal to learn from." Flat is also what a tiny learning rate looks like. Check the gradient norm: if it is healthy and the loss is flat, the step is too small, not the data.
"The loss went to NaN, so there is a bug in the data pipeline." Divergence produces NaN without any bug. Halve the rate and retry before auditing the pipeline.
"We used the learning rate from the paper, so it is correct." The paper's rate goes with the paper's batch size, loss scaling, warm-up and model. Any one of those differing makes the number a rumour.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThat the step size interacts with the local curvature applies to any model trained by gradient descent — linear models, boosted trees with a shrinkage rate, and deep networks alike; the numbers differ, the failure shapes do not.
SIMULATEDThe surfaces and the behaviour quoted here come from the visualiser at /ml/descent, a two-parameter model written for Engineer Atlas. Real loss surfaces have millions of dimensions and no picture; the shapes are illustrative of the argument, not measurements.
SIMPLIFIEDThe stability condition η < 2/c is exact for a quadratic and a local approximation elsewhere; with momentum, adaptive scaling and stochastic gradients the true bound moves, but the qualitative picture — a maximum curvature caps the rate — survives.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Computer Architecturegpu-parallelism

Domains that do not exist yet

— Numerical analysis — the stability condition on the step size is the same one that governs explicit integrators for differential equations, and the reason a stiff system needs a tiny step is the reason a badly conditioned loss does.