Gradient Descent Visualizer
SGD, momentum and Adam on three surfaces, with a learning rate you can push until it diverges. Every point on the path is a real update; too small a rate crawls, too large a rate blows up, and which optimizer wins depends on the surface.
Training is walking downhill on a surface you cannot see, in a number of dimensions you cannot picture, taking steps whose size you chose before you started. The learning rate is the size of the step. Too small and the run takes forever; too large and each step overshoots the valley floor and lands higher than it started, and the loss goes to infinity — not gradually, but in a few steps. Between those, the shape of the surface decides everything: a round bowl needs nothing clever, a narrow valley punishes any step that is safe on its steep axis by crawling along its flat one, and that second case is the whole reason momentum and Adam exist.
Surface, optimizer, learning rate
A badly conditioned quadratic, 0.5·x² + 10·y². Curvature is 20× larger along y than along x, so any learning rate that is stable on y crawls along x — and any learning rate that moves quickly along x zig-zags or diverges on y. Condition number 20.
θ ← θ − lr·∇L. The step is the gradient, scaled. Fast when the surface is round, unstable above lr = 2/λmax, and slow along any flat direction.
The path
SGD on the valley at lr = 0.05, up to 200 steps.
Compare optimizers
All three on the valley at lr = 0.05. The same rate is fair to none of them, which is itself the lesson.
| Optimizer | Outcome | Steps | Final loss |
|---|---|---|---|
| SGD | converged | 117 | 4.90e-5 |
| Momentum (β = 0.9) | converged | 106 | 9.24e-7 |
| Adam (β₁ = 0.9, β₂ = 0.999) | converged | 184 | 4.62e-5 |
How to read this page honestly
What the model is, and what it deliberately refuses to be.
- SIMULATEDThree analytic surfaces, three textbook update rules with their textbook constants (momentum β = 0.9; Adam β₁ = 0.9, β₂ = 0.999, ε = 1e-8, with bias correction), 200 steps at most. Every point on the path is a real update from the analytic gradient. "Diverged" means the loss really passed 1e6 or became NaN — on the quadratic surfaces SGD is unstable exactly when lr ≥ 2/λmax, so lr = 2.1 on the bowl diverges for the same reason it would in a real run.
- CONTESTEDNo optimizer wins everywhere, and the page is allowed to show it. On the bowl plain SGD at lr = 1 reaches the minimum in one step while Adam crawls at roughly lr per step per coordinate and momentum overshoots. Adam and momentum earn their keep on the valley, where SGD must use a rate small enough for the stiff axis and therefore creeps along the flat one. "Just use Adam" is a default, not a law.
- SIMPLIFIEDTwo parameters, so the surface can be drawn. A real loss has millions of dimensions and cannot be plotted; the intuitions about curvature, conditioning and step size transfer, the picture of a bowl does not. Gradient noise here is seeded Gaussian noise on the gradient — a stand-in for mini-batch sampling, not a simulation of it.
Take it further
Where the gradient these optimizers consume actually comes from.
Gradient Descent →The step, the rate, and the stability limit that decides when it explodes.
Optimisers: SGD, Momentum, Adam →Momentum and Adam as answers to conditioning — and what each one costs.
Batch Size and Learning Rate →Why the two knobs are one knob, and how noise turns into a schedule.