OptimisationGENERALMODEL-SPECIFIC

Vanishing and Exploding Gradients

Backpropagation multiplies one Jacobian per layer. A chain of factors below one shrinks the gradient to nothing by the early layers; a chain above one blows it up. Depth is hard to optimise for this reason, and every remedy attacks the product.

Target & data What to measure What must stay true ▶ Backpropagation Visualizer

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

The deep model trains worse than the shallow one it was supposed to improve on, and the early layers barely change. What is happening to the gradient on its way back?

The problem

A speech team replaced a six-layer model with a twenty-four-layer one expecting a better result and got a worse one: training loss plateaus higher than the shallow model's, and inspecting the weights shows the first ten layers are almost unchanged from initialisation after a full run.

The obvious approach

Deeper is more expressive. Stack more of the same layers, train with the same recipe, and expect at least the shallow result: the extra layers can always learn the identity if they are not needed.

Why it breaks

The extra layers cannot learn the identity because they never receive a usable gradient. Each saturating layer multiplies the backward signal by a factor well below one; twenty of them multiply it by something indistinguishable from zero. The early layers are frozen at their random initialisation, not by choice but by arithmetic.

How it breaks — usually after the offline metric looked fine

The extra layers cannot learn the identity because they never receive a usable gradient. Each saturating layer multiplies the backward signal by a factor well below one; twenty of them multiply it by something indistinguishable from zero. The early layers are frozen at their random initialisation, not by choice but by arithmetic.
A second attempt with a larger learning rate to "push the gradient through" makes the late layers, which do receive gradient, diverge; the loss goes to NaN and the depth is declared not to work on this data.
A third attempt with a different activation trains the early layers and produces exploding gradients in the recurrent part of the model, where the same weight matrix is multiplied in on every time step and any spectral radius above one compounds.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Train all layers of a deep network, so the capacity the depth adds is actually used; a deep network whose early layers never move is an expensive shallow network on top of a random feature extractor.
The surrounding system transcribes audio; its target is unchanged. Optimisation decides whether the deeper model can be trained at all.

Data

Long input sequences and a deep stack. The gradient of the loss with respect to an early layer's weights is a product of one factor per layer between it and the loss, and the number of factors is the depth.
The activations are saturating — a sigmoid-style nonlinearity — so each factor includes a derivative that is at most a quarter and near zero for large inputs.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

By the chain rule, the gradient at layer k of an L-layer network is the gradient at the output multiplied by the Jacobian of each layer from L down to k+1 (Backpropagation, Computational Graphs). Each Jacobian is a weight matrix times the diagonal of activation derivatives. The magnitude of the product behaves like a product of scalars: factors typically below one give exponential decay in depth, factors above one give exponential growth.
Saturating activations make the decay almost certain. The derivative of a sigmoid peaks at a quarter and is near zero wherever the unit is confidently on or off, so a layer of saturated units passes back a tiny fraction of the signal regardless of its weights. ReLU-family activations pass a derivative of one for active units, removing that factor from the product (Activation Functions).
Residual connections change the product into a sum: the output of a block is its input plus a learned correction, so the Jacobian is the identity plus something, and the gradient has a path back to any layer that is never multiplied by a small factor. That is the main reason very deep networks became trainable.
Explosion is the same product with factors above one, and it is the characteristic failure of recurrence, where one weight matrix is applied at every time step. Gradient clipping caps the norm of the update, which does not fix the product but keeps a single bad step from destroying the run.

The product

A gradient that must pass through twenty layers is multiplied twenty times. If the typical factor is a half, the early layers receive one millionth of the signal the late layers receive; if it is two, they receive a million times more. Neither is a bug in backpropagation. It is what a product does, and the only ways out are to make the factors close to one or to give the gradient a path that skips the multiplication.

The backpropagation visualiser at /ml/backprop shows the mechanism on a 2-2-1 network: each node's gradient is its downstream gradient times a local derivative, and the local derivative of a saturated sigmoid is the small number that starts the decay.

Why depth shrinks the gradient

1import math
2 
3def sigmoid_deriv(z):
4    s = 1 / (1 + math.exp(-z))
5    return s * (1 - s)          # at most 0.25, near 0 when |z| is large
6 
7# backward factor per layer ~ |w| * sigmoid'(z); take w = 1, z = 0 (the best case)
8factor = 1.0 * sigmoid_deriv(0.0)          # 0.25
9for depth in (2, 6, 12, 24):
10    print(depth, factor ** depth)           # 0.0625, 2.4e-4, 6e-8, 3.6e-15
11 
12# a residual block replaces the factor with (1 + factor): the product is anchored at 1
13# a ReLU replaces sigmoid'(z) with 1 for active units: the factor becomes |w|

This is the best case for the sigmoid — every unit at the point of maximum derivative. Real units are mostly saturated, the factor is far smaller, and twenty-four layers of it is zero in any floating-point format.

Which remedy attacks which factor

Each remedy changes one term of the product, and knowing which is the difference between adding all of them by habit and choosing the ones the architecture needs.

Remedy	Term it changes	Fixes	Introduces
ReLU-family activation	Activation derivative → 1 for active units	Vanishing from saturation	Dead units with exactly zero gradient
Residual connection	Jacobian → identity plus correction	Vanishing at any depth	Forward activation growth if blocks are not scaled at init
Variance-preserving init	Weight-matrix scale → factors near one at start	Both, at the fragile beginning	Nothing directly; drifts as weights train
Normalisation layer	Pre-activation range → derivatives healthy	Saturation and scale drift	Batch-size dependence and train/serve statistics (batch norm)
Gradient clipping	Update norm → capped	Explosion destroying a run	A hidden, varying effective learning rate when it fires often

Watching the norms after the defaults are in place

Modern defaults make the problem rare, and rare problems are found late. The gradient of an early layer is not something anyone looks at when the loss curve is behaving, so a layer that goes dark after an architecture edit is found when someone notices the model has stopped improving with depth.

The check is cheap: one number per layer, logged every few hundred steps. It turns a structural failure into a dashboard line.

must stay trueEvery layer still receives a usable gradient

The per-layer gradient norms stay within a bounded ratio of each other across the stack for the whole run, and the clip rate stays low.

holds when Residual paths, normalisation and a variance-preserving init are in place and unchanged; the activation family is non-saturating; the clip threshold is above the typical norm.

breaks when A block is inserted without a residual path; a normalisation layer is removed for latency; a gate with a saturating activation is added in the backward path; a data change raises the typical gradient norm above the clip threshold.

how you would know Per-layer gradient-norm logging with an alert on the first-to-last ratio; a dead-unit fraction per layer; the clip rate; and a smoke test that asserts the ratio on a fresh build (Training Smoke Tests).

respond Identify the layer where the norm collapses or grows and fix that term — the activation, the missing skip, the init scale — rather than raising the rate or the clip threshold.

How to build it

Most important first.

Use a non-saturating activation family as the default, and reserve saturating ones for gates and outputs where a bounded range is the point.
Give deep stacks residual connections and a normalisation layer per block (Normalisation Layers), so the backward path has an identity component and the forward activations stay in a range where derivatives are healthy.
Initialise weights so that each layer preserves the variance of its input and of its backward signal; a variance-preserving init keeps the factors near one at the start, which is when the product is most fragile (Initialisation and Convergence).
Clip the gradient norm as a safety net, with the clipping rate monitored — a run that clips on every step has an underlying problem that clipping is hiding.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Gradient norm per layer, logged through training. The diagnostic is the ratio between the first and last layers' norms: a ratio that collapses towards zero with depth is vanishing, one that grows is exploding. This one plot answers the problem's question.
The fraction of units with zero derivative per layer — saturated sigmoids, dead ReLUs — which shows where in the stack the signal is being cut off.
Do not diagnose from the loss curve alone. A plateau from vanishing gradients and a plateau from a too-small learning rate look the same at the loss; the per-layer norms distinguish them.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

The per-layer gradient norms stay within a bounded ratio of each other throughout training, so every layer keeps receiving an update of a usable size.
The fraction of dead or saturated units per layer stays small; a layer that goes dark mid-run has stopped learning even though the loss curve continues.
The clipping threshold is rarely reached; if the clip rate rises after a data or architecture change, the underlying product has changed and the run needs attention rather than a higher threshold.

How to verify — offline, online, and over time

Offline: on a freshly built architecture, run a few steps and check that the gradient norm at the first layer is within a small factor of the norm at the last; a variance-preserving init on a residual stack should give this before any training.
During training: log per-layer gradient norms and the clip rate every few hundred steps and alert on a first-to-last ratio drifting towards zero or on a clip rate above a few percent.
Over time: include a per-layer norm check in the training smoke test (Training Smoke Tests) so an architecture edit that reintroduces the product is caught before a full run.

What can go wrong

Failure modes in production

Clipping is set aggressively enough that every step is clipped; training proceeds but the effective learning rate is now the clip threshold divided by the norm, which changes every step, and the schedule means nothing.
A ReLU network suffers the opposite of saturation: a large early update pushes a layer's pre-activations negative for every input, its units are permanently off, and the gradient through them is exactly zero for the rest of training.
Residual connections are added but the block's output is scaled up at initialisation, so the sum of many blocks grows with depth in the forward pass and the activations explode before the gradients do.

What the recommended approach costs

Residual connections and normalisation add memory and compute per layer and constrain the architecture — a residual block must preserve shape, which is not free for every design.
Clipping is cheap and effective against explosion but hides its own overuse; someone must watch the clip rate.
Non-saturating activations remove the vanishing factor and introduce the dead-unit failure, which needs its own initialisation care and monitoring.

Misreads

"The deep model is worse, so the data does not have enough signal for a bigger model." The deep model was never trained: its early layers did not move. The comparison was between a shallow model and a shallow model on random features.
"The gradient vanished, so raise the learning rate." A larger rate multiplies a signal that is already nearly zero in the early layers and a healthy one in the late layers; it diverges the late layers first.
"We added clipping and the NaNs stopped, so the problem is solved." Clipping stopped the overflow. If it fires on most steps, the run is being driven by the threshold and the product that caused the explosion is still there.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThe product-of-Jacobians argument is the chain rule and applies to any composed differentiable model of nontrivial depth; the remedies are architecture-specific but the failure is not.
MODEL-SPECIFICExplosion is characteristic of recurrent models, where one matrix is applied repeatedly per time step; vanishing is characteristic of deep feed-forward stacks with saturating activations. Transformers with residual connections and layer norm mostly avoid both by construction.