Optimisation
Gradient descent, SGD, momentum and Adam; epochs, batches and steps; learning rate and batch size trade-offs; vanishing and exploding gradients; normalisation.
θ ← θ − η∇L. One update rule, one number that decides whether it crawls, converges or explodes. The learning rate is the single most important hyperparameter in deep learning.
Three ways to decide what multiplies the gradient. None is universally best: Adam converges fast and sometimes generalises worse; SGD with momentum is still the default in much of vision; the choice interacts with the learning rate and with weight decay.
An epoch is one pass over the data. A batch is the subset used for one gradient estimate. A step is one parameter update. "We trained for ten epochs" says nothing until you know the batch size.
Batch size trades memory, throughput and gradient noise against each other, and it moves the right learning rate with it. Larger batches want larger rates — up to a point — and small batches regularise for free.
Backpropagation multiplies one Jacobian per layer. A chain of factors below one shrinks the gradient to nothing by the early layers; a chain above one blows it up. Depth is hard to optimise for this reason, and every remedy attacks the product.
Batch norm normalises each feature over the batch; layer norm normalises each example over its features. The difference decides whether the layer behaves the same at training and inference — and batch norm does not, which makes it a train/serve skew source with a name.
Where the weights start decides whether training can begin; the warm-up and decay decide how it ends. A loss curve that plateaus, diverges or oscillates is a report on those choices, and a seed is not a reproducibility strategy.