Initialisation and Convergence
Where the weights start decides whether training can begin; the warm-up and decay decide how it ends. A loss curve that plateaus, diverges or oscillates is a report on those choices, and a seed is not a reproducibility strategy.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Two runs with the same config and different seeds ended far apart, and a third never left its starting loss. What did the initial weights and the schedule decide, and how much of the outcome is noise?
A team training a recommendation tower from scratch has a pipeline whose weekly retraining occasionally produces a model that is much worse than usual with no data change. One run in ten "does not converge"; two runs with the same data and config differ by more than the improvement the team is trying to ship.
Initialise with the framework default, fix a seed, train for the scheduled steps, and compare the result with last week's. If a run fails, re-run it with another seed; if two runs disagree, the better one is the better model.
The framework default was written for a different layer type and initialises the dense layers at a scale that, on this tower, makes the pre-activations either saturate or collapse. Most seeds happen to be fine; one in ten lands in a region where the early gradient is near zero and the run never leaves it. The failure is a property of the init scale, and the seed only selects whether it happens.
- The framework default was written for a different layer type and initialises the dense layers at a scale that, on this tower, makes the pre-activations either saturate or collapse. Most seeds happen to be fine; one in ten lands in a region where the early gradient is near zero and the run never leaves it. The failure is a property of the init scale, and the seed only selects whether it happens.
- The schedule has no warm-up. With Adam's moment estimates uninitialised, the first steps are large in random directions, and a seed that starts near a sharp region diverges in the first hundred steps. The run is restarted with a new seed and the cause is never found.
- "Pick the better seed" turns the comparison between weekly models into a comparison between seeds. The improvement being shipped is smaller than the seed-to-seed spread, so the team is selecting noise and calling it progress (Metric Uncertainty).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Reach a good minimum reliably, run after run, so that the difference between two retraining runs reflects the data and not the dice. The target is the reproducibility of the optimisation, not the value of one run.
- The surrounding system recommends items; a bad weekly model is served to every user for a week (Recommendation Systems).
- Sparse interaction data feeding a tower with several dense layers over embedding lookups. The early gradient signal is weak and the initial weights decide whether the first few hundred steps go anywhere.
- The seed controls the initial weights, the shuffle order, dropout masks and negative sampling, so "the same config" is at least four different random processes per run (Random Seeds).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- If every weight in a layer starts at the same value, every unit computes the same function and receives the same gradient, and they stay identical forever: symmetry is never broken. Random initialisation exists to give units different starting points, and the scale of that randomness decides what the forward and backward signals look like at step zero.
- A variance-preserving initialisation chooses the weight scale so that the variance of a layer's output roughly equals the variance of its input, given the fan-in and the activation; the same reasoning applied to the backward pass keeps gradient variance constant with depth. Get the scale wrong by a constant factor and, across many layers, activations and gradients grow or shrink geometrically (Vanishing and Exploding Gradients).
- Warm-up ramps the learning rate from near zero over the first steps, so the early updates — taken when the loss is steep, the adaptive moments are unestimated and the weights are wherever init put them — are small. Decay reduces the rate later so the iterate settles into a minimum instead of bouncing around it. Both are functions of the step index (Epoch, Batch, Step).
- The loss curve then reports the combination. A plateau at the starting loss is a dead start: the init scale or the rate left the gradient unusable. A steady fall that flattens high is a schedule that decayed too early or a rate too small for the late phase. Oscillation with growing amplitude is a rate above the stability limit for the region reached; a spike followed by recovery is a momentum overshoot; divergence is the rate at the start (Gradient Descent).
Symmetry and scale at step zero
Two things must be true before the first step. The units in a layer must differ, or they will never diverge; and the scale of the weights must keep the signal alive in both directions through the whole depth. Random initialisation handles the first for free and the second only if the scale is chosen for the layer's fan-in and activation.
The check is a forward and backward pass on one batch with no update, logging the standard deviation of activations and gradients per layer. It takes seconds and it distinguishes "the model cannot start" from every other explanation for a flat loss.
1import math, random2 3def init_weights(fan_in, fan_out, activation):4 # keep output variance ~ input variance for this activation5 gain = math.sqrt(2.0) if activation == "relu" else 1.06 std = gain / math.sqrt(fan_in)7 return [[random.gauss(0.0, std) for _ in range(fan_out)] for _ in range(fan_in)]8 9def step_zero_check(model, batch, tol=(0.1, 10.0)):10 # one forward + backward, no update: the scale at every depth should be near 111 acts, grads = model.forward_backward(batch)12 for i, (a, g) in enumerate(zip(acts, grads)):13 if not (tol[0] < a.std() < tol[1]) or not (tol[0] < g.std() < tol[1]):14 raise RuntimeError(f"layer {i}: act std {a.std():.3g}, grad std {g.std():.3g}")The gain is the whole content of the ReLU-specific formula: half the units are off, so the surviving half needs twice the variance. A default written for another activation is wrong by exactly that factor per layer, compounded over depth.
Reading the curve
A loss curve is the only thing most people look at, and it does carry the diagnosis if it is read together with the schedule plot and the per-layer norms. The table gives the readings in the order to try them.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Flat at the starting loss | No movement through warm-up; gradient norms tiny or huge at step zero | Init scale wrong for the layer or activation; symmetry not broken; rate orders of magnitude off | Step-zero scale check, then a log-scale rate sweep |
| Falls, then flattens well above the baseline | Rate already decayed when the plateau begins | Schedule decayed too early, or peak rate too low for the late phase | Lengthen the constant phase or raise the peak; compare against a longer-decay run |
| Oscillates with growing swings | Sawtooth in loss; validation unstable across checkpoints | Rate above the stability limit for the sharper region reached later in training | Start decay earlier or lower the peak; check that the served checkpoint is in the settled phase |
| Diverges in the first hundred steps | Loss and gradient norm rise from step one | No warm-up with an adaptive optimiser; init near a sharp region; peak rate too high | Add or lengthen warm-up before lowering the rate |
| Two seeds, same config, different endpoints | Held-out quality differs by more than the shipped effect | Seed spread of the recipe is large; the comparison is between draws | Run several seeds, report the spread, ship only differences outside it |
The pipeline that selects seeds
A weekly retraining pipeline that restarts on failure and promotes the better of two runs has, without anyone deciding it, become a seed-selection procedure. The improvements it ships are drawn from the spread; the failures it retries are a broken init nobody has looked at. Both are reproducible only in the sense that they keep happening.
The assumption to state and monitor is that the recipe's seed spread is known and small relative to what the pipeline is asked to detect. It is measured once per recipe change, and it is the number that turns "this week's model is better" from a hope into a claim.
The seed-to-seed spread of the training recipe is measured and smaller than the smallest improvement the pipeline is allowed to promote.
holds when The recipe has been run with several seeds since its last architecture, init or schedule change, the spread is recorded, and promotion requires a difference outside it (Promotion Is a Checklist, Not a Score).
breaks when A layer or activation is swapped without re-checking the init; the schedule is changed; the dataset shrinks so the run is noisier; the restart counter rises and nobody notices.
respond Re-measure the spread after any recipe change. If the weekly variance exceeds it, look for an init or schedule regression before looking at the data; if a claimed improvement is inside it, do not ship it.
How to build it
Most important first.
- Choose the initialisation per layer type and activation from the variance-preserving family, and verify at step zero that activations and gradients have a sensible scale at every depth before training.
- Use warm-up as a default for any adaptive optimiser and any large model, and pick the decay shape and end point with the same care as the peak rate; the schedule is the second most important hyperparameter after the rate.
- Run several seeds for any comparison that matters and report the spread with the mean; a difference smaller than the spread is not a result (Random Seeds, Metric Uncertainty).
- Make the retraining pipeline detect a dead or diverged start in the first few hundred steps and restart from a new seed automatically, with the event logged, rather than serving whatever the run produced (Training Smoke Tests).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Activation and gradient scale per layer at step zero, before any training; this is the number the init decides, and a bad init is visible here without spending a single step.
- Held-out quality across seeds at the end of the run: the mean is the recipe's quality and the spread is its reliability. The retraining pipeline's weekly variance should be no larger than this spread.
- Do not read "converged" off the loss curve alone. A run that plateaus high has converged, to a bad place; the per-layer gradient norms and the schedule plot say why.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The initialisation scale matches the current layer types and activations, so a fresh run starts with healthy activation and gradient scales at every depth — an assumption that breaks silently when a layer or activation is swapped.
- The seed-to-seed spread of the recipe is known and smaller than the effect sizes the pipeline is used to detect; a weekly model is not promoted on a single-run difference within that spread.
- The schedule reaches its decay phase before the checkpoint that is served, so the served weights come from a settled iterate rather than from a step in the high-rate phase.
- Offline: a step-zero check that asserts per-layer activation and gradient scales within a tolerance of one; a multi-seed run of the recipe whenever the architecture changes, recording the spread.
- During training: alert if the loss has not fallen by a set fraction by the end of warm-up, or if it rises for more than a few consecutive steps; both are cheap to detect in the first minutes.
- Over time: track the restart counter and the weekly held-out quality against the recorded seed spread; a weekly variance larger than the spread means something other than the seed is moving.
What can go wrong
- The seed is fixed for reproducibility and the run is still not reproducible, because data-loader threads, nondeterministic GPU kernels and a different library version all inject randomness the seed does not control (Reproducibility).
- The automatic restart-on-dead-start masks a broken init for months; the pipeline restarts one run in ten and nobody looks at the restart counter.
- The decay schedule is defined relative to the planned total steps, and a run that is stopped early by early stopping never reaches the low-rate phase; the checkpoint served was taken at a rate too high to have settled (Early Stopping).
- Multi-seed evaluation multiplies training cost by the number of seeds; for large models it is unaffordable, and the honest fallback is to report a single run as a single run with an unknown spread.
- Warm-up spends the first steps at a low rate — wasted compute if the start was fine, insurance if it was not.
- Automatic restarts make the pipeline robust and make the underlying init problem invisible; the restart count has to be a monitored number or the robustness hides a bug.
- "The run did not converge, so the model is too big for the data." It did not start. Check step-zero activation scales and the first hundred steps before drawing any conclusion about capacity.
- "We fixed the seed, so the runs are reproducible." The seed fixes the sampled randomness. Kernel nondeterminism, data-loader ordering across workers and library versions are not sampled from it.
- "This week's model beat last week's by a small margin, so the new data helped." If the margin is within the seed-to-seed spread, the comparison is between two draws. Run more seeds or ship nothing on it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALSymmetry breaking, variance preservation and the role of warm-up and decay apply to any deep network trained by gradient descent; the specific init formulas depend on the activation and layer type but the reasoning does not.
- SIMULATEDThe loss-curve shapes discussed here can be produced on the visualiser at
/ml/descentby choosing the start point, the rate and the optimiser; they illustrate the readings and are not measurements from any real training run. - CONTESTEDWhether multi-seed evaluation is worth its cost is a live disagreement. The strongest form of the "one run is enough" position is that for large models the seed spread is small relative to the effects being measured and the compute is better spent on a bigger model or more data; the reply is that the spread is only known to be small after it has been measured at least once per recipe, and that most teams asserting it have not measured it.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Programming Languages & Runtime Internals — nondeterministic reductions in GPU kernels and thread scheduling in data loaders are why a fixed seed does not fix a run; the mechanism lives in the runtime, not in the model.