Gradient Synchronisation
Workers agree on a gradient by all-reduce — a ring exchange that is bandwidth-optimal — and the choice between waiting for everyone and not waiting decides staleness, straggler exposure, and whether two runs can ever produce the same bits.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
N workers each have a gradient and all of them need the average — how does that exchange work, what happens when one worker is slow or dead, and why is the result never bit-identical between runs?
A vision team's sixteen-GPU run across two nodes gets through a step in about the time one GPU takes, which is not the sixteen-fold they planned for. Their monitoring shows every GPU "busy". Separately, two runs from the same seed produce different loss curves, and someone has filed it as a bug.
Every worker sends its gradient to worker 0. Worker 0 averages and sends the result back. Sixteen workers, sixteen messages in, sixteen out, done.
Worker 0 receives fifteen gradients — six gigabytes — through one network interface every step, then sends six gigabytes back. Its link is saturated while the other fifteen wait; the exchange takes longer than the compute.
- Worker 0 receives fifteen gradients — six gigabytes — through one network interface every step, then sends six gigabytes back. Its link is saturated while the other fifteen wait; the exchange takes longer than the compute.
- The throttled GPU finishes its gradient last on every step, and the average cannot be computed until it arrives. Fifteen workers idle for its lag on every step, and the run proceeds at the pace of the slowest card (One Slow Task Sets the Pace for Everything).
- When that GPU's node is preempted, worker 0 waits forever for a gradient that will never come. The run hangs rather than fails, and the hang is discovered by the morning bill.
- Two runs from the same seed differ in the third decimal of the loss after a hundred steps. Floating-point addition is not associative; the order in which sixteen gradients arrive and are summed differs between runs, and the difference compounds (Reproducibility).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model classifies product images; the label is a taxonomy node assigned by a merchandising team (Classification). This lesson's target is the averaged gradient every worker needs before it can take the next step, produced correctly and quickly enough not to dominate the step.
- Correct means the average of what every worker computed; quickly means the exchange takes a small fraction of the compute time per step; and "every worker" is where the difficulty lives.
- Sixteen workers, each holding the full model — roughly a hundred million FP32 parameters, four hundred megabytes of gradient — and each producing that gradient once per step.
- Eight workers per node on a fast intra-node interconnect; the two nodes joined by a network an order of magnitude slower.
- One GPU on node two is thermally throttled and runs each step noticeably slower than the rest.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- All-reduce computes the sum (or mean) of a vector held on every worker and leaves the result on every worker. The general algorithm belongs to Distributed Systems; the version that matters here is ring all-reduce: the workers form a ring, the gradient is cut into N chunks, and in N−1 steps each worker sends one chunk to its neighbour and adds what it receives (reduce-scatter), then in N−1 more steps the reduced chunks circulate so every worker ends with all of them (all-gather).
- Each worker sends and receives 2·(N−1)/N times the gradient size in total — under twice the gradient, regardless of N. No worker's link carries more than that. That is what bandwidth-optimal means: the exchange time is bounded by the gradient size over one link's bandwidth, not by the number of participants. The cost that grows with N is latency — 2·(N−1) sequential steps — which is why hierarchical schemes reduce within a node first and across nodes second.
- Synchronous updates wait for every worker's gradient before anyone steps, so all copies stay identical and each step uses a fresh gradient. The price is that the step runs at the pace of the slowest worker and stalls if one dies. Asynchronous updates let each worker push its gradient to a parameter store and pull the latest weights without waiting; no straggler stalls anyone, but a gradient computed against weights that other workers have since updated is *stale*, and stale gradients push the model in directions that were right several steps ago.
- Floating-point summation depends on order. Ring reduction sums chunks in ring order, which is deterministic for a fixed ring, but the collective library chooses algorithms and chunking by message size and topology, and hardware-level accumulation on the GPU is itself non-deterministic unless forced otherwise. Two runs of the same code thus produce averaged gradients that differ in the last bits, and gradient descent amplifies last-bit differences into visibly different trajectories.
Ring all-reduce: nobody's link carries more than twice the gradient
Sending every gradient to one worker makes that worker's link the bottleneck and the exchange time proportional to N. A ring spreads the work: the gradient is cut into N chunks, each worker adds its neighbour's chunk to its own and passes it on, and after N−1 steps each chunk's full sum sits on one worker. N−1 more steps circulate the sums. Every worker sends and receives just under twice the gradient in total, whatever N is.
The cost that remains is latency: 2·(N−1) sequential hops, each paying the interconnect's per-message overhead. For a small gradient at large N that term dominates, which is why libraries switch to tree schemes for small messages and reduce hierarchically — within a node, then across — when the links differ in speed. The general theory lives in Distributed Systems; the training-specific fact is that the exchange volume per worker is bounded by the model size, so the way to scale is to make each step's compute large enough to hide it.
Wait for everyone, or do not
Synchronous updates are a barrier: nobody steps until everyone's gradient is in. The copies stay identical, each gradient is computed against the weights it will update, and the run behaves like one large-batch device — which is also why the slowest worker sets the pace and a dead one stops the run. The remedies are a timeout that turns a hang into a failure, and an honest look at why one worker is slow.
Asynchronous updates remove the barrier and pay for it with staleness. A worker that pulls weights, computes for a while and pushes its gradient is pushing a gradient of a loss surface that other workers have since moved. Small staleness is a mild perturbation; large staleness is a different algorithm with its own convergence behaviour. It earns its place on genuinely heterogeneous hardware and rarely elsewhere.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| One GPU thermally throttled | Every step takes the slow card's time; scaling efficiency far below the design. | Synchronous barrier waits for the slowest gradient. | Fix or replace the card; a backup-worker scheme that drops the slowest contributor if it is structural. |
| A node preempted mid-step | The run stops making progress and keeps billing; GPUs show busy. | The collective waits for a participant that will never answer. | A timeout on every collective; fail the job; resume from checkpoint on a fresh allocation. |
| Inter-node network shared with another job | Step time doubles at the same hour each night. | The cross-node phase of the hierarchical all-reduce is bandwidth-starved. | Correlate step time with network metrics; isolate the link or schedule around it. |
| Same seed, different curves | A reported regression cannot be reproduced; or a real one is dismissed as noise. | Reduction order and non-deterministic kernels differ across runs; the difference compounds. | Deterministic mode while debugging; several seeds and a tolerance for any comparison. |
Why the bits differ
Floating-point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bit when the magnitudes differ. A ring sums each chunk in ring order; a tree sums in tree order; the GPU's own reduction over a block of threads sums in whatever order the hardware schedules unless forced to be deterministic. The averaged gradient is therefore the same number to many decimal places and not the same number.
Gradient descent is a dynamical system that amplifies small differences. Two runs whose first averaged gradient differs in the last bit take slightly different steps, land on slightly different points, compute more different gradients, and after a few thousand steps have visibly different loss curves. Neither run is wrong. The same code on the same seed is a distribution of results, and comparison protocols have to be built for that (Random Seeds).
Run-to-run variation from reduction order is small relative to the effect sizes the team acts on, and the comparison protocol accounts for it.
holds when The team has measured the variance across several seeds on the current configuration and requires an effect larger than it before promoting or reverting a change.
breaks when Someone compares one run against one run; the cluster or library version changes and the variance with it; a deterministic mode is switched off and nobody re-measures.
respond Re-run with several seeds before believing either a regression or an improvement. Switch on deterministic mode only to hunt a specific bug.
1import numpy as np2 3rng = np.random.default_rng(0)4g = rng.standard_normal(1_000_000).astype(np.float32) * 1e-35g[0] = 1.0 # one large component among many small ones6 7ring_order = np.sum(g) # left-to-right8tree_order = np.sum(g.reshape(1000, 1000).sum(axis=1)) # pairwise blocks9print(ring_order == tree_order) # False, in general10print(abs(ring_order - tree_order)) # last-bits difference11 12# In training this difference enters the parameters on step 113# and is amplified by every step after it.The point is not the size of the difference — it is tiny — but that it enters a feedback loop. A comparison that treats one run per configuration as a measurement is comparing two draws from two distributions and calling the difference a result.
How to build it
Most important first.
- Use a collective library's all-reduce, hierarchically: reduce within each node over the fast link, exchange the per-node results across nodes, then broadcast back within nodes. Never route everything through one worker.
- Overlap communication with computation. Gradients for the last layers are ready while the first layers are still in backward; start reducing them in buckets as they arrive so the exchange hides behind the remaining compute.
- Stay synchronous unless stragglers are structural. Synchronous training keeps every copy identical and every gradient fresh; handle stragglers by fixing the slow hardware, by a backup-worker scheme that drops the slowest, or by bounded-staleness schemes only when the workload genuinely has permanent slow participants.
- Put a timeout on every collective and make a missing worker a failure, not a hang. A hung run costs the full cluster until someone notices (Partial Failure: When 3 of 5 Succeed).
- Decide the reproducibility standard on purpose: bit-exact costs throughput (deterministic kernels, fixed algorithms, disabled autotuning) and is worth paying for while debugging; statistical reproducibility — several runs, compare the distribution — is the standard for production training.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Communication time per step, measured as the interval between the last gradient being ready and the update starting, against compute time per step. The ratio decides whether adding workers can help.
- Per-worker step time and its spread. The slowest worker's time is the step time under synchronous updates; the spread is the straggler cost.
- For asynchronous schemes: the staleness distribution — how many updates elapsed between the weights a gradient was computed on and the weights it was applied to.
- Do not read GPU utilisation as progress. A worker blocked in a collective is busy and idle at the same time.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The all-reduce completes within a bounded time each step, with every worker's gradient included — a timed-out or missing participant fails the step rather than silently averaging over fewer workers.
- Under synchronous updates, every copy of the parameters is identical after every step; under asynchronous updates, the staleness stays within a bound the schedule was tuned for.
- Run-to-run variation from reduction order is small enough that the team's comparison protocol — several seeds, a tolerance — can distinguish a real model change from noise.
- Offline: compare the all-reduced gradient against a single-device gradient on the union batch to floating-point tolerance, and repeat the same step several times to measure the reduction-order variance so the tolerance is grounded in a measurement.
- Before launch: kill one worker mid-step and confirm the collective times out and the job fails visibly within the timeout, rather than hanging.
- Over time: a per-step timeline of compute versus communication per worker, so a straggler or a saturated link shows as a widening gap on one worker rather than as a slower run.
What can go wrong
- The straggler is not a bad card but a shared network link: another team's job saturates the inter-node network for an hour every night, and the run's step time doubles on a schedule nobody correlates.
- Bucketed overlap is configured with buckets too small, so the all-reduce latency term dominates and the overlap buys nothing.
- A bounded-staleness scheme is left in place after the slow hardware is replaced, and the model trains on stale gradients for no reason, converging worse than the synchronous run would.
- A bit-exact reproducibility setting is switched off for throughput in production and never switched back on for debugging, so a real regression is dismissed as run-to-run noise.
- Synchronous training is exposed to every slow or dead worker; asynchronous training trades that exposure for staleness, which changes the optimisation and is harder to reason about.
- Bit-exact reproducibility costs throughput and constrains the library's algorithm choices; statistical reproducibility costs several runs per comparison.
- Hierarchical, overlapped all-reduce is fast and also a piece of configuration — bucket sizes, group topology, timeouts — that has to be tuned per model and per cluster.
- "Every GPU shows busy, so the run is compute-bound." A GPU blocked in a collective reports busy. The step timeline, not the utilisation gauge, says where the time goes.
- "Same seed, different loss — there is a bug." There may be, but the same seed does not fix the reduction order, and reduction order changes the last bits, which gradient descent amplifies. Several runs per configuration is the comparison; one run each is not.
- "Go asynchronous so the slow worker stops holding everyone up." It stops holding everyone up and starts applying gradients that are several steps out of date, with a different convergence behaviour that has to be re-tuned. Replacing the slow card is usually cheaper.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe communication bound of ring all-reduce and the non-associativity of floating-point addition hold for every model and every framework; what varies is whether the exchange is a small or a dominant fraction of the step.
- SIMPLIFIEDThe ring is described in its basic form; production libraries pick among ring, tree and hierarchical variants per message size and topology, and the illustrative message counts in the section below are for the shape of the argument, not a model of any specific interconnect.
- CONTESTEDWhether asynchronous training deserves a place outside research is genuinely disputed. Its strongest case is that at very large worker counts, on heterogeneous or preemptible hardware, synchronous training spends most of its time waiting and a bounded-staleness scheme recovers most of the throughput with a modest, tunable quality cost; the strongest case against is that nearly every production run at ordinary scale uses synchronous updates because identical copies and fresh gradients are worth far more than the straggler time they cost.
Where the depth lives
This domain teaches the model and hands the rest off by name.