Backpropagation Visualizer
A 2-2-1 network as a computational graph. Every node shows its forward value and its backward gradient; the backward pass is the forward order walked in reverse, one local derivative at a time. Take one training step and watch the weights move — or a hundred, and watch XOR get learned.
A framework hides this, which is fine until it is not. Backpropagation is not a formula for the gradient of a network; it is a bookkeeping rule — every node knows its own local derivative, and the loss's gradient flows backwards through the graph by multiplying those together along each path. Watch the gradient on x₁ after the backward pass: nothing wrote it down, it accumulated. Then look for a hidden unit whose z is negative. Its ReLU outputs zero and passes zero gradient back, so nothing upstream of it will ever change. That is a dead unit, and in a network with two of them it is the difference between learning XOR and never learning it.
The sample and the step
The graph
Values flow left to right: inputs → pre-activations → ReLU → output logit → sigmoid → loss.
- 1x₁= 0.000
- 2x₂= 1.000
- 3z₁= 0.600
- 4z₂= -0.500
- 5h₁= 0.600
- 6h₂= 0.000
- 7z₃= 0.340
- 8ŷ= 0.584
- 9L= 0.538
What the network knows right now
| w1[0][0] | 0.800 | ∂ 0.000 | → 0.800 |
| w1[0][1] | 0.900 | ∂ -0.374 | → 1.087 |
| b1[0] | -0.300 | ∂ -0.374 | → -0.113 |
| w1[1][0] | 0.700 | ∂ 0.000 | → 0.700 |
| w1[1][1] | 0.600 | ∂ 0.000 | → 0.600 |
| b1[1] | -1.100 | ∂ 0.000 | → -1.100 |
| w2[0] | 0.900 | ∂ -0.249 | → 1.025 |
| w2[1] | -1.200 | ∂ 0.000 | → -1.200 |
| b2 | -0.200 | ∂ -0.416 | → 0.008 |
The last column is where one step at lr = 0.5 on this sample would put each parameter. Gradients are checked against finite differences in the tests — the analytic and numerical values agree to 1e-5, which is the only proof the graph is wired correctly.
Press “Train 100 epochs” to run full-batch gradient descent over the 4 samples and draw the loss. One training step updates on the selected sample only, which is what stochastic descent does one sample at a time.
How to read this page honestly
What the model is, and what it deliberately refuses to be.
- SIMULATEDA 2-input, 2-hidden (ReLU), 1-output (sigmoid) network with binary cross-entropy loss, built as an explicit computational graph. The backward pass is a literal reverse traversal of the forward order — each node hands grad × local derivative to its inputs. There is no closed-form gradient for the network anywhere in the code; ∂L/∂w₁ falls out of the chain rule one edge at a time.
- SIMULATEDThe analytic gradients are checked against central finite differences — every parameter perturbed by ±1e-6 and the forward pass re-run — and the test pins agreement to 1e-5. That check is the only proof the graph is wired correctly, and it is the check a framework runs for you and you should still know exists.
- SIMPLIFIEDThe network is small enough to fail. A 2-2-1 ReLU network has exactly enough capacity for XOR and no spare; from a poor initialisation one hidden unit dies (z < 0 for every input, so its gradient is zero forever) and it never trains. The starting weights here were chosen so that XOR *does* train — and the AND set is the linearly separable case that trains from almost anywhere. The difference is the lesson on capacity and initialisation, and it does not scale up: a 2-2-1 network says nothing about a transformer's optimisation.
Take it further
What the optimizer does with the gradient once it has it.
Backpropagation →The chain rule as a traversal, and why no closed form is needed.
Computational Graphs →What a framework builds so that this walk can be automated.
Vanishing and Exploding Gradients →What happens to ∂ along a path of a hundred nodes instead of six.