Neural Networks

The neuron, activations, the forward pass, loss functions and backpropagation on a computational graph — the mechanism every deep learning framework hides.

Neural Networks
▶ lab

Input → linear layer → activation → hidden layers → output. A stack of learnable linear maps with nonlinearities between them, trained by gradient descent on a loss. Not always better.

Q · A network is a composition of linear maps and nonlinearities — what does that structure let it learn that a linear model cannot, and what does it demand in data, compute and monitoring in return?
The Neuron
▶ lab

z = w·x + b, then activation(z). One unit is a logistic regression; a layer is many of them sharing an input; the network is the same thing composed.

Q · A single unit computes a weighted sum, adds a bias, and applies a nonlinearity — what does each of the three parts do, and what does a unit assume about the scale of its inputs?
Activation Functions

ReLU, sigmoid, tanh, GELU. Nonlinearity is what stops a stack of linear layers collapsing into one; the choice decides which gradients survive the trip back.

Q · Why does a network need a nonlinearity between its layers at all, and how does the choice of activation decide whether the gradients reach the early layers?
The Forward Pass
▶ lab

Input → layers → prediction, as a sequence of matrix multiplications with a batch dimension. This is where the FLOPs go, and where inference cost is decided.

Q · A prediction is a chain of matrix multiplications — where does the compute go, what does the batch dimension change, and why does the same network cost so differently on different hardware?
Loss Functions
▶ lab

Prediction vs target → loss. MSE, binary and categorical cross-entropy, ranking losses. The loss is what the optimiser minimises; it is not the metric the business cares about, and the gap is the design.

Q · The optimiser minimises the loss, the business watches a metric, and the two are different functions — how is each common loss defined, what does it reward, and where does the proxy diverge from the objective?
Backpropagation
▶ lab

Forward pass → loss → backward pass → gradients → parameter update. The chain rule applied node by node, in reverse topological order, on the 2-2-1 network the lab runs.

Q · How does a network compute the gradient of one loss with respect to every parameter in a single backward sweep, and what can go wrong in that sweep that the loss curve will not show?
Computational Graphs
▶ lab

Nodes are operations, edges carry values forward and gradients backward. Reverse mode is cheap for many parameters and one loss, the framework builds the graph as you call it, and the activations it stores are the memory bill.

Q · Why does a framework record a graph of every operation, why is reverse mode the right direction for training, and why does the memory cost of training scale with the activations rather than the parameters?