The Neuron
z = w·x + b, then activation(z). One unit is a logistic regression; a layer is many of them sharing an input; the network is the same thing composed.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A single unit computes a weighted sum, adds a bias, and applies a nonlinearity — what does each of the three parts do, and what does a unit assume about the scale of its inputs?
A fraud analyst is being asked to trust a network's score. "Explain to me what one of these things actually computes. If it is just a weighted sum I already know how to think about that."
A neuron is a weighted sum with a nonlinearity on top, so the weights tell you what matters — a big positive weight on amount means large amounts look fraudulent. Read the first-layer weights the way you would read a logistic regression's coefficients.
The first-layer weights are on standardised inputs, so a weight on amount means "per standard deviation of amount", and the analyst's reading in currency is off by a factor of the amount's spread.
- The first-layer weights are on standardised inputs, so a weight on amount means "per standard deviation of amount", and the analyst's reading in currency is off by a factor of the amount's spread.
- The first layer's output is not the prediction; it feeds another layer. A large weight on amount in unit three means nothing until you know how unit three is used downstream, which may be to *suppress* a fraud score in combination with another unit.
- A unit with a ReLU activation that never receives a positive pre-activation outputs zero for every transaction — a dead unit — and its weights, however large, are irrelevant. The analyst was reading a dead unit as an important feature (Activation Functions).
- Unscaled serving inputs, after a standardiser bug, put every transaction far into the saturated region of the output sigmoid; the model returned scores near one for everything, with no error raised.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a card transaction is fraudulent, as a probability used to decide on manual review. The label is the chargeback, arriving weeks later.
- The analyst's need is to understand what the model computes well enough to know when to distrust it — which is a legitimate requirement on the model, not only on the explanation.
- One example is one transaction with a dozen numeric features — amount, time since last transaction, merchant-category risk score, distance from home — on very different scales, plus the delayed chargeback label.
- The amount feature ranges from a few cents to tens of thousands; time since last transaction from seconds to months. Neither is normally distributed.
- The network was trained on standardised features; the serving path receives raw values and applies the same standardisation from the artifact.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A unit computes z = w·x + b: a dot product of its weight vector with the input, plus a scalar bias. The weights say how much each input moves z; the bias shifts where z crosses zero, which for a ReLU is where the unit turns on and for a sigmoid is where the output is one half.
- The activation a = f(z) makes the unit nonlinear. With a sigmoid on a single unit fed by the inputs, the unit *is* a logistic regression — z is the log-odds and a is the probability (Sigmoid & Probability). With a ReLU, the unit is a hinge: zero below the threshold set by the bias, then linear.
- A layer is many units sharing the same input vector. Stack their weight vectors as rows and the whole layer is z = Wx + b, one matrix multiplication; the activation is applied elementwise. The output of the layer is the input of the next (The Forward Pass).
- Because z is a dot product with the raw scale of x, the unit's behaviour depends on input scale. A feature with a range in the thousands dominates z unless its weight is tiny; gradient descent will find those tiny weights slowly, if at all, which is why inputs are standardised and why the standardiser is part of the model (Bucketing & Normalisation, Initialisation and Convergence).
Three parts, three jobs
The weights decide direction and magnitude: which inputs push z up, which push it down, and how hard per unit of standardised input. The bias decides the threshold: for a ReLU, the value of w·x at which the unit switches from silent to linear; for a sigmoid, where the output crosses one half. The activation decides the shape of the response — hinge, S-curve, or the smoother variants.
A single sigmoid unit reading the inputs directly is a logistic regression, so an analyst who understands one already understands the unit. What changes in a network is that the unit's output is not a prediction but a feature for the next layer, and that its inputs may themselves be the outputs of other units.
1import numpy as np2 3def unit(x, w, b, f):4 z = w @ x + b # weighted sum: direction and magnitude, then threshold shift5 return f(z) # activation: the shape of the response6 7# a layer is the same thing for several units sharing one input:8def layer(x, W, b, f):9 return f(W @ x + b) # W rows are the units' weight vectors; f is elementwise10 11sigmoid = lambda z: 1 / (1 + np.exp(-z))12relu = lambda z: np.maximum(0, z)13 14# unit(x, w, b, sigmoid) is exactly a logistic regression on xThe dot product is with the input as given. If training standardised x and serving does not, w @ x is a different number for the same transaction and everything downstream is wrong without an error.
The unit that stopped firing
A ReLU outputs zero whenever w·x + b is negative. If the bias drifts negative enough during training that z is negative on every example, the unit outputs zero on all of them, its gradient is zero on all of them, and nothing will ever move it again. The weights stay wherever they were, which may be large, and an inspection of the weight matrix will still show them.
Saturation is the sigmoid's version: for large |z| the output is pinned near zero or one and the derivative is near zero. A layer of saturated sigmoids barely trains, and an output sigmoid fed inputs from outside the training range returns the same extreme score for everything.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Large learning rate or bad bias initialisation early in training | Validation loss plateaus early; many hidden units have activation rate near zero | Dead ReLUs: pre-activation negative on all inputs, gradient zero, no recovery | Lower the learning rate, re-initialise, or use a leaky variant; do not read the dead units' weights as meaningful |
| Serving inputs outside the training range, or an unstandardised feature | Scores cluster at the extremes for a slice of traffic; calibration on that slice collapses | Output sigmoid saturated: |z| large, derivative near zero, discrimination lost | Fix the scaling on the serving path; clip or flag inputs outside the training range |
| Deep stack of sigmoid or tanh layers | Early layers barely change during training; loss falls slowly then stalls | Gradients shrink through each saturating derivative on the way back | ReLU-family activations, normalisation layers, residual connections |
Reading a unit honestly
The analyst's question — what does it compute — has a precise answer for the unit and a harder one for the network. The unit computes a thresholded weighted sum of standardised inputs. The network computes a composition of those, and the meaning of any one weight depends on every layer after it. The first-layer weight on amount is not a coefficient on amount; it is a coefficient on amount *in the construction of feature three of layer one*.
What must stay true is the scaling contract. A unit's weights are correct for standardised inputs with the training fold's statistics; give it anything else and it is a different function.
Every feature reaching the first layer has been standardised by the training-fold statistics stored in the artifact, in the column order the weight matrix expects.
holds when The standardiser ships with the weights, the serving path applies it, and a contract test asserts the standardised vector for a known transaction.
breaks when The standardiser is refitted without a redeploy, a new feature is appended raw, the column order changes in the feature service, or an upstream unit change alters a feature's scale.
respond Fix the scaling on the serving path before considering retraining; a retrain on the same mismatch reproduces it with new weights.
How to build it
Most important first.
- Standardise inputs on the training fold and ship the standardiser inside the artifact; a unit trained on standardised inputs is a different function on raw ones (Preprocessing Lives in the Artifact).
- Explain the network through attribution over the whole function, not through first-layer weights; and tell the analyst what the attribution is — an approximation of a local gradient, not a coefficient (Explainability, Attribution Is Not Causality).
- Check for dead units and saturated outputs as part of training diagnostics: the fraction of examples on which each ReLU is active, and the distribution of the output pre-activation.
- For the analyst's need, keep a logistic regression on the same features as the interpretable companion; where the two disagree on a transaction is where the network is using an interaction the analyst should be told about.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-unit activation rate over a validation batch — units near zero are dead, units near one hundred percent are linear and adding no nonlinearity.
- Distribution of the output pre-activation z at serving time; a shift towards large magnitude means the sigmoid is saturated and the scores have stopped discriminating.
- Agreement between the network and the logistic companion at the operating threshold; the disagreement rate is the measure of how much the hidden layer is contributing.
- Do not measure "feature importance" by first-layer weight magnitude. It is scale-dependent and ignores everything downstream.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every input reaches the unit standardised by the same statistics used in training, so the weights mean what they meant.
- The units that carry the decision are alive — active on a meaningful fraction of inputs — and were alive at the end of training.
- The output pre-activation stays in the range where the sigmoid discriminates; inputs far outside the training range push it into saturation.
- The input feature set, in order and in scale, is exactly the one the weight matrix's columns correspond to.
- Offline: activation-rate histogram per unit and the output pre-activation distribution on a validation batch, recorded with the training run (Experiment Tracking).
- Online: a contract test on the serving path that feeds a known raw transaction and asserts the standardised vector and the score match the training-time computation (Serving Contract Tests).
- Over time: monitor the serving pre-activation distribution and the score distribution; a drift towards saturation is the unit-level symptom of input drift.
What can go wrong
- The standardiser is refitted on a new training window and the artifact is shipped without it; the serving path applies the old means and the units see shifted inputs.
- A large learning rate pushes a layer's biases negative early in training and half the units die permanently; the network trains to a worse solution and nothing reports it.
- A new feature is added in raw scale by someone who did not know the others were standardised; its weight dominates z on every transaction.
- The output sigmoid saturates on a subset of transactions with extreme amounts, and calibration on that slice is meaningless (Calibration).
- A network's units are cheap to compute and easy to state, and impossible to read individually; the understanding the analyst wants has to come from the whole function.
- Standardisation makes the units well-behaved and adds a component to the artifact that must be versioned and reproduced at serving time.
- Keeping a logistic companion doubles the models to maintain and gives the analyst something honest to compare against.
- "A big weight means an important feature." On standardised inputs, in a hidden unit, before downstream layers, it means very little. Attribution over the whole function is the honest version.
- "A neuron is like a biological neuron." It is a dot product and a hinge. The metaphor adds nothing to predicting its behaviour and invites overclaiming.
- "The bias is a minor term." The bias sets where a ReLU turns on and where a sigmoid is at one half; a badly initialised bias is the usual cause of a dead unit.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALz = w·x + b then an activation is the unit in every dense, convolutional and attention layer; convolutions share the weights across positions and attention computes the weights from the input, but each still ends in the same weighted sum.
- MODEL-SPECIFICThe scale sensitivity is a property of models that compute dot products with the inputs — networks, linear models, kernel methods; tree models split on thresholds and are indifferent to the scale of a feature.
- SIMPLIFIEDThe lesson treats one unit with a single input vector; in practice the input is a batch, the layer may be followed by a normalisation layer that rescales z, and the counts and activation rates mentioned are illustrative of the shape of the diagnostics, not measurements.
Where the depth lives
This domain teaches the model and hands the rest off by name.