Neural Networks
Input → linear layer → activation → hidden layers → output. A stack of learnable linear maps with nonlinearities between them, trained by gradient descent on a loss. Not always better.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A network is a composition of linear maps and nonlinearities — what does that structure let it learn that a linear model cannot, and what does it demand in data, compute and monitoring in return?
A product analytics team wants to predict which uploaded documents will need manual review. "The rules keep breaking. Someone said a neural network learns the rules itself, so we would not have to keep rewriting them."
Feed everything into a multi-layer network. It learns whatever features matter, so the analysts stop writing rules, and it gets better as more documents arrive. Deep learning has replaced hand features everywhere else.
The network trained on the tabular metadata alone did no better than logistic regression, at ten times the tuning effort. There was no representation to learn from six columns; the linear model already had it.
- The network trained on the tabular metadata alone did no better than logistic regression, at ten times the tuning effort. There was no representation to learn from six columns; the linear model already had it.
- On the text, a network trained from scratch on two hundred thousand documents overfit the vocabulary of the old reviewer pool and the training loss kept falling while validation stalled; the reviewers whose verdicts it learned had left (Overfitting).
- Serving a text model on every upload doubled the p99 latency of the upload path, because nobody asked what a forward pass costs on a CPU for a forty-page document (CPU or GPU for Inference).
- When a new upload source appeared with a different document template, the network's predictions shifted without any error, and the team had no coefficient to inspect and no rule to read.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an uploaded document will be sent back by a reviewer, from the document's content and metadata. The label is the reviewer's decision, known within a day.
- The decision is whether to route the document straight through or to a reviewer. A missed problem document reaches a customer; a needlessly reviewed one costs a few minutes.
- One example is one document: its extracted text, a page count, an upload source, and the reviewer's verdict. About two hundred thousand documents with verdicts, the text ranging from a paragraph to forty pages.
- The verdict is noisy — reviewers disagree with each other about a fifth of the time on a re-review sample — and the reviewer pool changed in composition over the two-year history.
- Metadata alone (source, page count) is tabular and small; the text is where the signal is expected to be, and it has no hand-engineered features.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A layer computes z = Wx + b — a linear map from its input vector to a vector of pre-activations — followed by an elementwise nonlinearity a = f(z). Stacking layers composes these functions; the output layer maps the last hidden vector to the prediction, through a sigmoid for a probability, a softmax for a distribution over classes, or nothing for a regression value.
- Without the nonlinearity the stack collapses: W₂(W₁x + b₁) + b₂ is itself a single linear map, however many layers are stacked. The activation is what lets the composition represent curved boundaries and interactions the inputs do not contain explicitly (Activation Functions).
- The parameters — every W and b — are found by minimising a loss over training examples with gradient descent. Backpropagation computes the gradient of the loss with respect to every parameter by applying the chain rule backwards through the layers (Backpropagation, Gradient Descent).
- What the network learns is a representation: the hidden vector is a set of features the training signal found useful. That is the whole advantage over a linear model on hand features, and it only pays off when the inputs are raw enough — text, pixels, audio, long sequences — that hand features are poor and the data is large enough to learn better ones (Raw Features vs Learned Representations).
What the stack is
Every layer does the same two things: a linear map, then an elementwise nonlinearity. The linear map is the learnable part — a weight matrix and a bias vector — and the nonlinearity is fixed. Input goes in as a vector, each hidden layer produces a new vector, and the output layer produces the prediction in whatever form the loss expects.
The picture is worth keeping literal, because everything later — the forward pass as matrix multiplications, backpropagation as the chain rule along these arrows, the memory cost of storing every activation — is a statement about this diagram.
1import numpy as np2 3def relu(z): return np.maximum(0.0, z)4def sigmoid(z): return 1.0 / (1.0 + np.exp(-z))5 6def network(x, params):7 W1, b1, W2, b2 = params # the learnable part, nothing else8 z1 = W1 @ x + b1 # linear9 h1 = relu(z1) # nonlinearity -- remove it and the next line10 z2 = W2 @ h1 + b2 # composes with this one into a single W @ x + b11 return sigmoid(z2) # probability for a binary decisionDelete relu and network becomes a logistic regression with a factored weight matrix. The nonlinearity is the entire difference between the two model families.
When the representation is worth learning
The advantage of a network is that the hidden vector is learned. On six tabular columns there is nothing to learn that the columns do not already say, and the network spends its capacity re-deriving what a linear model gets for free. On forty pages of text there is no hand feature that captures "this looks like a form the reviewer will reject", and a representation learned from data — or adapted from a pretrained one — is the only route to it.
The honest position for tabular data is unglamorous: a tuned tree ensemble or a linear model with a few interaction features usually wins, needs less data, serves on a CPU in microseconds and can be explained. The network earns its cost where the inputs are raw and the data is large.
Fine-tuned text model well ahead of the rule set on recall at the review threshold on a random split; training loss low and still falling at the end of the run.
Recall on documents from the two newest upload sources far below the offline number; upload p99 latency doubled; reviewers report the model passes a template it has never seen.
- 1The random split shared reviewer pools and templates between training and validation; a time-based split would have shown the drop on new sources.
- 2The "still falling" training loss was memorisation of the older reviewers' verdicts; validation loss had plateaued epochs earlier and nobody plotted it.
- 3The forward pass on forty-page documents was validated on a GPU and served on a CPU, and the latency budget was never stated.
What must hold after it ships
A network's weights encode a contract with its inputs that is stricter than a linear model's, because there is no coefficient to sanity-check and the learned representation is only meaningful for inputs like the ones it was trained on. The tokeniser, the preprocessing, the input distribution and the label semantics all have to stay where they were.
The compensating discipline is monitoring on the inputs and the predictions, not only on the delayed outcome. For a network that is the early-warning system, and it is the part most teams add last.
Serving-time inputs, after the same tokeniser and preprocessing, are drawn from a distribution close enough to training that the learned representation still separates the classes.
holds when Upload sources and templates are the ones in the training window; the preprocessing library version is pinned in the artifact; the drift monitors per source are quiet.
breaks when A new source or template appears; the tokeniser or a preprocessing dependency is upgraded on one side; the reviewer pool changes what "send back" means.
respond Route unseen sources to the rule fallback until they are labelled and included in a retrain; treat a preprocessing change as a model change with its own validation.
How to build it
Most important first.
- Start with the linear baseline on the tabular columns and a bag-of-words linear model on the text; the network must beat both at the operating point by more than the validation noise (The Linear Baseline).
- When the signal is in the text, adapt a pretrained representation rather than training one from two hundred thousand noisy labels; fine-tuning or a linear head on frozen embeddings is where the data budget goes furthest (Transfer Learning, Fine-Tuning).
- Decide the serving path before the architecture: batch scoring after upload, or online with a latency budget that bounds the model size (Choosing the Inference Mode).
- Treat the input distribution as a monitored assumption. A network has no coefficient to inspect, so the drift monitor on the inputs and the prediction distribution is the only early warning (Data Drift, Prediction Drift).
- Keep the rules. Deployed as a fallback and as a slice for evaluation, they are the baseline the network is measured against and the thing that serves when it cannot (Serving Fallbacks).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Recall of sent-back documents at the review-capacity threshold, on a time-based split, against the rule set and the linear text baseline. This number maps to what reaches customers.
- Validation loss against training loss per epoch — the gap is the overfitting signal, and it is the first number to look at with a network (Learning Curves).
- Forward-pass latency at the p99 for the longest documents on the serving hardware, measured before the architecture is fixed.
- Do not measure "the network reached a lower training loss" as improvement. Training loss on a network with enough parameters goes to zero on any labels, including random ones.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The inputs at serving time come from the distribution the network was trained on, including the tokeniser and preprocessing that produced the training tensors.
- The labels the network learned reflect the reviewers who will judge its output, not a pool that has since changed.
- The serving hardware can execute a forward pass for the largest input within the latency budget under peak load.
- The learned representation still separates the classes for new document templates and sources, which nothing in the artifact can tell you — only the drift and outcome monitors can.
- Offline: time-based split, per-source slices, the rule set and the linear baselines on the same split, and a learning curve over training-set size to see whether more data would still help (Evaluation Slices).
- Online: shadow the network for two weeks against the rules, comparing routing decisions and reviewer verdicts on both (Shadow Deployment).
- Over time: input drift per source, prediction-distribution drift, and the reviewer send-back rate on documents the model passed — the delayed label (Ground-Truth Delay).
What can go wrong
- The pretrained representation was trained on general text and the documents are a specialised register; the fine-tuned model is confident on general-looking pages and lost on the forms that matter.
- An upgrade of the tokeniser or the preprocessing library changes the inputs on the serving side only; the weights are unchanged and the predictions are wrong (Train / Serve Skew).
- Label noise from reviewer disagreement caps the achievable quality, and the team keeps enlarging the network to close a gap that is in the labels (Label Quality).
- The GPU the model was validated on is not the CPU it is served on, and numeric differences in the forward pass move a few predictions across the threshold.
- A learned representation removes the need to write features by hand and replaces it with the need for data, compute, a pretrained starting point and a tuning process that is harder to reason about.
- A network can fit any boundary the data supports, which is exactly why it fits the noise as well and why regularisation, early stopping and honest validation are not optional (Regularisation, Early Stopping).
- There is no coefficient to read. Explanations are post-hoc and approximate, and the monitoring burden rises to compensate (Explainability).
- "Neural networks are always better." On six tabular columns the linear model matched the network; on small tabular data a tree ensemble or a linear model usually wins and serves for a fraction of the cost. Networks win where the representation has to be learned and the data can pay for it.
- "The network learns the features, so we do not need to understand the data." It learns features from the training distribution and labels it was given, including their noise and their reviewer bias. Understanding the data is how you know what it learned.
- "Training loss is still falling, so keep training." Validation loss is the one that matters, and once it turns the network is memorising.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DATA-SPECIFICOn raw text, images, audio and long sequences a network with a pretrained representation is the default and usually wins by a wide margin; on small tabular data with engineered features it typically does not beat a tuned tree ensemble or a linear model, and costs more to train, serve and explain.
- SCALE-SPECIFICFrom scratch, a network needs data in proportion to its parameters; at two hundred thousand noisy labels the only viable text model is an adapted pretrained one, and at a few thousand labels even that may lose to a bag-of-words linear model.
- CONTESTEDA serious position holds that with modern pretrained encoders the tabular exception is shrinking — that an encoder over a textual rendering of the row, or a foundation model for tables, matches boosting on many benchmarks and will overtake it. The strongest form of that view is right that the gap is closing on benchmark suites; the counter is that on a specific business table with a few hundred thousand rows, a tuned boosted ensemble is still cheaper, faster, more explainable and rarely worse, and the benchmark improvements have not yet moved the default in production.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Programming Languages & Runtime Internals — a pinned tokeniser and preprocessing dependency in the artifact is a runtime-reproducibility requirement; the network is only as deterministic as the library stack that builds its input tensors.