The Forward Pass
Input → layers → prediction, as a sequence of matrix multiplications with a batch dimension. This is where the FLOPs go, and where inference cost is decided.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A prediction is a chain of matrix multiplications — where does the compute go, what does the batch dimension change, and why does the same network cost so differently on different hardware?
A search team's reranking model scores candidate documents per query. "The model is small — three layers. Why does scoring two hundred candidates per query take longer than the retrieval that found them, and why is it ten times faster on the training GPU than on the serving CPU?"
A three-layer network is tiny; inference is negligible. Score each candidate in a loop, return the top ten. If it is slow, add a GPU.
Scoring candidates one at a time is two hundred separate calls into the model, each doing a matrix-vector product with the same weight matrices, each paying the fixed overhead of the call and re-reading the weights. Batching the two hundred into one matrix-matrix product cut the time by an order of magnitude without changing a weight (Inference Batching).
- Scoring candidates one at a time is two hundred separate calls into the model, each doing a matrix-vector product with the same weight matrices, each paying the fixed overhead of the call and re-reading the weights. Batching the two hundred into one matrix-matrix product cut the time by an order of magnitude without changing a weight (Inference Batching).
- The GPU is fast at large matrix products and slow to start: for one query's two hundred rows the launch overhead and the transfer of the inputs across the bus dominate, and the CPU wins. "Add a GPU" made p50 latency worse (GPU Fundamentals).
- The team profiled the model and found the forward pass was a fraction of the request time; the feature service building two hundred vectors was the rest. The model was never the bottleneck (Latency Breakdown).
- A later change widened the first layer from 256 to 1024 units to "add capacity"; the first-layer product went from a few hundred thousand multiply-adds per candidate to over a million, and the latency budget was gone.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Score each (query, candidate) pair for relevance so the top ten can be shown. The label is a click or a judged relevance grade; the metric is a ranking one (Ranking).
- The serving constraint is a fixed latency budget per query across all candidates; the model's per-pair cost times two hundred must fit inside it.
- One example is one (query, candidate) pair represented as a few hundred features — text-match scores, popularity, freshness, a learned embedding similarity — with the label. Hundreds of millions of pairs.
- At serving time, one query arrives with two hundred candidates; the two hundred feature vectors are built by a feature service and handed to the model together.
- Training ran on a GPU with batches of thousands of pairs; serving runs on CPU pods with one query's candidates at a time.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- For one input, each layer computes z = Wx + b: a matrix of shape (out, in) times a vector of length in. That is out × in multiply-adds, then out activations. The whole forward pass is the sum over layers, and a three-layer network of widths 300 → 256 → 128 → 1 costs roughly 300·256 + 256·128 + 128 multiply-adds per input.
- For a batch of B inputs, stack them as rows of a matrix X of shape (B, in) and compute Z = XWᵀ + b, shape (B, out). The FLOPs are B times larger but the weights are read once, the operation maps onto hardware built for matrix multiplication, and the per-input cost falls until the hardware is saturated. The batch dimension is a free axis on which every layer is independent.
- On a GPU, thousands of threads compute the tiles of the matrix product in parallel and the operation is limited by how fast the weights and activations can be fed from memory; small batches leave most of the chip idle and pay a fixed launch cost. On a CPU, a vectorised BLAS routine uses SIMD lanes and the cache; for small matrices it wins on latency because there is nothing to launch and nothing to transfer (Memory Bandwidth & VRAM).
- Where the FLOPs go is therefore readable from the shapes: the widest layer dominates, the batch multiplies everything, and the ratio of FLOPs to bytes moved decides whether the hardware is compute-bound or memory-bound. For a small dense reranker on a single query the answer is "memory-bound and overhead-bound", and a GPU is the wrong tool.
The pass, and the batch axis
One input is a vector; a batch is a matrix whose rows are inputs. Every layer is independent across rows, so the batched forward pass is the same code with one more dimension, and the weight matrix is read once for the whole batch instead of once per input. That is the entire reason batching is faster: the arithmetic is identical, the memory traffic is not.
The shapes tell you the cost. Write them down for every layer, multiply, and you have the FLOPs per input; multiply by the batch and you have the FLOPs per request. There is no other place the compute can hide in a dense network.
1import numpy as np2 3def forward(X, layers):4 # X: (B, d_in). Each layer is (W: (d_out, d_in), b: (d_out,), f)5 H = X6 for W, b, f in layers:7 H = f(H @ W.T + b) # (B, d_in) @ (d_in, d_out) -> (B, d_out)8 return H9 10def multiply_adds_per_input(layers):11 return sum(W.shape[0] * W.shape[1] for W, _, _ in layers)12 13# widths 300 -> 256 -> 128 -> 1:14# 300*256 + 256*128 + 128*1 = 109,696 multiply-adds per candidate15# x 200 candidates per query -> ~22M per query, weights read once if batched16# widen layer one to 1024: 300*1024 + 1024*128 + 128 = 438,400 -> 4x per candidateThe comment is the cost model. The first layer dominates because it is the widest product; "adding capacity" there quadruples the FLOPs, and at serving batch size on a CPU that is the latency.
Where the time goes on each kind of hardware
A GPU executes a matrix product as thousands of parallel tiles and is limited by feeding them: it wants large matrices, and it charges a fixed cost to launch a kernel and to move the inputs across the bus. A CPU executes the same product with a handful of vectorised cores and no launch cost, so for a small matrix it is simply done sooner. The crossover is a batch size, and it is measured, not assumed.
The reranker's regime is one query, two hundred rows, a hundred thousand multiply-adds each. That is a few milliseconds of vectorised CPU work and a GPU kernel launch that costs about as much before doing anything. The training run, at batches of thousands, was in the other regime, which is why the GPU number was irrelevant.
| Option | Quality | Latency | Cost | Operational | Note |
|---|---|---|---|---|---|
| CPU, unbatched loop | Same output; two hundred calls re-reading the weights. | ||||
| CPU, batched per query | One matrix product per layer; the right regime for this size. | ||||
| GPU, batched per query | Launch and transfer overhead exceed the compute at this batch size. | ||||
| GPU, batched across queries | Throughput improves; queueing delay moves the p99; a scheduler to operate. |
caveat Quality is identical across rows because the weights are the same — the matrix only says where the time goes for this model at this batch size. Grow the model to a transformer over token sequences or the batch to thousands and the GPU rows move to the top; the crossover is a measurement on the actual hardware.
The cost is an assumption too
The latency budget was met for two hundred candidates through a 300 → 256 → 128 → 1 network on a particular CPU with a particular BLAS. Every one of those is an assumption. Retrieval may return more candidates, a feature may be added to the input width, a "small" capacity increase may quadruple the first product, and a library upgrade may change the backend and its numerics.
So the forward pass has a monitor like any feature: its latency at the p99, the batch size it receives, and a numerical equivalence check against the training environment on a fixed input set, so a backend change that reorders the top of the ranking is caught before a ranking metric drops.
At the serving batch size, on the serving hardware and backend, the forward pass completes within the budget and produces outputs equivalent to the training environment's to a tolerance that preserves the top-k order.
holds when Candidate count and layer shapes are the validated ones; the backend is pinned; the promotion benchmark runs on the serving hardware.
breaks when Retrieval is loosened, a layer is widened, a serving library changes the matrix-multiply implementation, or traffic grows so cross-request batching is introduced with its queueing delay.
respond Rebenchmark on the serving hardware before changing weights; if a numerics shift reorders the top-k, pin the backend or re-validate the ranking metric under the new one.
How to build it
Most important first.
- Score all candidates for a query in one batched forward pass; the batch dimension is what the matrix product is for.
- Compute the FLOPs per input from the layer shapes before choosing hardware, and measure the actual latency at the serving batch size on the candidate hardware — not on the training GPU (CPU or GPU for Inference).
- Profile the whole request: feature construction, transfer, forward pass, post-processing. Optimise the largest term (Latency Breakdown).
- Treat layer widths as a serving-cost decision with a budget; a width change is a latency change and needs the same review as a dependency change (Inference Cost).
- Where the model must run per request on a CPU, consider quantisation and a smaller width before considering a GPU (Quantization).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Forward-pass latency at the p99 for the serving batch size on the serving hardware; this is the number the latency budget is checked against.
- Multiply-adds per input from the layer shapes, and bytes of weights read per forward pass; together they say which hardware regime the model is in.
- The fraction of end-to-end request time spent in the model, so an optimisation is aimed at the right term.
- Do not measure throughput at training batch size and quote it as serving speed; the per-input cost at batch two hundred and batch four thousand are different numbers.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The number of inputs per request stays near the batch size the latency was measured at.
- The layer shapes in the deployed artifact are the ones the latency budget was validated against — a "small" width change is a cost change.
- The serving hardware and its matrix-multiply backend produce numerically equivalent outputs to the training environment, to a tolerance that does not reorder the top of the ranking.
- The forward pass remains the fraction of request time it was profiled at; a change in the feature service can move the bottleneck without touching the model.
- Offline: a benchmark of the forward pass at the serving batch size on the serving hardware, run in the promotion pipeline with a latency threshold (Promotion Is a Checklist, Not a Score).
- Online: a per-request breakdown of time spent in feature construction versus the model, exported as metrics.
- Over time: candidate count per query and model latency at the p99, alerting when either trends towards the budget; a numerical equivalence test between the training and serving forward pass on a fixed input set.
What can go wrong
- A serving library upgrade changes the matrix-multiply backend and the numerics shift at the fourth decimal; a handful of candidates swap order at the top of the ranking, and the change is invisible until a ranking metric is compared.
- Batching across queries to fill a GPU adds queueing delay, and the p99 rises even as throughput improves (Throughput vs Latency).
- The candidate count per query grows from two hundred to two thousand as retrieval is loosened; the forward pass that fit the budget no longer does.
- Padding variable-length inputs to a fixed shape for batching multiplies the FLOPs by the padding ratio, and a few long inputs make every batch expensive.
- Batching cuts per-input cost and couples the request to the number of inputs; a variable candidate count becomes a variable latency.
- A GPU makes large batches cheap and small batches slow; choosing it is a bet that the serving batch size will be large enough to amortise the overhead.
- Reducing width or quantising cuts the FLOPs and can move the ranking; the saving has to be validated on the ranking metric, not assumed.
- "GPU automatically makes inference faster." For one query's two hundred rows through a small network, the launch and transfer overhead exceed the compute, and the CPU is faster. A GPU is faster where the matrices are large enough to fill it.
- "The model is small, so inference cost is negligible." Multiply the per-input cost by the inputs per request and the requests per second; then compare with the feature service. Small networks scored two hundred times per query are not small.
- "We doubled the width and latency barely moved offline." At training batch size on a GPU, it would not. At serving batch size on a CPU, the first layer's product is the whole cost.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a forward pass is a chain of matrix products with a batch axis holds for dense, convolutional and attention layers alike; convolutions are matrix products over unrolled patches, and attention adds products whose size grows with sequence length.
- SCALE-SPECIFICFor a small dense model on a single request the CPU is the right hardware and batching within the request is the optimisation; for a large transformer at high throughput the GPU is mandatory and cross-request batching with its queueing delay becomes the design problem.
- SIMPLIFIEDThe multiply-add counts are for the shape of the argument and ignore bias adds, activations, memory traffic and the fixed overheads that dominate at small batch; a real cost model needs a measurement on the serving hardware.
Where the depth lives
This domain teaches the model and hands the rest off by name.