Transformer Fundamentals
Tokens become embeddings, attention mixes information across positions, a feed-forward layer transforms each position on its own, and residual connections plus normalisation let dozens of those blocks stack. Knowing where the parameters and FLOPs live is what turns "use a transformer" into a cost you can budget.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does a transformer block actually compute, where do its parameters and FLOPs live, and why does the context length set the serving cost?
A product team wants to classify and summarise customer reviews with a pretrained transformer. Finance has asked for a serving cost estimate and the answer so far has been "it depends". The team needs to know what the model does with a 2,000-token review that it does not do with a 200-token one, and whether a bigger model is a bigger bill in proportion.
A transformer is a black box that takes text and returns text. Cost is per request; a bigger model is proportionally more expensive; measure it with a load test and move on.
Cost is not per request. Attention over n tokens builds an n×n matrix per head per layer, so the 2,000-token review costs a hundred times the attention compute of the 200-token one, and the load test — run on the median length — measured the wrong bill.
- Cost is not per request. Attention over n tokens builds an n×n matrix per head per layer, so the 2,000-token review costs a hundred times the attention compute of the 200-token one, and the load test — run on the median length — measured the wrong bill.
- The parameter count says how much memory the weights need, not how the FLOPs scale with length. At short context nearly all the FLOPs are in the per-position feed-forward and projection layers and scale linearly with tokens; at long context attention dominates and scales quadratically. Two models with the same parameter count can have very different long-input costs.
- The summary is generated one token at a time, and each new token attends over everything before it. The cost of a generated token grows with the context, so the "summarise a long review" path is expensive twice — once to read, and again per token written (Inference Cost).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding system predicts a sentiment class and produces a short summary per review; the label for sentiment is a star rating, and the summary has no label beyond human spot checks.
- The question this lesson answers is upstream of either: what the model computes per token and per position, so that quality, latency and cost can be reasoned about before a benchmark is run.
- One input is a review tokenised into subword pieces — roughly four characters per token in English — so a 2,000-token input is a long review, not an unusual one.
- The pretrained model comes with its tokeniser and vocabulary; those are part of the artifact, and a review in a language the tokeniser fragments badly costs several times the tokens (What a Model Artifact Contains).
- Serving traffic is a distribution of lengths, not a mean; the tail of long reviews decides the tail of latency (Tail Latency: Why p50 Being Fine Does Not Help).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The pipeline is tokens → embeddings → attention → feed-forward → representations. A lookup table maps each token id to a d-dimensional vector; positional information is added (Positional Information); then a stack of identical blocks transforms the sequence of vectors, and a task head reads the final representations.
- Each block has two sub-layers. Self-attention lets every position pull information from every other position (Self-Attention); it is the only place positions interact. The feed-forward sub-layer applies the same two-layer MLP to each position independently, usually expanding to 4d and back. Each sub-layer is wrapped as
x = x + sublayer(norm(x)): the residual connection gives gradients a direct path through dozens of blocks, and the normalisation keeps activations in range so that path stays trainable (Normalisation Layers). - Parameters per block are dominated by four d×d attention projections (4d²) and the two feed-forward matrices (8d²), so about 12d² per block, times the number of blocks, plus the embedding table. FLOPs per token are roughly twice the parameter count for the per-position work, plus attention, which costs on the order of n·d per token per layer and so n²·d per layer for the whole sequence. That last term is the one that grows with context.
Tokens → embeddings → attention → feed-forward → representations
The input is a sequence of token ids. An embedding table turns each into a d-dimensional vector; positional information is added so the model can tell order. The result is an n×d matrix, and everything after this is transformations of that matrix.
Each block first lets positions exchange information through attention, then transforms each position independently through a feed-forward layer. Stack twelve, thirty-two or ninety-six of those and the final n×d matrix is the representation: a classifier reads one row (or a pooled row) of it; a generator reads the last row and predicts the next token.
1def block(x, params):2 # x: n x d — one row per token position3 # attention is the ONLY place rows talk to each other4 x = x + attention(layer_norm(x), params.attn) # 4 d*d projections5 # the feed-forward runs on each row independently6 x = x + ffn(layer_norm(x), params.ffn) # d -> 4d -> d7 return x # still n x d8 9# residual: gradient reaches early blocks through the "+ x" path unchanged10# norm: keeps each row's scale in range so that path stays trainableTwo things to notice. Nothing in ffn depends on n, so it is linear in tokens; attention compares every row to every row, so it is quadratic. And the residual add is what lets ninety-six of these stack — without it, the gradient to block one is a product of ninety-six Jacobians, the same disease recurrence had (Vanishing and Exploding Gradients).
Where the parameters and the FLOPs live
Count one block. Attention has four d×d projections — query, key, value, output — so 4d². The feed-forward expands to 4d and contracts, so 2 × 4d² = 8d². About 12d² per block; with d = 4,096 that is roughly 200 million per block, and a 32-block model lands around 6–7 billion before the embedding table. This is the number people quote, and it is a memory number: at two bytes per weight, 13 GB of VRAM before a single token is processed.
FLOPs are a different count. Each token passes through every matrix, so the per-position work is about 2 × parameters per token — linear in length. Attention adds n·d per token per layer for the scores and again for the weighted sum, so n²·d per layer for the sequence. At 200 tokens the attention term is a rounding error; at 8,000 it is the bill.
Production inputs and outputs stay within the token-length range the latency budget, batch size and VRAM allocation were computed for.
holds when Inputs are bounded by product design — a review field with a character limit — and the length distribution is monitored, so the quadratic tail is known and truncated deliberately.
breaks when A new input path accepts pasted documents; a tokeniser fragments a new language; a feature starts concatenating history into the prompt. The p99 of tokens moves and the cost curve is on its steep part.
respond Fix the input path or the truncation policy first. Changing the model or the hardware to absorb an unbounded length is a decision about product scope disguised as an infrastructure one.
per block, d = model width, n = tokens in context parameters (memory) FLOPs per forward pass attention 4 d^2 per-position work ~ 2 * params * n (linear in n) feed-forward 8 d^2 attention scores ~ 2 * n^2 * d (quadratic in n) ---------- ----------------- ~12 d^2 per block crossover when n ~ 6 d ... beyond that, attention dominates generation: each new token attends over all previous ones, so writing m tokens after reading n costs ~ sum over (n + k) for k in 1..m
Why context length is a serving decision
The load test told finance a cost per request because it ran at the median review length. The real bill is a function of the length distribution: linear in tokens for most of it, quadratic in the tail, and multiplied again for every generated token of summary. Two thousand tokens is not ten times two hundred; it is a hundred times, in the term that matters at that length.
That makes context length a product and serving decision rather than a model setting. What part of the review must the classifier see? How long may a summary be? Each answer is a line on the bill, and none of them is visible in the parameter count that gets quoted.
Load test at the median review length gave a comfortable p95 latency and a per-request cost finance accepted.
The p99 latency was several times the tested figure on the first day and the monthly bill was well above the estimate, with the excess concentrated in a small fraction of long reviews.
- 1Attention cost is quadratic in tokens and the length distribution has a long tail; the median-length test measured the linear regime only.
- 2Summaries are generated token by token and each step attends over the whole input, so the long reviews cost extra once per output token as well.
- 3A minority of reviews in a language the tokeniser fragments badly cost several times the tokens of the same text in English.
How to build it
Most important first.
- Budget by tokens, not requests, and by the length distribution rather than the mean. Log input and output token counts per request from day one; they are the cost model (Inference Cost).
- Choose the context length you actually need and truncate deliberately — first and last paragraphs of a review carry most of the sentiment — rather than paying quadratic cost for text the model does not need.
- Separate the two workloads: classification is one forward pass over the input; summarisation is that plus a generated token per step. Serve them with different latency budgets and batching policies (Inference Batching, Latency Breakdown).
- Treat the tokeniser as part of the model. A tokeniser change is a model change; a language the tokeniser handles badly is a cost and a quality problem before it is a fairness one (Preprocessing Lives in the Artifact).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Cost and latency as a function of input tokens, plotted, not averaged — the curve's shape tells you when attention starts to dominate for this model on this hardware.
- Tokens per second at the batch sizes production actually sees. Weight memory bandwidth, not FLOPs, is usually the ceiling at small batch (Memory Bandwidth & VRAM).
- Do not measure "cost per request" from a load test at the median length. The tail of the length distribution is the bill.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The production distribution of input and output token lengths stays within the range the cost estimate and the VRAM allocation were sized for; a shift in the length tail is a cost and availability change before it is a quality change.
- The tokeniser deployed at serving time is byte-identical to the one the model was trained with; a mismatch is skew that no metric on the model side can see (Train / Serve Skew).
- The residual-and-norm structure that made the model trainable is part of the artifact and is not "simplified" for serving; removing a normalisation layer for speed changes the function.
- Offline: run a length sweep — 100, 500, 2,000, 8,000 tokens — and record latency, peak memory and cost per input; fit the curve and check where the quadratic term takes over.
- Online: alert on the p99 of input tokens and on the fraction of truncated requests; both are leading indicators of cost and of a quality drop the accuracy metric will only show later (Percentiles: Which One, and How Many Users Is That?).
- Over time: re-run the length sweep whenever the serving stack changes — a new attention kernel, a quantised weight format — because the crossover point moves with the hardware (Quantization).
What can go wrong
- A customer pastes a 20,000-token document. Either the request is rejected, truncated silently, or the attention matrix does not fit in VRAM and the server falls over for everyone in the batch (Memory Bandwidth & VRAM).
- A reviewer writes in a language the tokeniser fragments into single characters; the same review is now five times the tokens, the context is exceeded, and quality drops for that group of users specifically (Fairness).
- The team switches to a model with the same parameter count but a longer default context, assuming equal cost; long inputs are now accepted rather than truncated and the bill triples.
- The attention that makes long-range context available is the term that makes long context expensive; every architecture that reduces it — sparse, windowed, linear attention — gives up some of the all-to-all mixing.
- A larger d improves representation quality and scales parameters and per-token FLOPs quadratically in d; the bill for "a bigger model" is not linear in the number that gets quoted.
- Truncation controls cost and changes the function: the model classifies the part of the review it was shown, and a complaint in paragraph six is invisible by design.
- "Twice the parameters, twice the cost." Per-token cost is roughly linear in parameters, but the attention term is independent of parameter count and quadratic in length. For long inputs the smaller model with the longer context can be the more expensive one.
- "The context window is a limit, not a cost." Every token in the window is attended to by every subsequent token, at every layer, on every generated step. A window you fill is a bill you pay, whether or not the content mattered.
- "A transformer is a black box." The block structure is short enough to write on a page, and where the FLOPs go follows from it. Budgeting without that is guessing.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SIMPLIFIEDParameter and FLOP counts are order-of-magnitude for the shape of the argument: 12d² per block ignores biases, layer norms and the embedding table, and the attention cost ignores the KV cache and fused kernels that change constants without changing the n² term.
- MODEL-SPECIFICThe block structure described is the standard dense pre-norm transformer. Mixture-of-experts models have far more parameters than active FLOPs per token, and models with sliding-window or linear attention change the length scaling; the token-based budgeting still applies, the curve does not.
- SCALE-SPECIFICAt small batch sizes on a GPU the ceiling is weight memory bandwidth rather than FLOPs, so the arithmetic here over-predicts latency; at high batch the FLOP count is the right model. Which regime you are in depends on traffic, not on the architecture.
Where the depth lives
This domain teaches the model and hands the rest off by name.