Architectures

A convolution slides one small set of weights over the whole input. That weight sharing is a belief about the data — the same pattern matters wherever it appears — and it is the reason a CNN needs far fewer examples than a fully-connected net on pixels.

A recurrent network carries a hidden state step by step through a sequence; that is elegant and it is why long dependencies were hard. Transformers replaced the recurrence with attention so every position can be computed in parallel — and simpler models still win many forecasting problems.

Tokens become embeddings, attention mixes information across positions, a feed-forward layer transforms each position on its own, and residual connections plus normalisation let dozens of those blocks stack. Knowing where the parameters and FLOPs live is what turns "use a transformer" into a cost you can budget.

Every token asks a question (query), every token advertises what it holds (key), and each token's new representation is a softmax-weighted mix of what the relevant tokens carry (value). The formula fits on one line; the weights it produces are a computation, not an explanation.

Attention is a weighted sum over a set: shuffle the tokens and it computes the same thing. Order has to be injected explicitly — learned, sinusoidal, relative or rotary — and the scheme you pick decides whether the model can say anything sensible past the lengths it was trained on.