k-NN, Naive Bayes & SVM

Three mental models, their assumptions, strengths and limits — taught as ways of thinking about data rather than as API calls.

k-Nearest Neighbours

▶ lab

No training, all the cost at inference, and a distance that is only meaningful in a scaled space. The mental model behind every embedding retrieval system.

Q · A model with no training step at all beat the tuned one offline — what is it assuming about distance, and what will it cost to serve?

Naive Bayes

Count the evidence per class and multiply, pretending every feature is independent. Wrong about the world, surprisingly right about text, and confidently miscalibrated.

Q · The independence assumption is plainly false for words in a sentence — so why does the classifier rank documents well, and why must its probabilities never be shown to anyone?

Support Vector Machines

The widest street between the classes, defined by the few points closest to it. Kernels make a linear boundary curve; scaling is mandatory; the cost grows badly with the number of rows.

Q · The model is defined by a handful of boundary points and ignores everything else — when is that a strength, when does it make training unaffordable, and what does a kernel actually buy?

Which Model Should We Use?

▶ lab

The interview question whose red-flag answer is a product name. A strong answer asks about the data, the latency, the interpretability, the metric and the cost before naming anything.

Q · Someone asks "which model should we use?" — what has to be known before any family can be named, and what does each family cost once it is chosen?