Naive Bayes
Count the evidence per class and multiply, pretending every feature is independent. Wrong about the world, surprisingly right about text, and confidently miscalibrated.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The independence assumption is plainly false for words in a sentence — so why does the classifier rank documents well, and why must its probabilities never be shown to anyone?
A support team receives twenty thousand tickets a week and wants them routed to the right queue automatically. "We have three years of tickets with the queue they ended up in. We need something that runs on the ticket text the second it arrives, and we need to know how sure it is so unsure ones go to a human."
For each queue, count how often each word appears in that queue's tickets. For a new ticket, multiply the per-word probabilities for each queue, multiply by the queue's prior, and pick the largest. Train in seconds on millions of tickets, predict in microseconds, and the arithmetic yields a probability for free.
The "probability" is almost always within a hair of 0 or 1. A ticket with two hundred words multiplies two hundred likelihood ratios that all point the same way because the words are correlated, so the classifier is absurdly overconfident. The confidence gate that routes unsure tickets to a human never fires: nothing is ever unsure.
- The "probability" is almost always within a hair of 0 or 1. A ticket with two hundred words multiplies two hundred likelihood ratios that all point the same way because the words are correlated, so the classifier is absurdly overconfident. The confidence gate that routes unsure tickets to a human never fires: nothing is ever unsure.
- A ticket mentioning a new error code that never appeared in training gets a zero count in every queue, and the product of probabilities is zero for all twelve. Every class ties at zero and the argmax is whichever queue is listed first.
- Offline accuracy on a random split looked strong because the same product-name and error-code tokens appear on both sides. In production the vocabulary moves every release, and the words that decided last month's tickets are absent from this month's.
- The prior does the work for rare queues in the wrong direction: with a one-percent prior and correlated evidence, a rare queue needs overwhelming word evidence to win, and the model routes its tickets to the forty-percent queue by default.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict which of twelve queues a new ticket belongs to, from its subject and body. The label is the queue the ticket was finally resolved in, which may differ from where it was first routed.
- Two decisions hang on the output: the routing itself, and whether the confidence is high enough to skip the human triage step. The second decision needs a calibrated probability; the first only needs the right class on top.
- One example is one resolved ticket: the text, tokenised into words, and the resolving queue. Around three million tickets, heavily skewed — one queue takes forty percent of volume, two queues take under one percent each.
- Labels come from where the ticket was resolved, not where a human would have routed it; some tickets bounce between queues and the last one wins.
- Vocabulary is large and long-tailed, product names and error codes appear in a handful of tickets each, and new error codes appear weekly as software ships.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Bayes' rule: P(class | words) ∝ P(class) · P(words | class). The "naive" step is the factorisation P(words | class) = ∏ P(word_i | class), which assumes every word is independent of every other given the class. That is false — "password" and "reset" co-occur far more than independence predicts — but it makes the likelihood a product of counts that can be estimated from a table.
- Each P(word | class) is a smoothed count: (count of the word in the class + α) / (total words in the class + α · vocabulary size). Without α, an unseen word has probability zero and annihilates the product; with α, every word gets a small floor. α is a hyperparameter and it matters most for rare words and rare classes.
- Classification takes the argmax over classes of log P(class) + Σ log P(word_i | class). Only the ranking of classes matters for the top-1 decision, and the ranking is robust to correlated evidence: double-counting "password" and "reset" inflates the score for the right class and the wrong ones in similar proportion. The *magnitude* of the posterior, however, is inflated by every redundant word (Calibration).
- The model is linear in the log-count features, which is why it behaves like a logistic regression with weights fixed by counting instead of by optimisation. Logistic regression learns to discount correlated words; Naive Bayes cannot, so it trades calibration for a training step that is a single pass over the data.
Count, smooth, multiply
The entire model is a table of counts: for each queue, how many times each word appeared in its tickets, plus how many tickets it had. Prediction turns those counts into probabilities and multiplies them. The only subtlety is what to do about a word that never appeared in a class — and without smoothing, that one word sets the whole product to zero.
Working in log space is not a numerical convenience; it makes the model's structure visible. The score for a class is its log prior plus a sum of per-word log-likelihoods, which is a linear model whose weights were counted rather than learned. That is why it trains in one pass and why it cannot learn that two words say the same thing.
1import math2from collections import Counter, defaultdict3 4def fit(docs, labels, alpha=1.0):5 word_counts = defaultdict(Counter) # class -> word -> count6 class_counts = Counter(labels)7 for words, c in zip(docs, labels):8 word_counts[c].update(words)9 vocab = {w for cnt in word_counts.values() for w in cnt}10 return word_counts, class_counts, vocab, alpha11 12def log_scores(words, model):13 word_counts, class_counts, vocab, alpha = model14 n_docs, V = sum(class_counts.values()), len(vocab)15 out = {}16 for c, n_c in class_counts.items():17 total = sum(word_counts[c].values())18 s = math.log(n_c / n_docs) # log prior19 for w in words:20 # Laplace smoothing: an unseen word gets alpha / (total + alpha * V), never zero21 s += math.log((word_counts[c][w] + alpha) / (total + alpha * V))22 out[c] = s23 return out # argmax is the routing decision; the *values* are not calibratedThe comment on the last line is the lesson. Turning these scores into "probabilities" with a softmax gives numbers that are almost always within a rounding error of 0 or 1, because every correlated word is counted as independent evidence.
Why the wrong assumption works, and where it does not
Independence is violated everywhere in text, yet the top class is usually right. The reason is that violations push the scores of all classes in similar directions: "password" and "reset" together over-count the evidence for the account queue, but they also over-count whatever weak evidence they carry for every other queue. The argmax survives; the gap between the classes is exaggerated.
That exaggeration is precisely the calibration failure. A team that uses the argmax and ignores the magnitude gets a fast, honest router. A team that reads the magnitude as confidence gets a gate that never opens, and an offline accuracy number that says nothing about it.
Accuracy here is above ninety-nine percent and the queue is missing forty percent of its tickets. The prior is doing what the counts told it to; the fix is a per-queue threshold on the log-score margin, not a better overall number.
What has to stay true
A counted model is a snapshot of the vocabulary. It keeps working as long as the words that separate queues this week are the words that separated them in the training window, and as long as the serving tokeniser produces the same words the counts were built from. Both drift, and both drift silently — a new error code is simply an unseen token that the smoothing floor quietly absorbs.
The assumption to monitor is therefore not "the model is accurate" but "the vocabulary the model knows still covers the tickets". That is measurable without labels, which for a routing problem with delayed re-routing signals is a large advantage.
The tokens that carry the routing signal this week appeared, with the same meaning, in the training window and are tokenised identically at serving time.
holds when Retraining runs at least as often as the product ships new error codes and names; the serving tokeniser is the training tokeniser; per-queue volumes are close to the training priors.
breaks when A release introduces new codes that dominate a queue; a tokeniser change alters case or punctuation handling on one side only; a campaign shifts the queue mix so the priors are stale.
respond Retrain on the new window — cheap for this model — and re-fit the calibration layer; if the tokeniser moved, fix the skew before retraining or the new counts inherit it.
How to build it
Most important first.
- Use it as the routing baseline it is good at: top-1 class from log-scores, trained in seconds, easy to retrain nightly as vocabulary moves (Baselines Are Mandatory, The Rule Baseline).
- Never expose the raw posterior as a confidence. Calibrate on a held-out fold — isotonic or Platt — or replace the "confidence" with the log-score margin between the top two classes, which is what actually separates sure from unsure tickets.
- Set α by validation, not by default, and watch it as a knob on rare-queue recall: larger α shrinks rare-word evidence towards uniform, which hurts the classes whose signal lives in rare words.
- Split by time so validation measures the vocabulary shift the model will face, and report per-queue recall, because aggregate accuracy is the forty-percent queue (Time-Based Split, Accuracy Under Imbalance).
- When correlated-evidence overconfidence matters — a gate, a shown score, a downstream cost model — move to logistic regression on the same tokens, which is the same model with learned rather than counted weights (Logistic Regression).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-queue recall and the confusion between queues on a temporal validation split; the two under-one-percent queues are the ones the business escalates about, and they vanish inside a single accuracy number.
- Calibration curve of whatever score gates the human step. This is the number that decides whether the gate works; top-1 accuracy says nothing about it.
- Fraction of tickets containing at least one token unseen in training, per week. When it climbs, the smoothing floor is doing more of the work and the rankings degrade.
- Do not measure the posterior's mean confidence and call it "the model is 99% sure on average". That number is a property of the independence assumption, not of the tickets.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The word distribution per queue is stable enough between retrains that counts from last month still rank this month's tickets correctly.
- The tokeniser at serving time produces the same tokens the counts were computed over, including case, punctuation and number handling.
- The score used to gate the human step has been calibrated on a recent held-out fold, and the calibration is re-checked as vocabulary moves.
- The class priors in the training data match the queue mix in production — a marketing campaign that doubles one queue's volume changes the prior the model still assumes.
- Offline: temporal split, per-queue recall against a majority-class baseline, and a reliability diagram for the gating score before and after calibration.
- Online: sample tickets that skipped the human step and have a human re-label them weekly; the disagreement rate is the calibration check the diagram cannot give you.
- Over time: track unseen-token rate, per-queue volume against training priors, and the rate at which tickets are re-routed after initial assignment — the delayed ground truth for this problem (Ground-Truth Delay).
What can go wrong
- Calibration was fitted once; six months later the vocabulary has moved and the calibrated score drifts too, so the gate lets through tickets that should have gone to a human.
- A nightly retrain ingests tickets labelled by the model itself (routed and never corrected), and the classifier learns its own routing decisions as ground truth (Feedback Loops).
- Tokenisation differs between the training job and the serving path — one lowercases and strips punctuation, the other does not — so
Error-4012anderror 4012are different words with different counts (Train / Serve Skew). - α was tuned for the large queues; a product launch adds a rare queue whose discriminating words appear five times, and the smoothing floor flattens them into noise.
- Training is a counting pass and retraining is free, so it can follow vocabulary drift nightly; the cost is that it can never learn to discount redundant evidence, so its probabilities are not usable without a calibration layer.
- It needs almost no data per class to produce a sensible ranking, which is why it works on the rare queues at all; the same property makes it fragile to a handful of mislabelled tickets in those queues.
- Its explanations are exact — the log-likelihood contribution of each word — but they describe counts, not causes, and a word that appears in one queue by coincidence looks as decisive as one that defines it.
- "The independence assumption is false, so the model is wrong." It is wrong about the posterior's magnitude and often right about its ranking. The assumption breaks calibration, not classification.
- "The model says 0.999, so we can skip the human." That number is the product of correlated evidence and is not a probability of being right. Calibrate it or use the top-two margin.
- "It is a toy; a transformer would obviously be better." On short tickets with a fixed queue set, a counted model retrained nightly often matches a fine-tuned encoder on routing accuracy at a thousandth of the serving cost. Better at what, measured how, and at what latency?
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- MODEL-SPECIFICThe calibration failure is a property of multiplying correlated likelihoods and is unique to Naive Bayes; a logistic regression on the same token features learns to discount redundancy and its probabilities are usually reasonable without a calibration step.
- DATA-SPECIFICOn bag-of-words text with many weakly informative tokens the independence assumption is harmless for ranking; on dense tabular data with a few strongly correlated numeric features it actively misranks, and a tree or linear model should be the baseline instead.
- CONTESTEDOne position holds that Naive Bayes has no place in a modern text pipeline because a small pretrained encoder is nearly as cheap to serve and strictly more accurate on anything longer than a subject line. The strongest form of that view is right about accuracy on long, nuanced text; the counter is that routing tickets by queue is a short-text, high-volume, nightly-drifting task where a model that retrains in seconds and explains itself by word counts is operationally simpler, and the gap on top-1 accuracy is often within the label noise of "where the ticket was finally resolved".
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Programming Languages & Runtime Internals — the counting model is a sparse linear scorer, and its serving cost is a hash-map lookup per token; the difference between a microsecond and a millisecond here is the tokeniser implementation, not the model.