Anomaly Detection
The model ranks how unusual each point is. Unusual is not the same as bad, positives are rare, and someone has to read the top of the list — so precision there is the whole product.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Positives are rare and mostly unlabelled, and the model flags what is unusual. Anomalous relative to what, and who checks whether the top of the list is worth reading?
A payments risk lead says: "New fraud patterns show up before we have any labels for them. We want a system that flags unusual transactions so analysts can look before the chargebacks arrive."
Fit a model of normal behaviour — an isolation forest, a density estimate, an autoencoder — on all transactions, score each new one by how far it is from normal, and send the top of the list to analysts. No labels needed, and new fraud is unusual by definition.
The most unusual transactions are the first purchase in a new country, a large legitimate refund, a merchant's end-of-month batch. Analysts open the queue and find the strange and the fine; precision at the top is low and they stop reading after a week.
- The most unusual transactions are the first purchase in a new country, a large legitimate refund, a merchant's end-of-month batch. Analysts open the queue and find the strange and the fine; precision at the top is low and they stop reading after a week.
- Fraud that imitates normal — small amounts at common merchants — is by construction not unusual and never reaches the top. The system is best at flagging what fraudsters have learned not to do.
- The model of normal was fitted on all transactions, including the fraud. Established fraud patterns are part of "normal" and score low (Data Poisoning is the adversarial version of this).
- There was no offline metric to look fine, because there were no labels; the system was judged on the number of alerts it produced, which is not a quality metric.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- There is no observed label for most rows. The model produces an anomaly score — distance from a learned notion of normal — and the implicit target is "an analyst would want to see this". That target is defined by the analyst, not the data.
- The decision is which transactions enter the review queue, which has a fixed capacity. The output is therefore a ranked list cut at queue size, and the metric is precision at the top of that list (Designing a Fraud Detection System).
- One example is one transaction with its cardholder's recent aggregates and merchant context. Almost all are legitimate; a small, unknown fraction are fraud; a larger fraction are unusual and fine.
- Some labels exist from chargebacks, which arrive weeks later and cover only fraud that the cardholder noticed and disputed (Ground-Truth Delay).
- Analysts' past review decisions exist, but they were made on transactions the previous rules selected, so they say nothing about what the rules missed (Selection Bias).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An unsupervised anomaly detector models the data distribution and scores each point by low density, isolation depth or reconstruction error. It answers "how unusual under this model of normal", which depends on the features, the model family and what was in the fitting set. Nothing in the objective distinguishes unusual-and-fraud from unusual-and-fine.
- A semi-supervised detector fits normal on rows believed clean, or uses the few labelled positives to shape the score. It still ranks by unusualness, but relative to a cleaner notion of normal, and it can be evaluated on the labelled positives it did not see.
- The queue has a capacity, so what matters is the ordering at the top: how many of the first N rows an analyst confirms. That is precision at N, and it depends on the base rate and on how well "unusual" correlates with "fraud" in this feature space — a correlation that fraudsters have an incentive to break (Adversarial Inputs).
Anomalous to whom
A density model, an isolation forest and an autoencoder each define unusual differently, and all three define it relative to whatever they were fitted on. Fit on all transactions and established fraud is normal. Fit on one region's traffic and every traveller is an anomaly. The definition of normal is a modelling choice with the weight of a label.
The risk lead does not want unusual; they want fraud the rules miss. Unusualness is a proxy, and the queue is where the proxy is checked. Every analyst decision is a label that says how good the proxy was this week.
1def build_queue(txns, score, capacity, review_rate=0.02, exclude_known_fraud=True):2 # score: unusualness under a model of normal fitted on rows NOT known to be fraud3 s = score(txns)4 order = np.argsort(-s) # most unusual first5 flagged = order[:capacity] # the queue analysts will read6 below = order[capacity:]7 rng = np.random.default_rng(week_seed())8 sampled = rng.choice(below, int(len(below) * review_rate), replace=False)9 # both sets are reviewed; both produce labels; only the second measures recall10 return {"queue": flagged, "recall_sample": sampled, "cut_score": s[order[capacity - 1]]}11 12# weekly: precision at capacity from analyst decisions on "queue";13# fraud rate in "recall_sample" is what the detector missedThe recall_sample is the part teams cut first, because it spends analyst time on transactions the model thinks are fine. Without it the system reports precision and has no idea what it misses.
Precision at the top of the list
The queue has a capacity. Analysts read from the top and stop when they run out of time. The entire value of the detector is therefore in how many of the first N rows are worth their time, and that number is measured by the analysts themselves, every day, whether or not anyone records it.
Recall is invisible unless it is bought: a random sample of unflagged transactions, reviewed at the same standard, gives the fraud rate outside the queue and therefore the detector's blind spot.
The fn count is an estimate scaled up from the random below-threshold sample; without that sample it would be unknown. Accuracy on this matrix is dominated by the true negatives and would be nearly the same if the queue were random.
Unusual correlates with bad — until it does not
The detector depends on one assumption: in this feature space, on this population, unusualness is correlated with fraud strongly enough that the top of the ranked list is worth reading. That assumption is weakened by legitimate novelty — a product launch, a holiday — and attacked by fraudsters who learn what is flagged.
It is checked by the analysts and by the random sample, and it is not checked by the score distribution, which drifts for many harmless reasons.
Ranking by unusualness under the current model of normal still places enough fraud at the top of the queue that analysts confirm a worthwhile share of what they read, and the fraud rate below the cut stays low.
holds when Weekly precision at capacity from analyst decisions is above the agreed floor; the random below-threshold sample shows a low fraud rate; the normal set has been refreshed with recent labels.
breaks when A legitimate segment becomes unusual (a launch, a season); fraudsters adapt to imitate normal; the normal set silently includes an established fraud pattern; analysts stop recording decisions.
respond For legitimate novelty, add the segment to normal and re-fit; for adaptation, move the detector's features and combine with the supervised score; for a silent inclusion, rebuild the normal set from labels. Retraining on the same normal set with the same features changes nothing.
How to build it
Most important first.
- Define "anomalous to whom" as a decision: the queue exists to find fraud the rules miss, so the score should rank by expected fraud, and unusualness is one input to that, not the output (Decision Before Model).
- Fit normal on a set that is as clean as available evidence allows — exclude confirmed fraud and chargeback-flagged rows — and refit as labels arrive, so known patterns are not baked into normal.
- Close the label loop: every analyst decision on the queue is a label, and a random slice of below-threshold transactions is reviewed too, so the training set includes what the detector did not flag (Exploration vs Exploitation).
- Combine with supervised scoring as labels accumulate — the anomaly score as a feature in a classifier trained on analyst and chargeback labels — and measure the combination on precision at queue size (Semi-Supervised Learning).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Precision at queue size, from analyst decisions, weekly. That is the number the queue lives or dies by, and analysts feel it before any dashboard does.
- Recall of chargeback-confirmed fraud among flagged transactions, once chargebacks arrive, and the fraud rate in the random below-threshold sample, which says what the detector misses.
- Alert volume and the anomaly score distribution are not quality metrics. A detector that flags more is not finding more fraud; a score that drifts is not necessarily wrong.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Unusualness under the fitted model of normal still correlates with fraud in the served population, as precision at queue size from analyst decisions confirms weekly.
- The set used to fit normal excludes known fraud and is refreshed as labels arrive, so an established fraud pattern does not become part of normal.
- The random below-threshold review continues, so the fraud rate outside the flagged set is measured and the detector's blind spot is visible.
- Offline: on the labelled positives held out from fitting, the rank each one receives under the detector; precision at queue size on a replayed week with analyst labels.
- Online: analyst confirmation rate at the top of the queue per day; fraud found in the random below-threshold sample; time-to-first-flag for each newly confirmed pattern.
- Over time: chargeback recall among flagged transactions as chargebacks mature; score distribution drift as a signal to inspect, not as a failure (Drift Is Not Failure).
What can go wrong
- The random below-threshold review is dropped to give analysts more queue capacity, and from then on recall is unmeasurable and the detector's blind spot is unknown.
- A new legitimate product launch makes a whole segment unusual for a month; the queue fills with it, precision collapses, and analysts learn to ignore the queue.
- Fraudsters probe the queue by making small unusual transactions, learn what is flagged, and adapt; the detector's notion of unusual is now a map of what not to do.
- A random below-threshold review spends analyst time on transactions the detector thinks are fine; it is the only measurement of recall the system will ever have.
- Excluding known fraud from the normal set makes the detector depend on labels, which was the thing it was supposed to work without.
- Optimising precision at queue size makes the detector conservative and slow to surface a genuinely new pattern, which is the case it was built for.
- "Anomaly detection needs no labels." It needs labels to know whether the anomalies are the ones that matter, and someone must produce those labels by reading the queue.
- "New fraud is unusual by definition, so the detector will catch it." Fraud that imitates normal is the profitable kind, and it is not unusual under any model fitted on normal.
- "Alert volume went up, the detector is finding more." Volume is a property of the threshold. Precision at the top is the only number that says the alerts are worth reading.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat an anomaly score measures unusualness under a chosen model of normal, not badness, and that the product metric is precision at the top of a capacity-limited list, holds for fraud, intrusion, defect and fault detection alike.
- DOMAIN-SPECIFICIn payments fraud the positives are adversarial and adapt to the detector, so unusualness decays as a signal; in equipment fault detection the positives are physical and do not adapt, and a model of normal fitted on healthy periods stays useful much longer (Time-Series Anomaly Detection).
- CONTESTEDA serious position holds that unsupervised anomaly detection should be skipped in fraud: the labels from chargebacks and analysts are enough for a supervised model, and the anomaly queue is a low-precision distraction that trains analysts to ignore alerts. The counter is that supervised models find only what has already been labelled, and the anomaly queue — with a random review slice and a closed label loop — is the mechanism by which new patterns get their first labels.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a random review slice is a sampling-based measurement of a quantity the system cannot otherwise observe, and keeping that measurement alive is an operational discipline this lesson assumes.