Designing a Fraud Detection System
Rare positives, a strict online latency budget, asymmetric costs, an adversary who adapts to the model, and labels that arrive ninety days late. Every constraint in the domain at once.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How is a fraud detection system designed when the positives are rare, the decision must be made in milliseconds, the labels arrive months later, and the people generating the positives are trying not to be caught?
Chargebacks are eating our margin and the card networks are threatening penalties. We have a rules engine that flags obvious things and a review team that can look at a few hundred transactions a day. We want a model that catches more fraud without blocking good customers, and it has to answer before the payment completes.
Train a classifier on labelled history, report accuracy and ROC AUC, deploy at a threshold of 0.5 behind the payment endpoint, replace the rules engine. Accuracy is extremely high because almost every transaction is legitimate.
At a threshold chosen on the validation score distribution the review queue is either empty or ten times over capacity, because the number that mattered — flags per day — was never in the evaluation.
- At a threshold chosen on the validation score distribution the review queue is either empty or ten times over capacity, because the number that mattered — flags per day — was never in the evaluation.
- ROC AUC looked excellent and was nearly uninformative: with positives that rare, the false-positive rate barely moves across thresholds that change the queue size tenfold (PR AUC).
- Six weeks after launch fraud rings have found the transaction shapes the model does not flag and moved there; the model's recall on new fraud falls while its training-set metrics are unchanged, and the labels that would show it will not exist for two more months (Concept Drift).
- Serving features are computed from a stream with a different window than the warehouse batch job used for training; velocity features skew and the model is confidently wrong near window boundaries (Train / Serve Skew).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a transaction will be charged back as fraudulent within ninety days. The label is the chargeback event, which arrives up to three months later and is itself disputed and sometimes wrong (Label Quality, Ground-Truth Delay).
- The decision is three-way — approve, decline, or send to manual review — and the review queue has a fixed daily capacity, so the operating point is the score at which the queue is full, not a probability that feels right (Threshold Selection).
- One example is one transaction with the cardholder's recent behaviour joined as of the transaction timestamp: velocity over the last minutes, hours and days; merchant category history; distance and device signals. Positives are a small fraction of a percent of rows (Class Imbalance).
- Labels come from chargebacks, which means the training set of any given day is only labelled for transactions older than ninety days; the freshest quarter of data has features and no answers.
- The features that matter most are aggregates over the last few minutes, which no batch pipeline can produce; they are computed from an event stream at request time (Streaming Inference, Feature Freshness).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Rare positives change which metrics carry information. Accuracy is dominated by negatives and says nothing (Accuracy Under Imbalance). ROC AUC is computed against the false-positive *rate*, and with a tiny positive class the rate stays near zero while the absolute number of false positives — the queue — grows enormously. PR AUC and precision at the queue capacity are the numbers that track the decision.
- Adversarial behaviour means the data-generating process reacts to the model. A classifier trained on last year's fraud has learned last year's fraud; the fraud that reaches production is, by selection, the fraud the current model does not catch. This is concept drift by construction, and it shows up as falling recall on fresh labels while feature distributions look stable — or as new feature patterns with no labels yet.
- The label delay decides the monitoring architecture. For ninety days the only signals are feature drift, prediction drift, the review team's decisions on the flagged subset (a biased early label) and the merchant's own reports. Any dashboard that shows "precision this week" is computing on the fastest chargebacks, a subset that looks nothing like the population.
- Disputed labels are an attack surface. A fraudster who can get their transactions marked "not fraud" on dispute, or a merchant who mislabels friendly fraud, is writing the training set; a retraining pipeline that ingests labels without provenance is being poisoned by the people it is meant to catch (Data Poisoning).
The confusion matrix at review-queue capacity
Rare positives mean the two cells that matter are small and the two mistakes have nothing in common. A false positive is a good customer declined or delayed and a reviewer's time; a false negative is a chargeback, a penalty, and — because fraud rings probe — an invitation. The matrix below is drawn at the operating point the review capacity forces, with illustrative counts for one day.
Read the columns, not the diagonal. The right-hand column is the review queue and the payment declines; the bottom row is the fraud that reached the merchant. Accuracy would add the corners together and report a number dominated by the top-left cell, which nobody is deciding anything about.
The 600 flags are the review capacity. Moving the threshold trades cells in the right column for cells in the bottom row; which trade is right is a business number — the chargeback cost against the decline cost — not a property of the model.
Ninety days of not knowing
The architecture has to be drawn around the label delay because it decides what can be monitored when. For the first ninety days after any change the system has feature distributions, prediction distributions, the review team's decisions on the flagged subset, and nothing else. Reviewer decisions are a label for the flagged population only, and the flagged population is what the model chose to show them.
The diagram shows the two clocks. The scoring path runs in milliseconds and the label path runs in months, and they meet in the prediction log, which is the only place a transaction's features, score and eventual chargeback ever sit in the same row.
The adversary writes the training set
Two things make fraud different from ordinary concept drift. First, the positives adapt: whatever the model catches stops arriving, and whatever it misses arrives more. Falling recall on fresh labels is the expected steady state, not an incident, and the retraining trigger is tied to it. Second, the labels are contestable: a chargeback can be disputed and reversed, a merchant can mislabel a refund, and a ring that learns the dispute process learns how to write negatives into next quarter's training set.
The assumption a fraud system depends on is therefore about provenance rather than distribution: that every label entering retraining is one the adversary could not have chosen. When that fails the poisoning is quiet — the model becomes slightly more permissive about one pattern, and the metric that would show it is ninety days out.
A transaction's label in the training set reflects a resolved chargeback outcome with recorded provenance, not a dispute in progress or a label the counterparty could set.
holds when The label pipeline records source, dispute state and resolution date; unresolved disputes are held out; labels from a single merchant or cardholder are capped in influence; retraining data is at least the label-delay old.
breaks when Reviewer decisions are ingested as final labels; disputes are treated as negatives on filing; a bulk label correction arrives from a partner without provenance; retraining pulls the freshest quarter to "keep up".
respond Quarantine the affected label batch, retrain from the last clean cut, and treat the pattern the poisoned labels favoured as a rule until clean labels exist (Data Poisoning, The Model Supply Chain).
1-- one row per transaction scored at least 90 days ago2select t.txn_id,3 t.scored_at,4 t.artifact_version,5 case when c.status = 'lost' then 1 -- chargeback stood: fraud6 when c.status = 'won' then 0 -- disputed and reversed: not fraud7 when c.txn_id is null then 0 -- no chargeback in 90 days8 else null end as label, -- 'open': hold out of training9 c.status as label_status,10 c.resolved_at as label_resolved_at11from prediction_log t12left join chargebacks c13 on c.txn_id = t.txn_id14 and c.filed_at <= t.scored_at + interval '90 days'15where t.scored_at <= now() - interval '90 days';The null branch is the whole point: an open dispute is not a negative. The filter on scored_at is what stops the training set from containing the last quarter, which has features and no answers — and is exactly the quarter a "keep the model fresh" instinct reaches for.
How to build it
Most important first.
- Design the operating point from the review capacity: sort by score, take the top N per day, and report precision at N and recall at N on the validation weeks. The threshold is whatever score the Nth transaction has; it moves with volume and prevalence and is monitored as such.
- Keep the rules engine alongside the model, not under it: rules encode known patterns that must never pass (a card reported stolen), give the model a fallback when it times out, and provide an explainable reason code the review team and the disputes process need (The Rule Baseline, Serving Fallbacks).
- Compute the velocity features once, from the stream, and serve them to both training and scoring — log the serving vector and train on the log — so that the window definition cannot differ between paths (Feature Stores or a logged-feature training set; either works, one is required).
- Build the label pipeline with provenance: which chargeback, disputed by whom, resolved how, and when; hold disputed labels out of retraining until resolution; and monitor label arrival rates and dispute rates as first-class signals (Label Construction).
- Give the latency budget to the model explicitly and measure the breakdown — feature lookup, transformation, model, network — at the tail, because the payment endpoint's timeout is a hard limit and the fallback on timeout is a rule, not a guess (Latency Breakdown).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Precision at the review-queue capacity and recall at that capacity, on validation weeks split by time with a ninety-day gap, are the numbers that map to the decision. PR AUC summarises the trade-off across capacities.
- Online, before labels arrive: flags per day against capacity; the review team's agree rate on flagged transactions; feature drift per velocity feature; the share of requests served by the rule fallback.
- After ninety days: chargeback recall on the population, including transactions the model approved, and the dispute rate on the labels themselves. ROC AUC and accuracy are reported nowhere the decision is made.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The velocity features the model receives at scoring time are computed over the same windows, from the same event stream, as the vectors it was trained on, and the stream is fresh to within its stated lag.
- The review-queue capacity behind the threshold is the capacity the team actually has this week, and the score at the Nth flag is monitored as the operating point moves.
- Labels that enter the retraining set have resolved provenance, and disputed labels are held out until resolution.
- The fraud population the model faces is adapting, so recall on fresh labels is expected to decay and the retraining trigger is tied to that decay rather than to a calendar.
- Offline: time-split validation with a ninety-day label gap; precision and recall at the capacity N; a leakage audit on every post-transaction column (dispute status, chargeback reason codes) that could have entered the features.
- Online: shadow the model behind the rules engine for several weeks, comparing its flags to the reviewers' decisions and its latency breakdown to the budget, before it decides anything (Shadow Deployment).
- Over time: when chargebacks arrive, compute recall on the *approved* transactions — the ones the model let through — because that is the only unbiased population; compare against the rules engine on the same weeks.
What can go wrong
- The review team's decisions are used as early labels and the model is retrained on them; it learns to predict what reviewers flag, which is what the previous model flagged, and the loop closes (Feedback Loops).
- The stream feature service degrades and returns stale velocity counts; every transaction looks quiet, the model approves more, and nothing alerts because the endpoint is fast and error-free.
- A retrain on the last quarter includes unresolved disputes as negatives; a fraud ring that disputes aggressively has taught the model that its pattern is legitimate.
- The rule fallback is tuned to be conservative and the model times out under a traffic spike; the decline rate triples for an hour and good customers are the ones who notice.
- A capacity-based threshold means precision falls when fraud prevalence falls, and the review team will report the model "got worse" when the world got better.
- Holding disputed labels out of retraining delays learning about exactly the cases the adversary cares most about; including them lets the adversary write the training set. There is no setting that avoids both.
- Rules alongside the model are a second system to maintain, with their own owner and their own drift; without them the model has no fallback and no reason codes.
- "99% accuracy means the model is good." At a positive rate under one percent, approving everything scores higher. Accuracy measures the negatives; the decision is about the positives.
- "Recall dropped, so the model degraded — retrain." Check first whether the labels changed (a new dispute process), the features skewed (a stream lag), or the fraud moved (the retrain case). Only the third is fixed by retraining, and only if the new labels have arrived.
- "The model is fast enough; the endpoint p50 is well under budget." The payment endpoint fails at the tail. Measure p99 with the feature lookup included, under the traffic shape the busiest hour produces (Tail Latency: Why p50 Being Fine Does Not Help).
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DOMAIN-SPECIFICThe adversarial dynamic and the ninety-day disputed label are specific to payments fraud and its neighbours (account takeover, abuse); a medical screening model shares the rare positives and the asymmetric costs but its positives do not adapt to the classifier.
- SCALE-SPECIFICAt low transaction volume a rules engine plus a daily batch-scored review list may beat an online model on cost and on quality, because the review team is the bottleneck; the online design is forced by the payment-time decision, not by volume.
- SIMPLIFIEDCounts in the confusion matrix and any percentages in this lesson are illustrative and show the shape of the argument; they are not measurements of any real transaction population.
- CONTESTEDPractitioners disagree about whether to threshold or to calibrate and hand the probability to a downstream expected-value rule. Calibration lets the decline decision weigh transaction amount and customer value explicitly and stays valid as prevalence moves; the threshold camp answers that calibration decays fastest under exactly the adversarial drift fraud has, and a queue-capacity threshold is at least honest about what it is.
Where the depth lives
This domain teaches the model and hands the rest off by name.