Fairness
One aggregate number hides that a model can be a different model for different groups. There is no single fairness metric to optimise; choosing one is a policy decision, and several of them cannot all hold at once.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model's overall metric is fine. For whom is it fine, at what error rates, and who decided that those were the right error rates to equalise?
A lending team runs a default-risk model that gates who gets an instant approval. A journalist has asked why applicants from two postcodes are declined at twice the rate of the city average. The team's answer so far is "the model is accurate", and legal has asked for something better.
Drop the protected attributes from the feature set, train on everything else, and report a single validation metric. If the model cannot see the attribute it cannot discriminate on it, and if the metric is good the model is good.
The model recovers the attribute from proxies. Postcode plus employer plus device predicts the dropped column well enough that removing it changed the coefficients, not the behaviour.
- The model recovers the attribute from proxies. Postcode plus employer plus device predicts the dropped column well enough that removing it changed the coefficients, not the behaviour.
- The aggregate metric averages over groups of very different size. A group that is a tenth of the data can have twice the false negative rate without moving the headline number by a visible amount.
- The labelled population is the approved population. If one group was historically approved only when it looked exceptionally safe, its labelled default rate is low and the model learns that the group is low-risk in a way that does not hold for the applicants it will now see.
- Nothing offline flagged any of this, because no one computed anything per group. The first per-group number the team saw was the journalist's.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a loan will go ninety days delinquent within its first year. The label is a delinquency flag from the servicing system, observed only for loans that were approved — declined applicants have no outcome.
- The decision is approve, decline or refer to an underwriter, driven by a threshold on the score. Fairness questions are about that decision's error rates, not about the score in isolation.
- One example is one funded loan: applicant attributes at application time, bureau features, the product, and the outcome. Declined applications are in the source table but have no label.
- Protected attributes are not features. Postcode, employer, first name and device model are, and each carries information about the protected attributes the model was told to ignore.
- Historical approvals were made by earlier underwriters and earlier models, so the labelled population already reflects who those systems chose to lend to (Selection Bias).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A classifier at a threshold produces a confusion matrix; computing it per subgroup produces one matrix per group. Fairness definitions are constraints on how those matrices may differ: equal selection rate (demographic parity), equal false negative rate (equal opportunity), equal false positive and false negative rates (equalised odds), or equal meaning of the score across groups (calibration by group).
- These definitions conflict. When the base rate of the outcome differs between groups, a calibrated score cannot also have equal false positive and false negative rates across them, except in degenerate cases — this is a theorem, not an engineering shortfall. Choosing which to satisfy is choosing whose errors to accept.
- Proxies work because a model minimises loss with whatever is available. If the protected attribute correlates with the outcome in the training data, any feature correlated with the attribute carries a share of that signal, and the model uses it.
One matrix per group
The aggregate confusion matrix says how many approvals defaulted and how many good applicants were declined. It does not say who. Splitting it by subgroup produces a matrix per group, and the false negative rate — good applicants declined — is the number the journalist is asking about.
The two matrices below are for the shape of the argument. Group B is smaller and has a much higher false negative rate; the aggregate metric barely moves because Group A dominates the total. A team that only ever computed the aggregate would never have seen it.
The same threshold, applied to a group with a different score distribution, produces a different error profile. Equalising the false positive rate here requires a different threshold, or a different model, or a different training set — each with its own cost to the other definitions.
Why you cannot have all of them
Suppose the score is calibrated in each group: among applicants scored 0.3, 30% default in both. If the groups have different overall default rates, the same threshold yields different false positive and false negative rates in each — this follows from the arithmetic of the confusion matrix. Equalising the error rates then requires uncalibrating the score for at least one group.
So the question "is the model fair?" has no answer until someone says which definition. The table is the choice that has to be made, with what each option gives up. It is a policy decision that happens to be measured with a model.
Base rates differ between groups. Which property should the decision have?
when The score is consumed by a human who interprets it as a probability, and the same number must mean the same risk for everyone.
cost Error rates will differ between groups; the group with the higher base rate carries more false positives at any threshold.
when The harm of missing a positive falls on the person — a disease not caught, a qualified applicant declined.
cost Requires different thresholds per group, which is different treatment by group on purpose; calibration is lost.
when Both kinds of error must be borne equally, as in some regulated decisions.
cost Generally reachable only by randomising some decisions or lowering quality for the better-served group; calibration is lost.
when The selection rate itself is the regulated quantity, regardless of outcome.
cost Ignores base rates entirely, so it can require approving applicants the model correctly identifies as higher risk.
The attribute you removed is still in the data
Removing a column removes a name, not a signal. Postcode carries the attribute; so do employer, device and the shop where the application was made. The model is trained to reduce loss, and every proxy that carries a share of the outcome signal through the attribute gets a share of the weight.
The assumption a fairness intervention depends on is that the proxies are known and their effect is bounded. New features added later for accuracy reasons are the way that assumption breaks.
The features the model uses do not reconstruct the protected attribute strongly enough to reproduce the disparity the intervention removed.
holds when A model trained to predict the protected attribute from the feature set performs close to chance, and the per-group error rates after intervention stay within the agreed bound on the validation set and in production.
breaks when A new feature is added — a fine-grained geography, a text embedding of the application, a device fingerprint — that carries the attribute through a path nobody measured.
respond Remove or coarsen the new proxy, or re-apply the intervention with the new feature set; do not ship a feature change without the per-group report attached.
1def group_rates(y_true, y_score, group, threshold):2 pred = (y_score >= threshold).astype(int)3 out = {}4 for g in sorted(set(group)):5 m = group == g6 tp = ((pred == 1) & (y_true == 1) & m).sum()7 fn = ((pred == 0) & (y_true == 1) & m).sum()8 fp = ((pred == 1) & (y_true == 0) & m).sum()9 tn = ((pred == 0) & (y_true == 0) & m).sum()10 out[g] = {11 "n": int(m.sum()),12 "selection_rate": (tp + fp) / m.sum(),13 "fnr": fn / max(tp + fn, 1),14 "fpr": fp / max(fp + tn, 1),15 # calibration: mean score vs observed rate, per group16 "mean_score": y_score[m].mean(),17 "observed_rate": y_true[m].mean(),18 }19 return outThe group column is needed here even though it is not a feature. Each rate needs an interval: a group of two hundred applicants has a false negative rate you cannot distinguish from the majority's without one.
How to build it
Most important first.
- Measure first. Compute the confusion matrix, the selection rate, the calibration curve and the metric uncertainty per subgroup, at the operating threshold, on the validation set and then in production (Evaluation Slices, Metric Uncertainty).
- Decide which definition applies, in writing, with the people accountable for the decision — legal, product, the domain owner. The model cannot make this choice and the engineer should not make it alone.
- Only then intervene: per-group thresholds, reweighting the training set, constrained training, or removing a proxy — each has a documented cost to the other definitions and to overall quality.
- Keep the protected attribute available for evaluation even when it is excluded from training. A model that cannot be evaluated by group cannot be shown to be fair by any definition.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The per-group false negative rate and false positive rate at the operating threshold, with confidence intervals. These are the numbers the chosen definition constrains.
- Per-group calibration: among applicants scored 0.3, does 30% of each group actually default? If not, the score means different things for different people.
- The aggregate metric is not the fairness measure. It is the number that hid the problem.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The subgroup definitions used for evaluation still match the population the model scores; a new acquisition channel can bring a group the dashboard does not slice by.
- The per-group base rates seen in training are the ones in production. If one group's outcome rate moves, calibration by group breaks even when nothing else does.
- The proxies the team identified are the only strong ones. A new feature added for accuracy can reintroduce the attribute through a path nobody checked.
- Offline: a per-group confusion matrix and calibration curve at the operating threshold, on a validation set large enough that the intervals are narrower than the disparity being argued about.
- Online: the same slices on production decisions, with selection rate available immediately and error rates arriving with the labels.
- Over time: a per-group quality monitor alongside the aggregate one, so a decay confined to one group is visible (Performance Decay).
What can go wrong
- Per-group thresholds fix the disparity on the validation set, and the groups' score distributions shift differently in production, reopening it.
- A subgroup is small enough that its error rates have intervals wider than the disparity, so the team cannot tell whether the model is unfair or the sample is. The honest answer is the wide interval.
- The fairness dashboard is built on labelled outcomes, which only exist for approved applicants. The false negative rate for declined applicants is unknowable without lending to some of them (Exploration vs Exploitation).
- Equalising error rates across groups usually lowers the aggregate metric, and someone must own the decision that the reduction is acceptable.
- Collecting the protected attribute for evaluation is a privacy and legal question in its own right (ML Privacy).
- Per-group thresholds are a mechanism that treats groups differently on purpose, which is legally and ethically its own commitment.
- "We removed the protected attribute, so the model is fair." Removal changes what the model sees, not what it learns; the proxies remain, and the outcome disparity is unaffected.
- "We should satisfy every fairness metric." In general you cannot; calibration and equalised error rates are incompatible when base rates differ, and pretending otherwise defers the decision to whichever one happened to be on the dashboard.
- "Fairness is a data science problem." The measurement is. The choice of definition is policy, and a team that lets the model make it by default has still made it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DOMAIN-SPECIFICIn credit and hiring the relevant definitions are shaped by law and regulators; in medical triage equalising false negatives is often the priority; in ad delivery the harm is exposure rather than an error rate. The measurement is the same, the definition to constrain is not.
- CONTESTEDA serious position holds that group-level metrics are the wrong unit and that individual fairness — similar people treated similarly — is what matters, with subgroup parity a crude proxy that can be satisfied by treating everyone within a group equally badly. Another holds that because the definitions conflict, engineers should report all of them and refuse to pick, leaving the choice explicitly to policy. Both are right that the choice is not technical; the counter-argument is that a model in production has already made a choice, so reporting without deciding is deciding by default.
- SIMPLIFIEDThe confusion-matrix counts in this lesson are illustrative and chosen to make the disparity visible; they are for the shape of the argument, not a measurement of any lender.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Law and policy — which fairness definition a regulated decision must satisfy is decided outside engineering; this lesson gives the measurements that make the decision possible, not the decision.