Disparate impact audit

Disparate-Impact Audit

When a system decides things about people, one number — "94% accurate" — is a trap. It hides whose 6% the errors fall on, and automated errors have a consistent habit of concentrating on the people least able to absorb or contest them. A fraud system flags 48,000 people with a 93% error rate. A benefits algorithm weights nationality and pushes families into homelessness. A hiring model rejects the same applicant at every employer using it. This skill is the playbook for finding those failures before deployment — and it starts from a rule: a system that decides about people is not evaluated by its average.

The six checks

Run all six. Each is a distinct way these systems harm people that aggregate accuracy conceals.

1. Disaggregated error rates — whose errors?

• Break false-positive and false-negative rates out by group (race, gender, age, disability, income, language, geography). The aggregate can look fine while one group's error rate is multiples higher — e.g. facial-recognition false-positive rates varying by two orders of magnitude across demographic pairings.
• Apply the four-fifths rule as a screen: if a protected group's selection (or adverse) rate is below 80% of the most-favored group's, that's a legal red flag. Pooling across all decisions can wash this out — check per decision/posting, not just in aggregate.
• False positives and false negatives are not symmetric in harm. Name which one this system inflicts and on whom (a missed fraud flag vs. a wrongful denial land on different people).

2. Construct validity — is it measuring the thing, or a proxy for it?

This is the deepest check, and the one more data won't fix.

• Systems often claim to measure a target (maltreatment risk, job fit, health need) but actually measure administrative proxies that encode historical surveillance and bias: benefits receipt, prior contact with the system, diagnosis codes. A parent flagged "high risk" because their life is more surveilled by the state, not because they pose more risk.
• Ask: what is the label actually recording? If the training label is "was investigated" rather than "was actually harmful," the model reproduces who got investigated and calls it risk. That's a construct-validity failure, not a tuning problem — it will not resolve with more compute.

3. The rubber-stamp check — is "human in the loop" real?

• A human reviewer is the standard safeguard and it frequently fails silently: reviewers mirror the algorithm's recommendation (up to ~90% in some studies), or get a review window too short to think (seconds per case). That's automation with a human alibi, not human judgment.
• Test it: do humans override the system enough to matter, and — critically — do their overrides reduce disparity? In at least one benchmark case, caseworkers overriding the algorithm reduced racial disparity, meaning the humans were the fairness mechanism and the algorithm the bias source. Measure override rate and its direction.

4. Appeal suppression — who gets to contest, and who actually does?

• When denials are appealed, they're often overturned at high rates (80%+ in some settings) — but only a small fraction are ever appealed (low double digits). That gap means many wrong decisions stand simply because contesting them is hard.
• The decisive missing number is usually appeal rate disaggregated by income/race/education. Those least able to navigate an appeal absorb the most uncontested errors. If that data doesn't exist, its absence is itself a finding — flag it as the highest-value thing to measure.

5. Monoculture — correlated rejection at scale

• When many deciders use the same vendor/model, its bias isn't diluted across independent judgments — it's replicated. An applicant rejected by one becomes an applicant rejected by all of them. Per-employer bias can look small while system-wide a group is locked out everywhere at once (e.g. a slice of applicants rejected by every job using the same screener).
• Ask: how many deciders share this model, and does a rejection here follow the person elsewhere? Shared scores that persist across time and employers turn one biased judgment into a structural barrier.

6. Benchmark-vs-deployment & feedback loops

• Vendor fairness claims need the same skepticism as any performance claim (see validate-the-claim): commissioned "no bias found" studies vs. independent audits reaching the opposite result.
• Watch for feedback loops: a system trained on enforcement data directs enforcement, which generates more of the same data, which confirms the system. The output can't validate the model when the output shaped the training set.

Procedure

1. Identify the decision the system makes and its consequence for the person.
2. Demand disaggregated FP/FN rates by group; apply four-fifths per-decision.
3. Interrogate the training label — target construct or biased proxy?
4. Measure human-override rate and whether overrides cut or track disparity.
5. Check appeal reversal rate and appeal access by group; flag if the disaggregation is missing.
6. Map monoculture — how many deciders share the model, does rejection follow the person.
7. Independent (not commissioned) validation; check for feedback loops.
8. Verdict: deploy with named guardrails | pilot with instrumentation | do-not-deploy — and the specific data still needed.

Output template

SYSTEM: <what it decides> | CONSEQUENCE TO PERSON: <what a wrong call costs them>

1. ERROR RATES BY GROUP: <FP/FN disaggregated; four-fifths per-decision result>
2. CONSTRUCT VALIDITY: <target vs. proxy — what's the label actually recording?>
3. HUMAN REVIEW: <override rate; do overrides reduce or mirror disparity?>
4. APPEALS: <reversal rate; access by group; missing disaggregation?>
5. MONOCULTURE: <how many deciders share it; does rejection follow the person?>
6. VALIDATION/LOOPS: <independent audit? feedback loop present?>

MISSING DATA THAT MATTERS MOST: <the highest-value unmeasured thing>
VERDICT: deploy-with-guardrails | pilot | do-not-deploy — <why>

Example (compressed)

System: AI résumé screener used by 150+ employers via one vendor.

1. ERROR RATES: aggregate adverse-impact ~zero — but disaggregated per posting,
   a quarter of Black applicants and a sixth of Asian applicants hit roles
   crossing the four-fifths threshold. Pooling hid it.
2. CONSTRUCT VALIDITY: "predicts good hire" trained on past hiring outcomes —
   i.e. past hiring bias. Label records who got hired before, not who performs.
3. HUMAN REVIEW: recruiters mirror the AI's ranking up to ~90% of the time.
   The "human in the loop" is a rubber stamp.
4. APPEALS: applicants can't appeal or even see the score. No contest path.
5. MONOCULTURE: shared model + scores that persist for months → an applicant
   rejected once is rejected across every employer using it. Correlated lockout,
   not distributed chance.
6. VALIDATION: vendor's own audit says "fair"; independent audit at scale found
   the per-posting disparities above.

MISSING DATA: per-posting adverse-impact monitoring by group, published.
VERDICT: do-not-deploy as-is. At minimum: per-posting four-fifths monitoring,
   real appeal path, and breaking score persistence across employers.

Gotchas

• Aggregate fairness is not fairness. Per-decision and per-group is where these systems fail; the pooled number is designed to look clean.
• Construct validity won't yield to more data. If the label encodes bias, a bigger model reproduces it more efficiently. This is the check people most often skip.
• "Human in the loop" is a claim to test, not a safeguard to assume. Measure the override rate and its direction.
• Pairs with disaggregate (its dimensions drive the group breakdown), validate-the-claim (vendor fairness studies), and value-frames (distributive-justice frame on who bears the error).