Bias assessment

Bias Assessment

Bias in AI systems causes real harm. A model that appears accurate overall can systematically disadvantage specific groups. You MUST complete this assessment before training or evaluating any model.

Phase 1: Data Audit (Before Training)

1.1 Check Representation

Run the representation check script:

python skills/bias-assessment/scripts/check_representation.py --data <your_dataset>

If no script applies, manually verify:

# For each protected attribute in your dataset:
for attr in ['age', 'sex', 'ethnicity', 'disability', 'postcode']:
    if attr in df.columns:
        print(f"\n{attr} distribution:")
        print(df[attr].value_counts(normalize=True))
        
        # Flag underrepresented groups (<5% of dataset)
        underrepresented = df[attr].value_counts(normalize=True)
        flagged = underrepresented[underrepresented < 0.05].index.tolist()
        if flagged:
            print(f"⚠️  UNDERREPRESENTED: {flagged}")

Stop and fix if: Any group that will be affected by predictions has <5% representation.

1.2 Check for Proxy Variables

Proxy variables appear neutral but encode protected characteristics:

Proxy Variable	May Encode
Postcode / ZIP code	Ethnicity, deprivation
Name	Ethnicity, sex
School attended	Socioeconomic status, ethnicity
Job title history	Sex, disability
Device type	Socioeconomic status

Action: For each proxy variable, decide: remove it, transform it, or document the risk explicitly.

1.3 Check Label Quality

Biased labels produce biased models:

• Were labels assigned by humans? → Check inter-annotator agreement across annotator demographics
• Were labels derived from historical decisions? → Those decisions may contain historical bias
• Are labels consistent across demographic groups? → Run: df.groupby(protected_attr)['label'].mean()

---

Phase 2: Model Evaluation (After Training)

2.1 Disaggregated Performance

NEVER report only aggregate metrics. Always disaggregate:

from sklearn.metrics import classification_report

for group_val in df[protected_attr].unique():
    mask = df[protected_attr] == group_val
    print(f"\n=== {protected_attr} = {group_val} ===")
    print(classification_report(y_true[mask], y_pred[mask]))

2.2 Fairness Metrics — Compute All Three

# 1. Demographic Parity: positive prediction rate per group
for group in groups:
    mask = df[attr] == group
    rate = y_pred[mask].mean()
    print(f"Demographic parity [{group}]: {rate:.3f}")

# 2. Equal Opportunity: true positive rate per group
for group in groups:
    mask = (df[attr] == group) & (y_true == 1)
    tpr = y_pred[mask].mean()
    print(f"Equal opportunity [{group}]: {tpr:.3f}")

# 3. Predictive Parity: precision per group
for group in groups:
    mask = df[attr] == group
    from sklearn.metrics import precision_score
    prec = precision_score(y_true[mask], y_pred[mask])
    print(f"Predictive parity [{group}]: {prec:.3f}")

2.3 Bias Thresholds — Do Not Proceed If Exceeded

Metric	Maximum Acceptable Gap	Action If Exceeded
Demographic parity difference	0.05	Investigate data; apply reweighting
Equal opportunity difference	0.05	Check label quality; consider threshold adjustment
Predictive parity difference	0.05	Review training data balance
False positive rate gap	0.05	Adjust decision threshold per group

---

Phase 3: Mitigation

If thresholds are exceeded, you MUST apply mitigation before proceeding:

Pre-processing

# Reweighting: give underrepresented groups more influence during training
from sklearn.utils.class_weight import compute_sample_weight
sample_weights = compute_sample_weight('balanced', y=df[protected_attr])
model.fit(X_train, y_train, sample_weight=sample_weights)

Post-processing

# Threshold adjustment: use different decision thresholds per group
thresholds = {}
for group in groups:
    mask = df[attr] == group
    # Find threshold that equalises FPR across groups
    thresholds[group] = find_threshold(y_true[mask], y_scores[mask], target_fpr)

Document your choice

Whatever mitigation you apply, add a comment in the code and update the model card.

---

Completion Checklist

Before leaving this skill:

• Representation checked for all relevant protected attributes
• Proxy variables identified and decision documented
• Label quality assessed
• Disaggregated performance metrics computed and recorded
• All three fairness metrics computed
• Any exceeded threshold addressed with documented mitigation
• Findings written to docs/bias-assessment-<date>.md

Now proceed to fairness-testing to write tests that will catch bias regressions.