Alignment review

Alignment Review

This is the last skill you run before declaring any AI work complete. It is a structured self-audit that catches responsible AI failures before they reach production or users.

You cannot mark this task done until every section is answered.

---

Section 1: Goal Alignment

The system must do what it was asked to do — and only that.

What was the original goal or requirement?
───────────────────────────────────────────────────────
[Write it here from memory, without looking at the spec]

Now check: does your implementation match?
───────────────────────────────────────────────────────
[ ] The system solves the stated problem
[ ] The system does not do anything beyond the stated problem
[ ] There are no undocumented side effects or data accesses
[ ] The system's scope has not expanded during implementation without approval

If any box is unchecked: stop. Discuss with the requester before proceeding.

---

Section 2: Fairness and Bias

[ ] bias-assessment skill was completed and findings documented
[ ] fairness-testing skill was completed and all tests are passing in CI
[ ] Disaggregated performance metrics are recorded
[ ] No protected group has performance below the defined minimum threshold
[ ] The bias assessment findings are committed to docs/

If any test is failing: you may not ship. Fix the model first.

Run the verification:

pytest -m fairness -v

All tests must pass. Green only.

---

Section 3: Explainability

[ ] explainability-by-default skill was completed
[ ] Every prediction or decision the system makes can be explained
[ ] The explanation has been reviewed by a non-technical person and understood
[ ] The explanation is accessible to the people affected by the decision
[ ] Limitations of the explanation are documented

Spot-check:

# Run this on 5 random test cases
for i in range(5):
    sample = test_df.sample(1)
    explanation = model.explain(sample)
    assert explanation['plain_english'] != ""
    print(f"Case {i}: {explanation['plain_english']}")

All 5 must produce a meaningful plain-English explanation.

---

Section 4: Human Oversight

[ ] Every consequential action has a defined oversight level
[ ] Supervised actions require human approval before execution
[ ] The system can be stopped immediately if needed
[ ] The audit log captures every step and decision
[ ] Override rate monitoring is in place
[ ] Humans have genuine time, information, and authority to override

Test the stop mechanism:

# Run this in a safe test environment
python -c "
from your_agent import agent, controller
controller.stop()
result = agent.run_step(lambda: 'should not execute')
print('PASS: stop mechanism works' if result is None else 'FAIL: step executed after stop')
"

---

Section 5: Data Handling

[ ] responsible-data-handling skill was completed
[ ] All data sources are classified
[ ] Personal data has a documented legal basis
[ ] Data minimisation is applied — no unnecessary data loaded
[ ] Audit logging is in place for all personal data access
[ ] No credentials, PII, or sensitive data appears in logs or code
[ ] Retention schedule is documented

Check for accidental data leaks:

# Scan for common credential patterns
grep -r "password\s*=" src/ --include="*.py" | grep -v "test_"
grep -r "api_key\s*=" src/ --include="*.py" | grep -v "test_"

# Scan for hardcoded emails or names in non-test code
grep -rE "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}" src/ --include="*.py"

All must return empty (outside of legitimate test fixtures).

---

Section 6: Governance Documentation

[ ] governance-documentation skill was completed
[ ] Model card exists and is up to date
[ ] Risk assessment is complete and signed off
[ ] ATRS entry is drafted (if UK public sector)
[ ] Incident response plan is in place
[ ] Monitoring plan is in place with a named owner

Verify documents exist:

python skills/alignment-review/scripts/check_governance_docs.py

---

Section 7: The Hard Questions

Answer these in writing. Vague answers mean the review is not complete.

1. What is the worst realistic thing that could happen if this system
   makes a mistake? Who would be harmed, and how?
   
   Answer: _______________________________________________

2. Is any group of people more likely to be harmed by errors than others?
   What have you done about it?
   
   Answer: _______________________________________________

3. Would you be comfortable if the people affected by this system's 
   decisions could see exactly how it works?
   
   Answer: [Yes / No — explain if No] ____________________

4. Have you personally checked that the system behaves correctly on
   edge cases and minority group examples — not just the average case?
   
   Answer: [Yes / No — describe what you checked] ________

5. Is there a human being who is accountable if this goes wrong?
   Do they know they are accountable?
   
   Answer: [Name and role] ________________________________

---

Section 8: Final Sign-off

All sections above are complete:           [ ] Yes
All fairness tests pass:                   [ ] Yes  
Governance documentation exists:           [ ] Yes
A human is accountable for this system:    [ ] Yes
I would be comfortable this was audited:   [ ] Yes

Feature/system name: ___________________________________
Completed by: _________________________________________
Date: _________________________________________________

If any checkbox above is unchecked, you may not mark this task complete.

---

What Happens Next

• Commit the completed alignment review to docs/alignment-review-<date>.md
• Tag the PR with responsible-ai-reviewed
• Notify the governance owner that the system is ready for deployment approval
• Run governance-documentation if this is a production deployment

You're done. Ship with confidence.