Product ai risk

Product AI Risk

Model how the full sociotechnical system can fail, detect failure early, limit exposure, and recover safely.

Inputs

• Users, affected non-users, workflow, and environment
• Model capabilities, data flows, tools, and external actions
• Decision impact and error tolerance
• Human oversight and operational ownership
• Evaluation, monitoring, rollback, and incident capabilities
• Regulatory, contractual, privacy, and policy constraints

Workflow

1. Map the system boundary: inputs, data, model, retrieval, tools, people, actions, feedback, and downstream dependencies.
2. Identify harms from wrong output, omission, delay, disclosure, manipulation, overreliance, disparate performance, and unauthorized action.
3. Assess severity, likelihood, detectability, exposure, and reversibility by affected group.
4. Define preventive controls: scope limits, permissions, data minimization, grounding, validation, confirmation, rate limits, and human approval.
5. Define detective controls: quality slices, drift, data integrity, latency, cost, abuse, override, and incident signals.
6. Design uncertainty communication, safe fallback, correction, appeal, and manual recovery.
7. Set staged exposure, kill switch, rollback, and incident ownership.
8. Establish launch blockers, accepted risks, residual risk owner, and review schedule.
9. Escalate high-impact or regulated decisions to qualified legal, security, privacy, safety, and domain owners.

Output contract

Return system map, risk register, affected groups, controls, monitoring, human-oversight design, fallback and recovery plan, launch blockers, residual risk acceptance, owners, and review triggers.

Quality gate

• The review covers the workflow and organization, not only the model.
• Critical actions use least privilege and confirmation appropriate to impact.
• Distribution shift and data feedback loops are monitored.
• Safe failure is usable under real operating conditions.
• Residual risk has an accountable human owner.

Avoid

• A generic ethics checklist without system-specific failure paths
• Treating disclaimers as controls
• Human review without time, expertise, authority, or interface support
• Monitoring averages that hide harmed segments
• Launching without rollback because the model passed offline tests

Source grounding

Operational synthesis informed by uncertainty, trust, feedback, and responsible AI practices in Building AI-Powered Products and data quality, distribution shift, monitoring, latency, and feedback-loop principles in Designing Machine Learning Systems.