Ai incident response

AI Incident Response

An AI incident is any event where an AI system's behaviour deviates from its intended, safe, and fair operation. Do not minimise, ignore, or quietly fix these. Follow this process every time.

Immediate Triage (First 15 Minutes)

1. Stop and Assess

Before doing anything else, answer these questions:

Is the system currently causing or about to cause active harm?
  YES → Immediately escalate to P1. Skip to EMERGENCY SHUTDOWN.
  NO  → Continue triage.

Is the incident affecting real citizens / users right now?
  YES → Treat as P1 or P2.
  NO  → Continue triage.

Is personal or sensitive data involved?
  YES → ICO notification may be required within 72 hours.
  NO  → Continue triage.

2. Classify the Incident

Type	Examples
Performance degradation	Accuracy drops significantly, response times increase
Fairness violation	Metrics show disproportionate impact on a group
Safety failure	Harmful, dangerous, or illegal output generated
Data incident	Unauthorised access, accidental disclosure, data poisoning
Manipulation	Prompt injection, adversarial attack, model inversion
Compliance failure	GDPR breach, ATRS not updated, override log missing

3. Assign Severity

P1 — Critical: System actively causing harm. Shut down immediately.
P2 — High:     Significant risk of harm. Pause within 4 hours.
P3 — Medium:   Fairness or performance threshold exceeded. Fix within 24 hours.
P4 — Low:      Anomaly detected. Investigate within 72 hours.

---

Emergency Shutdown (P1 Only)

Execute this in order. Do not skip steps.

# Step 1: Stop new requests immediately
# (Replace with your actual shutdown command)
systemctl stop ai-inference-service
# or: kubectl scale deployment ai-service --replicas=0
# or: disable the API gateway route

# Step 2: Verify the system is stopped
curl -f https://your-ai-endpoint/health && echo "STILL RUNNING - escalate" || echo "System stopped"

# Step 3: Preserve all logs — do NOT rotate or delete
cp -r /var/log/ai-service/ /var/log/ai-service-incident-backup-$(date +%Y%m%d%H%M%S)/

# Step 4: Activate manual fallback
# (Document your manual fallback procedure here)
echo "Activating manual fallback process — notify [team lead name]"

# Step 5: Notify emergency contacts immediately
# (Add your actual contacts)

---

Investigation Process

Preserve Evidence First

import json
import shutil
from datetime import datetime, timezone
from pathlib import Path

def preserve_incident_evidence(incident_id: str) -> Path:
    """
    Create a timestamped evidence package.
    Call this IMMEDIATELY when an incident is identified.
    Do not alter any logs before running this.
    """
    timestamp = datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')
    evidence_dir = Path(f'incidents/{incident_id}_{timestamp}')
    evidence_dir.mkdir(parents=True, exist_ok=True)
    
    # Copy all relevant logs
    log_sources = [
        '/var/log/ai-service/',
        'agent_audit.jsonl',
        'model_predictions.log',
        'fairness_metrics.log',
    ]
    
    for source in log_sources:
        src = Path(source)
        if src.exists():
            dest = evidence_dir / src.name
            if src.is_dir():
                shutil.copytree(src, dest)
            else:
                shutil.copy2(src, dest)
    
    # Record incident metadata
    metadata = {
        'incident_id': incident_id,
        'preserved_at': datetime.now(timezone.utc).isoformat(),
        'evidence_location': str(evidence_dir),
    }
    (evidence_dir / 'metadata.json').write_text(json.dumps(metadata, indent=2))
    
    print(f"Evidence preserved at: {evidence_dir}")
    return evidence_dir

Root Cause Analysis

Work through these systematically — do not jump to conclusions:

1. WHAT happened?
   - Exact behaviour observed
   - First occurrence timestamp
   - Frequency / scale of impact
   - Which model version / deployment?

2. WHEN did it start?
   - Compare with last known good state
   - Any deployments, data changes, or config changes before onset?
   - Check model drift metrics around that time

3. WHO was affected?
   - Number of users / citizens
   - Any patterns in affected group (age, location, ethnicity)?
   - Is the impact disproportionate for any group?

4. WHY did it happen?
   - Data distribution shift?
   - Model degradation / drift?
   - Adversarial input?
   - Code change / configuration error?
   - Training data contamination?

5. HOW do we verify the root cause?
   - Reproduce the failure in a safe environment
   - Test the hypothesis before declaring the cause

---

Fairness Incident Protocol

If the incident involves disproportionate impact on a protected group:

def investigate_fairness_incident(
    model,
    test_data,
    affected_group_attr: str,
    affected_group_val: str,
    incident_period_start: str,
    incident_period_end: str,
) -> dict:
    """Structured fairness incident investigation."""
    from sklearn.metrics import classification_report
    
    results = {}
    
    # Compare performance: affected group vs others
    mask_affected = test_data[affected_group_attr] == affected_group_val
    mask_others = ~mask_affected
    
    results['affected_group'] = classification_report(
        test_data[mask_affected]['label'],
        model.predict(test_data[mask_affected].drop(columns=['label'])),
        output_dict=True,
    )
    results['other_groups'] = classification_report(
        test_data[mask_others]['label'],
        model.predict(test_data[mask_others].drop(columns=['label'])),
        output_dict=True,
    )
    
    # Check if this was a regression from baseline
    # Load baseline from fairness_baseline.json
    ...
    
    return results

---

Incident Log Template

Every incident must be logged. No exceptions.

## Incident Report — [INCIDENT-ID]

**Date/Time Detected:** 
**Detected By:** 
**Severity:** [P1 / P2 / P3 / P4]
**System Affected:** 
**Incident Type:** 

### What Happened
[Plain-English description of the behaviour observed]

### Impact
- Users/citizens affected: 
- Duration: 
- Decisions affected: 
- Disproportionate impact on any group: [Yes/No — detail]

### Immediate Actions Taken
| Time | Action | By Whom |
|---|---|---|
| | | |

### Root Cause
[Once determined — do not fill until investigation complete]

### Resolution
[What was done to fix the problem]

### Prevention
[What changes prevent this from happening again]

### Lessons Learned
[What we'd do differently]

### Sign-off
- Investigated by: 
- Reviewed by (SRO): 
- Date closed: 
- Post-incident review scheduled: [Yes/No — date]

---

Regulatory Notification Decision Tree

Was personal data involved in the incident?
  NO  → No mandatory ICO notification
  YES ↓
Was there a risk to individuals' rights and freedoms?
  NO  → Document internally; consider voluntary notification
  YES ↓
Notify ICO within 72 hours of becoming aware.
Include: nature of breach, categories of data, approximate number of people,
         likely consequences, measures taken or proposed.

ICO breach notification portal: https://ico.org.uk/for-organisations/report-a-breach/

---

Completion Checklist

• Incident classified and severity assigned within 15 minutes
• Emergency shutdown executed if P1
• Evidence preserved before any investigation
• Incident log opened and timestamped
• Root cause investigation completed
• Regulatory notification assessed (and made within 72h if required)
• Fix deployed and verified
• Fairness metrics re-checked after fix
• Incident report completed and signed off
• Post-incident review scheduled (P1/P2 mandatory)
• Governance board notified of P1/P2 incidents

Do not close the incident until all items are checked.