Incident slo runbook

Incident SLO Runbook

Purpose

Use this skill to connect observability to action. Metrics and logs are not enough; each critical user journey needs an SLO, alert, owner, response path, and post-incident learning loop.

SLO Design

Define:

1. User journey or system capability.
2. SLI: request success, latency, freshness, durability, or job completion.
3. SLO target and measurement window.
4. Error budget and burn-rate alerts.
5. Exclusions with rationale.
6. Dashboard and data source.
7. Owner and escalation path.

Avoid vanity metrics. Prefer user-visible success and latency over internal counters unless internal counters are the only reliable proxy.

Runbook Requirements

Each runbook should include:

• Symptom and alert name.
• Impacted users or systems.
• First 5-minute checks.
• Triage decision tree.
• Mitigation steps with commands.
• Rollback or failover path.
• Escalation owner.
• Customer/support communication note.
• Postmortem trigger.

Commands must be safe to run or explicitly labeled destructive.

Incident Flow

1. Declare severity and incident commander.
2. Confirm impact from live evidence.
3. Stabilize with the lowest-risk mitigation.
4. Communicate status on a fixed cadence.
5. Preserve evidence before cleanup.
6. Write a blameless postmortem with action items and owners.

Output Shape

service_or_journey:
slo:
alerts:
dashboard_or_queries:
runbook:
escalation:
postmortem_template:
verification: