Incident response

Incident Response

You are managing an incident through its full lifecycle using structured incident management practices.

Red Flags — STOP if you're

• Fixing before communicating (stakeholders must know first)
• Skipping severity classification
• Writing a postmortem with blame (blameless only)
• Closing an incident without prevention items
• Ignoring rollback as a mitigation option

Communicate first. Fix second. Learn always.

---

Pre-Check

1. Check for Draft context:

ls draft/ 2>/dev/null

This skill works standalone — incidents don't wait for project setup.

2. If available, follow the base procedure in core/shared/draft-context-loading.md.

Step 1: Parse Arguments

• /draft:incident-response new <description> — Start new incident
• /draft:incident-response update <status> — Post status update
• /draft:incident-response postmortem — Generate postmortem report
• /draft:incident-response (no args) — Interactive: ask which mode

---

NEW Mode — Start Incident

Step 2: Triage

Classify severity:

Level	Response Time	Who	Examples
SEV1	Immediate, all-hands	Entire team	Data loss, complete outage, security breach
SEV2	15 minutes	On-call + team lead	Major feature broken, significant degradation
SEV3	1 hour	On-call	Minor feature broken, workaround exists
SEV4	Next business day	Assigned engineer	Cosmetic issue, minor inconvenience

Assess:

1. What is broken? (from description or Jira ticket)
2. Who is affected? (from draft/product.md user types if available)
3. What is the blast radius? (from draft/.ai-context.md service topology if available)
4. Is data at risk? (escalate to SEV1 if yes)

Step 3: Communicate

Generate initial status update:

INCIDENT: {description}
Severity: SEV{1-4}
Impact: {who/what is affected}
Status: Investigating
Commander: {name or "unassigned"}
Next update: {time — SEV1: 15min, SEV2: 30min, SEV3: 1hr}

Step 4: Gather Evidence

• If Jira ticket linked: pull details via MCP (get_issue, get_issue_description, get_issue_comments)
• Extract URLs and log paths from ticket
• Use curl/wget to fetch dashboards or error pages mentioned
• Use ssh to access remote log paths if mentioned
• If GitHub MCP available: check recent deployments (last 24h)
• Record all evidence in incident timeline

Step 5: Mitigate

Following core/agents/ops.md production-safety mindset:

1. Can we rollback? If yes and severity ≥ SEV2: recommend rollback first, investigate after
2. Can we hotfix? If rollback not possible: identify minimal fix
3. Can we mitigate? Feature flag, config change, traffic routing
4. Need to escalate? If none of above work, escalate severity

Document all actions taken with timestamps.

Step 6: Save Incident File

Save to: draft/incidents/incident-<timestamp>.md or draft/tracks/<id>/incident.md

# Incident: {description}

| Field | Value |
|-------|-------|
| **Severity** | SEV{N} |
| **Status** | {Investigating/Mitigating/Resolved} |
| **Started** | {timestamp} |
| **Commander** | {name} |

## Timeline
| Time | Action |
|------|--------|
| {time} | Incident detected |
| {time} | Triage: classified as SEV{N} |
| {time} | {mitigation action} |

## Evidence
| Source | Finding |
|--------|---------|
| {source} | {finding} |

## Status Updates
{chronological updates}

---

UPDATE Mode

1. Read existing incident file
2. Add new timeline entry with timestamp
3. Update status field if changed
4. Update severity if changed (with justification)
5. Generate formatted status update for stakeholders

---

POSTMORTEM Mode

Step 2: Gather Timeline

• Read incident file for timeline and evidence
• git log for related commits during incident window
• If Jira MCP: pull ticket history and transitions
• If GitHub MCP: pull changes submitted during/after incident

Step 3: Root Cause Analysis

Reference core/agents/rca.md methodology:

1. 5 Whys Analysis:

   • Why did {symptom} happen? → Because {cause 1}
   • Why {cause 1}? → Because {cause 2}
   • Continue until root cause reached (typically 3-5 levels)

2. Root Cause Classification:

   • Logic error | Race condition | Data corruption | Configuration error
   • Dependency failure | Capacity exceeded | Security exploit | Human error

3. Detection Lag: When was the bug introduced vs when was it detected?

4. SLO Impact: Which SLOs were affected and by how much?

5. HLD Claims vs Reality (): If the affected service has a hld.md (search draft/tracks/*/hld.md for §Detailed Design components matching the failing module), compare incident behavior against HLD claims:

   • Did §Resiliency claims (graceful degradation, circuit breakers, timeout handling) actually hold during the incident?
   • Did §Multi-tenancy claims (tenant isolation, predictable performance) hold? Did one tenant impact another?
   • Did §Upgrade claims (backward compat, dependent service order) hold? Did an upgrade trigger this?
   • Did §Scale claims hold under the load that triggered the incident?
   • Did §Observability claims hold — were the listed key metrics sufficient for detection?
   • Cite the specific HLD §section using section text (draft/tracks/<id>/hld.md §Resiliency) for each gap — avoid markdown anchor slugs since renderers (GitHub, mkdocs, Hugo) generate different slugs for nested headings. These citations feed the §Action Items as "amend HLD §X — claim was {claim} but reality showed {reality}."

Step 4: Generate Postmortem

MANDATORY: Include YAML frontmatter with git metadata. Follow core/shared/git-report-metadata.md.

Save to: draft/incidents/postmortem-<timestamp>.md with symlink postmortem-latest.md Or track-scoped: draft/tracks/<id>/postmortem.md

# Postmortem: {incident title}

## Summary
{2-3 sentences: what happened, impact, duration}

## Impact
- **Duration:** {start} to {end} ({total time})
- **Users affected:** {count or percentage}
- **SLO impact:** {which SLOs, by how much}
- **Data impact:** {any data loss or corruption}

## Timeline
| Time | Event |
|------|-------|
| {time} | {event} |

## Root Cause
{1-2 sentence root cause statement}

### 5 Whys
1. Why? → {answer}
2. Why? → {answer}
...

### Classification
- **Type:** {classification}
- **Detection Lag:** {introduced} → {detected} = {gap}

## What Went Well
- {positive observations}

## What Went Wrong
- {things that made the incident worse}

## Design Claims vs Reality
{populated when an HLD was available — list each HLD claim that did not hold, citing the specific §section}

| HLD Section | Claim | Reality During Incident | Recommended HLD Amendment |
|-------------|-------|-------------------------|---------------------------|
| §Resiliency | {what was claimed} | {what actually happened} | {how to update HLD} |

## Action Items
| # | Action | Owner | Deadline | Status |
|---|--------|-------|----------|--------|
| 1 | {detection improvement} | {name} | {date} | [ ] |
| 2 | {process improvement} | {name} | {date} | [ ] |
| 3 | {code improvement} | {name} | {date} | [ ] |
| 4 | Amend `draft/tracks/<id>/hld.md` §{section} (if claim drift identified) | {design owner} | {date} | [ ] |

Step 5: Jira Sync

Follow core/shared/jira-sync.md:

• Attach postmortem to Jira ticket
• Post comment: "[draft] Postmortem complete. Root cause: {1-line summary}. {N} action items."

⚠️ Test Writing Guardrail: If postmortem identifies missing tests, ASK: "Want me to create regression test tasks? [Y/n]"

Cross-Skill Dispatch

• Triggered by: /draft:new-track when incident keywords detected in description
• Postmortem feeds into: /draft:regression (find the breaking commit), /draft:learn (update guardrails)
• Can create: Bug track via /draft:new-track for the fix

Error Handling

If no incident file found (update/postmortem mode): List available incidents, ask which one If no Jira ticket: Proceed without sync, note: "Link a Jira ticket for automatic sync"