agentsclimarketplace

Skill incident management

Skill saitarrun/sdlc-ai-workflow/skills/skill-incident-management

A comprehensive Claude Code plugin that automates the complete Software Development Lifecycle with 20 role-specific agents, 12 knowledge skills, and 8 commands, all guided by principles.

Install
npx -y skills add saitarrun/sdlc-ai-workflow --skill skill-incident-management

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Incident response, postmortem facilitation, root cause analysis (RCA), blameless culture, incident timeline documentation, and incident classification. Use when responding to production incidents, writing incident reports, facilitating postmortems, or improving incident response processes.

SKILL.md

6.6 KB, as published. Nobody here has run it

Skill: Incident Management

Systematic incident response and postmortem process for production issues. Emphasizes speed (MTTR), clarity (timeline), and learning (blameless culture).

Incident Classification (SEV Levels)

SEV 1 (Critical)

  • Entire service down OR
  • Core functionality unavailable AND
  • Large user base affected AND
  • Revenue/reputation impact
  • MTTR target: < 30 minutes
  • Declare: Immediately
  • Escalation: Engineering Manager + Tech Lead + On-Call lead

SEV 2 (High)

  • Partial degradation (slow, some users affected) OR
  • Non-core features completely down
  • MTTR target: < 2 hours
  • Declare: Within 15 min
  • Escalation: Tech Lead + Service Owner

SEV 3 (Medium)

  • Non-critical feature broken OR
  • Workaround exists AND
  • Low user impact
  • MTTR target: < 24 hours
  • Declare: Next business day
  • Escalation: Service Owner

SEV 4 (Low)

  • Minor issue, cosmetic OR
  • Zero user impact
  • Can wait for next release
  • Escalation: None (treat as bug)

Incident Timeline Template

## Incident: [Title]

### Summary
- Service: [service-name]
- Duration: [start] - [end] (X minutes)
- Users affected: [X]%
- Severity: SEV [1-4]

### Timeline

| Time | Event | Owner |
|------|-------|-------|
| 14:32 UTC | ❌ Alert: API error rate > 10% | Monitoring |
| 14:33 UTC | 🟡 Incident declared (SEV 2) | On-Call |
| 14:35 UTC | 🔍 Root cause identified: DB connection pool exhausted | Backend lead |
| 14:37 UTC | 🔧 Mitigation: Increased pool size, restarted service | DevOps |
| 14:39 UTC | ✅ Incident resolved, metrics normal | Monitoring |
| 14:45 UTC | ✅ All systems nominal (monitoring for 10 min) | On-Call |

### Root Cause
Database connection pool was set to 50 but app scaled to 8 instances (50 x 8 = 400 connections but pool limit was 50). New deployment didn't adjust pool size.

### Contributing Factors
1. No automated pool size calculation (manual config)
2. Pre-deployment validation didn't catch pool exhaustion risk
3. Alerting was on error rate, not pool usage (too late)

### Resolution
1. Temporarily increased pool size from 50 to 200 (immediate)
2. Disabled auto-scaling during hot-fix (reduced concurrent instances)

### Monitoring During Incident
- API error rate: 8% → 0.2% after fix
- DB response time: 2s → 50ms after fix
- User-visible errors: Resolved

### Follow-up Actions
1. [Action] Implement auto-sizing of connection pool based on instance count
2. [Action] Add alert on pool usage % (critical at 80%+)
3. [Action] Add pre-deployment validation check for pool size
4. [Action] Document connection pool sizing in runbook

Postmortem Process (Blameless)

Before Postmortem

  • Wait 24-48 hours (emotions settle, facts clear)
  • Send survey to participants (get thoughts while fresh)
  • Collect logs, metrics, alerts

During Postmortem (60 min)

## Postmortem: [Service] incident on [Date]

### 1. Incident Summary (5 min)
[What happened, impact, duration]

### 2. Timeline (10 min)
[When did each major event occur? Focus on facts, not blame]

### 3. Root Cause Analysis (15 min)

**Ask "Why?" 5 times**:
1. Why did error rate spike?
   → Connection pool exhausted
2. Why was pool exhausted?
   → Pool size fixed at 50, scaled to 8 instances
3. Why wasn't pool size updated?
   → No automated calculation, manual config
4. Why wasn't this caught in testing?
   → Load tests didn't use production instance count
5. Why don't we catch this automatically?
   → Pre-deployment validation doesn't check pool usage

**Root cause**: No automated pool sizing based on instance count

### 4. Contributing Factors (not root causes)
- Alert on error rate was too late (pool full = errors immediate)
- No pre-deployment pool validation
- Runbook didn't document pool sizing

### 5. Action Items (10 min)
| Action | Owner | Deadline | Priority |
|--------|-------|----------|----------|
| Auto-size pool based on instances | Backend lead | 2 weeks | P0 |
| Add pool usage alert (80%+ critical) | Oncall | 1 week | P0 |
| Add pre-deploy validation check | DevOps | 1 week | P1 |
| Document pool sizing in runbook | Tech writer | 3 days | P1 |

### 6. Prevention Ideas (10 min)
[Brainstorm how to prevent similar incidents]

### 7. Blameless Check (5 min)
- Did we focus on facts/systems, not people? ✓
- Did we assume good intent? ✓
- Are all participants comfortable with the report? ✓

After Postmortem

  • Distribute report to all team (transparency)
  • Track action items (check in 2 weeks)
  • Close the loop (what did we learn?)

RCA (Root Cause Analysis) Framework

Don't stop at the symptom:

❌ Root cause: Operator error
✓ Root cause: No automated validation → human forgot step

❌ Root cause: Connection pool too small
✓ Root cause: Pool size is manual config, no auto-sizing based on scale

❌ Root cause: Developer didn't check metrics
✓ Root cause: No alert on pool usage %, alert was on error rate too late

Blameless Culture Principles

  1. Assume good intent — Engineer did their best with information available
  2. Focus on systems — What failed in the process/tooling/documentation?
  3. No "stupid" mistakes — If someone made a mistake, how do we prevent it systematically?
  4. Psychological safety — Team must feel safe to admit mistakes and learn
  5. Shared responsibility — Incident is team's problem, not individual's
  6. Action items — Every postmortem must end with concrete improvements

Incident Communication

During incident (update every 5-10 minutes):

🟡 INCIDENT: API service degraded
Status: INVESTIGATING
Impact: Users may experience 50% error rate
Last update: 14:35 UTC
Next update: 14:40 UTC

https://status.example.com/incidents/[id]

When resolved:

✅ RESOLVED: API service
Status: RESOLVED (14:39 UTC)
Duration: 7 minutes
Impact: ~2000 users affected
Postmortem link: [link]

Thank you for your patience.

Metrics to Track

  • MTTR (Mean Time To Recovery): Target < 30 min SEV1
  • Detection time: How fast did we notice?
  • Resolution time: How fast did we fix?
  • Repeat incidents: Same root cause within 6 months?
  • Postmortem quality: All action items tracked?
  • Prevention success: Did fixes prevent recurrence?

Status: Ready for production incident response
Best for: Production incidents, postmortems, RCA, blameless culture

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.