Postmortem

Postmortem

Structured incident and bug analysis. Blameless, evidence-based, actionable. Covers production incidents, failed deployments, data loss, bug investigations, and any "what went wrong" scenario.

Important

• Blameless. Focus on systems, processes, and conditions — not individuals. "The deploy script had no rollback" beats "the engineer forgot to test."
• Evidence-based. Every claim in the timeline must cite a source: a log line, commit hash, error message, or metric. Unsupported assertions are not analysis.
• Actionable. Prevention items must be specific enough to become tickets. "Be more careful" is not a prevention. "Add a pre-deploy check that validates the migration is reversible" is.
• Proportional. Match depth to severity. A small bug gets a 5-minute writeup. A production outage gets a deep dive. Do not over-analyze low-impact incidents.

Instructions

Step 1: Parse Incident Description

Read $ARGUMENTS. This is the incident description — what the user says went wrong.

If $ARGUMENTS is empty or vague, ask one clarifying question: "What broke, and when did you notice it?" Do not proceed with guesswork.

Identify incident type from the description:

• outage — service unavailable, significant user impact
• degradation — service slow or partially broken
• data-loss — data deleted, corrupted, or inaccessible
• deploy-failure — deployment rolled back or failed mid-deploy
• bug — incorrect behavior in production or staging
• security — unauthorized access, data exposure, credential compromise

Incident type drives the depth of the postmortem. outage and security always get full analysis. bug may get abbreviated treatment if low-severity.

Step 2: Gather Evidence

Before analyzing, gather all available evidence. Do not theorize without data.

Check these sources (as applicable to the incident):

• Logs — error logs, application logs, system logs. Look for first occurrence of the error.
• Git history — recent commits and merges. git log --oneline -20 covers most bugs. Check git diff for the change that introduced the issue.
• Error messages — exact stack traces, exception types, HTTP status codes.
• Metrics/monitoring — when did error rate spike? When did latency increase? When did a service go down?
• Deployment records — what was deployed, when, by what process?
• Configuration changes — env vars, feature flags, infrastructure changes.
• External dependencies — third-party API status pages, library changelogs.

For each piece of evidence, note the source explicitly (e.g., "app.log line 1842", "commit a3f9d2e", "Sentry event #12345").

If evidence is unavailable, note the gap. Missing evidence is itself a contributing factor — it means monitoring or logging is insufficient.

Step 3: Build the Timeline

Construct a chronological sequence of events from first cause to resolution.

Timeline format:

[TIME or RELATIVE]  [EVENT]  [SOURCE]

Example:

14:32  Deploy triggered for v2.4.1 — CI pipeline
14:34  Migration ran: dropped column `user.legacy_id` — deploy log
14:35  Error rate spiked to 40% — Datadog alert
14:36  On-call paged — PagerDuty
14:41  Root cause identified: API still reading `legacy_id` — Sentry stack trace
14:44  Rollback initiated — deploy log
14:47  Error rate returned to baseline — Datadog

If exact timestamps are unavailable, use relative times (T+0, T+5min, etc.) or sequence markers (Step 1, Step 2, etc.).

Every event should have a source citation. Events without citations are hypotheses, not facts — mark them as [inferred].

Step 4: Root Cause Analysis

Use the 5 Whys method. Start from the observable failure and ask "why" until you reach a systemic root cause.

Distinguish:

• Trigger — the immediate event that caused the failure (e.g., "a deploy was pushed")
• Root cause — the systemic condition that made the failure possible (e.g., "there was no pre-deploy check that the API was backward-compatible with the migration")

The trigger is usually obvious. The root cause requires 3-5 "why" iterations.

Example:

Why did the service fail?       → API threw 500 errors
Why did the API throw errors?   → It referenced a column that no longer exists
Why did the column not exist?   → The migration dropped it
Why was an incompatible migration deployed?  → Migration and API were deployed in the wrong order
Why were they deployed in the wrong order?   → The deploy process has no dependency ordering — ROOT CAUSE

State the root cause in one sentence at the end. It should be a systemic failure, not a human error.

Step 5: Contributing Factors

List conditions that made the incident possible or worse. These are not the root cause but they amplified the impact or lowered the barrier to failure.

Common contributing factors:

• Missing tests (unit, integration, or regression)
• Missing monitoring or alerting (silent failures)
• No rollback plan or procedure
• Unclear ownership (nobody knew they were responsible)
• Technical debt (fragile code that couldn't tolerate the change)
• Process gaps (no review step, no staging environment)
• Documentation rot (runbook was wrong or missing)

Each contributing factor should be a complete sentence explaining the condition, not just a label.

Step 6: Impact Assessment

Quantify what was affected. Be specific.

• Duration — when did the incident start? When was it resolved? Total time impacted.
• Scope — which systems, services, users, or data were affected?
• User impact — errors seen, requests failed, features unavailable. Percentage or absolute numbers if available.
• Data impact — any data lost, corrupted, or exposed? If so, what and how much?
• Downstream effects — did dependent systems fail? Were customers charged incorrectly? Did alerts fire?

If impact is unknown, say so explicitly. Unknown impact is worse than known bad impact — it means observability is broken.

Step 7: Prevention

For each root cause and contributing factor, write a concrete prevention item.

Prevention items must be:

• Specific — name the exact file, system, or step to change
• Actionable — someone could pick this up as a ticket today
• Proportional — a 30-minute outage doesn't need six months of infrastructure work

Format:

[PRIORITY]  [ACTION]  [RATIONALE]

Priorities: P0 (do before next deploy), P1 (do this sprint), P2 (backlog, important), P3 (nice to have)

Example:

P0  Add pre-deploy validation that checks API compatibility with pending migrations  Prevents deploy-order failures
P1  Add integration test covering migration + API together  Would have caught this before deploy
P2  Add column-existence check to startup health endpoint  Surfaces this class of failure in seconds, not minutes
P3  Document deploy order dependency in runbook  Reduces future debugging time

Do NOT write preventions that amount to "be more careful," "add more documentation," or "review more thoroughly" without naming the specific artifact to create.

Step 8: Deliver Report

Print the postmortem report inline to the user as structured markdown. NEVER save it to a file inside the project being investigated — the incident repo must stay clean. If the user explicitly asks for a persisted copy, save it under ~/.claude/postmortems/<project>-<YYYYMMDD-HHMMSS>.md, not in the project. Adapt length to severity.

Full report (outage, data-loss, security):

# Postmortem: [Brief Incident Title]
**Date:** YYYY-MM-DD
**Severity:** outage | degradation | data-loss | deploy-failure | bug | security
**Status:** resolved | ongoing | mitigated

## Summary
[One paragraph. What happened, when, and impact. Written for someone unfamiliar with the system.]

## Timeline
| Time | Event | Source |
|------|-------|--------|
| ...  | ...   | ...    |

## Root Cause
[One sentence root cause]

### 5 Whys
1. Why did X? → Because Y
2. Why Y? → Because Z
...
Root cause: [systemic condition]

## Contributing Factors
- [Factor 1]
- [Factor 2]

## Impact
- **Duration:** X minutes / hours
- **Scope:** [what was affected]
- **User impact:** [quantified if possible]
- **Data impact:** [none | quantified]

## Prevention
| Priority | Action | Rationale |
|----------|--------|-----------|
| P0 | ... | ... |
| P1 | ... | ... |

Abbreviated report (bug, degradation, low-severity):

# Postmortem: [Brief Incident Title]
**Date:** YYYY-MM-DD  **Severity:** bug

## What Happened
[2-3 sentences: trigger, root cause, resolution]

## Timeline
[Key events only, prose format acceptable]

## Root Cause
[One sentence]

## Prevention
- [Specific action]
- [Specific action]

Error Handling

1. No evidence available — Proceed with what the user described. Mark all timeline entries as [inferred]. Note that the first prevention item should be "add logging/monitoring so this is diagnosable next time."
2. Incident still ongoing — Set status to ongoing. Skip impact duration and root cause if not yet determined. Focus on timeline and immediate mitigation actions.
3. Ambiguous severity — Default to the more severe classification. Over-preparing a postmortem wastes less time than under-preparing one.
4. User provides git repo or log files — Read them. Don't ask the user to summarize — that defeats the purpose.

Examples

Example 1: Deploy failure

Input: /postmortem deployed a migration that dropped a column the API still used — had to roll back, 12 minutes of errors

Output: Full report. Timeline from deploy trigger to rollback. Root cause: no pre-deploy API/migration compatibility check. Contributing factors: missing integration test, no staging environment parity. Impact: 12 minutes, ~40% error rate. Prevention: P0 add compatibility check, P1 add integration test.

Example 2: Bug investigation

Input: /postmortem users are seeing stale prices — cache not invalidating after updates

Output: Abbreviated report. Root cause traced via git log to a cache key change that broke invalidation. Prevention: add a test that updates a price and asserts the cache returns the new value within 5 seconds.

Example 3: Data loss

Input: /postmortem ran a migration without a backup, dropped the wrong table

Output: Full report. Status: resolved (or ongoing if data not recovered). Impact section emphasizes data loss quantification. Prevention leads with P0: add pre-migration backup step to deploy runbook, enforce with a migration script wrapper that refuses to run without a recent backup.