Incident response and postmortems

Incident Response and Postmortems

Overview

Incidents are guaranteed; chaos is optional. A good incident process is unglamorous: mitigate first, communicate constantly, learn afterward, blame never. The goal during the incident is restoring service — root cause can wait for the postmortem.

Mitigation beats diagnosis. Rolling back a bad deploy in 5 minutes beats finding the perfect root cause in 3 hours. Restore service with the fastest safe action; understand it later.

Blameless is not optional. If people fear punishment, they hide information, and the next incident takes longer. Postmortems analyze systems, not character.

When to Use

• A production alert fires with user impact
• Users report degradation or outage
• Security incident suspected (see also secrets-management-and-identity for credential leaks)
• After any incident, for the postmortem
• Running incident drills / game days

Severity Levels (declare one immediately)

Level	Meaning	Response
SEV1	Critical user impact / full outage / data breach	All hands, incident commander, comms every 15–30 min
SEV2	Significant degradation or single major feature down	On-call + relevant owner, comms hourly
SEV3	Minor impact, workaround exists	Ticket, business hours

Declaring severity early sets the response shape. Undeclared incidents drift; over-declared ones exhaust people. Re-classify as you learn.

The Response Workflow

1. DECLARE          "SEV2: checkout error rate 15%, user impact confirmed"
                    One person is Incident Commander (IC). The IC coordinates;
                    the IC does not debug.
2. MITIGATE         Fastest safe restoration: rollback, failover, feature flag off,
                    scale up, block the bad actor. Diagnosis is for later.
3. COMMUNICATE      Status page + internal channel. Every N minutes even if
                    "no update": silence reads as chaos.
4. DIAGNOSE         In parallel or after mitigation. Capture evidence FIRST
                    (logs, dashboards, current state) before restarting things.
5. RESOLVE          Verify recovery with the same signals that detected it.
6. POSTMORTEM       Within days, blameless, with action items that have
                    owners and dates.

Evidence before reboot. The classic error: restart the service, then try to figure out why it failed — with all the evidence gone. Snapshot logs, top output, connection counts, error rates first (see linux-system-administration).

The Rollback Decision

Rollback when ALL of these hold:

1. A recent change correlates with the incident start (deploy timeline vs error spike — check the dashboards)
2. The rollback is known-safe (rollback plan exists — see gitops-and-progressive-delivery)
3. Data compatibility confirmed (no incompatible schema change — this is why expand/contract matters)

If any fails, mitigate differently: feature flags, config changes, scaling, traffic shifting.

Roles

• Incident Commander (IC): owns coordination, decisions, and the timeline. Delegates everything else. Single point of "what are we doing right now."
• Tech lead(s): investigate and execute fixes, assigned by the IC.
• Communications lead (SEV1/SEV2): status page, stakeholders, customer support alignment.
• Scribe: timeline of events/actions — the postmortem's raw material. Chat logs help, but deliberate notes are better.

Small team? One person can hold multiple roles, but name them explicitly — role confusion is how incidents take 3x longer.

Communication Templates

Internal (every update interval):

[SEV2] Checkout errors — Update 3 (14:30 UTC)
Status: Mitigating. Rolled back release 1.4.2 at 14:15; error rate falling (now 2%).
Impact: ~15% of checkouts failed between 13:55–14:15 UTC.
Next update: 14:45 UTC or on resolution.
IC: @ana

External (status page):

Identified — We are experiencing elevated errors on checkout. A fix has been
deployed and we are monitoring recovery. Next update in 30 minutes.

Rules: say what you know, say what you don't, commit to the next update time, and keep the commitment. Never speculate externally about cause before the postmortem.

The Blameless Postmortem

Held within 2–5 days of SEV1/SEV2 (memory fades), 30–60 minutes, with the timeline from the scribe.

# Postmortem: Checkout outage 2026-07-25

## Summary
One paragraph: what happened, impact (users/duration/revenue), root cause class.

## Impact
- 15% of checkout attempts failed for 20 minutes (13:55–14:15 UTC)
- ~340 failed orders; support tickets: 28

## Timeline (UTC)
- 13:50 — Release 1.4.2 deployed (PR #412, approved by X)
- 13:55 — Error rate alert fires (CheckoutErrorBudgetBurn)
- 13:58 — SEV2 declared, IC assigned
- 14:15 — Rollback to 1.4.1 complete via Argo CD sync of revert commit
- 14:30 — Error rate back to baseline; monitoring
- 15:00 — Resolved

## Root Cause
The 1.4.2 release enabled a new DB connection pool with `max_connections`
higher than the RDS limit; under load, new connections were refused.
WHY didn't defenses catch it: staging load is 1/50th of prod; no load test
covers connection saturation; the alert that fired was built for 5xx, which
caught it — but the deploy verification window was only 5 minutes.

## What Went Well
- Detection: 5 minutes from deploy to page (alert worked)
- Mitigation: rollback executed in 17 minutes, runbook followed
- Communication: updates every 15 min, support pre-briefed

## What Went Poorly
- No load test covers DB connection limits
- Post-deploy verification window too short for a saturation failure
- RDS connection limit not documented anywhere

## Action Items (owner — due date)
1. [P1] Add connection-pool saturation load test to release pipeline — @chen — Aug 8
2. [P1] Extend post-deploy verification window to 20 min with auto-halt — @ana — Aug 1
3. [P2] Document RDS limits in service README + alert at 80% pool usage — @raj — Aug 15
4. [P2] Add "verify recent deploys" to the on-call triage runbook — @ana — Aug 1

Postmortem rules:

1. Name the timeline, not the culprit. "The deploy at 13:50" not "Dave's deploy."
2. "Why" applied to defenses: the question is never "who messed up" but "why did our testing/review/gates/verification not catch this."
3. Action items have owners, dates, and priorities — and they're tracked like any other work. A postmortem whose action items die in a wiki page is a ceremony, not a process.
4. Celebrate what worked. Detection speed and clean rollbacks are worth naming — they're what you want repeated.
5. Share postmortems widely. They're the organization's cheapest reliability education.

Incident Response Readiness (the pre-incident work)

• On-call rotation defined, with escalation policy and tested paging path
• Runbooks for the top failure modes, linked from the alerts themselves
• Rollback tested recently (a rollback that's never been executed is a hypothesis)
• Status page and comms templates ready
• Access to dashboards/logs/credentials works for the on-call at 3 AM (test it)
• Game days / failure drills run periodically in staging
• Post-incident review of the process (not just the tech): were roles clear, was comms timely?

Common Rationalizations

Rationalization	Reality
"Let's find the root cause before we roll back"	Users are impacted now. Mitigate with the fastest safe action; root cause survives for the postmortem, the SLA doesn't.
"It's probably not worth declaring a SEV"	Undeclared incidents have no IC, no comms rhythm, and no end. Declare; downgrade later if wrong — that's cheap.
"Postmortem can wait until things calm down"	Things never calm down; memories fade in days. Postmortem within the week or the learning is lost.
"We know who caused it, let's be honest about that"	The moment blame enters, information leaves. People who fear blame stop declaring incidents early — the worst possible outcome.
"Action items from last time are still open, but this time we'll really do them"	Untracked action items are how the same incident repeats quarterly. Postmortem items go into the same backlog as feature work, with priority.
"Restarting fixed it — incident closed"	You destroyed the evidence and guaranteed a repeat. Capture state first, restart second, and still do the postmortem.

Red Flags

• Incidents with no declared severity, no IC, no timeline
• The same person always "just fixes it" silently (hero culture)
• Rollback that's never been tested, discovered broken during a SEV1
• Postmortems that assign blame or produce no action items
• Action items perpetually at the bottom of the backlog
• Status page updated only after customers complain on social media
• On-call who can't access what they need at 3 AM
• The same incident signature appearing every few months

Verification

• Every SEV1/SEV2 has: declared severity, named IC, comms log, timeline, postmortem
• Mitigation time (detect → mitigate) tracked and trending down
• Postmortems blameless: timeline-based, defense-focused, with owned/dated action items
• Action item completion rate measured (target: P1s done by their dates)
• Rollback executed successfully in the last quarter (drill or real)
• Runbooks exist and were used during the last incident (updated afterward)
• Repeated incident signatures are zero — the same root cause never pages twice