Incident management

Incident Management

Fail-Fast Rule

When an alert is real, declare and assign before you investigate. Never defer severity classification while you "look around for two minutes" — the two minutes become twenty, and nobody is in charge while it happens. Never silence a page without a written reason in the incident channel. Never end an incident on the basis of "metrics look better" alone; recovery is confirmed by user-facing verification AND a sustained observation window AND no fresh alerts in that window. If any of those three are missing, the incident is still open.

What This Skill Is (and isn't)

This is operational doctrine for handling production incidents — declaring, running, ending, and learning from them. It pairs with core/observability-and-ops (which tells you how to see the system) and core/chaos-and-resilience (which tells you how to design for failure). When an alert fires, this skill answers: who is in charge, what gets said, what gets done first, and how the post-incident loop closes.

This skill is NOT a runbook. Per-service runbooks live next to the service they cover — this skill governs the meta-process that all runbooks share. When in doubt about whether a given page is operational doctrine or a service-specific procedure, consult docs/MECHANISM-SELECTION.md §Quick flowchart: if the content is reusable across services, it is doctrine (this skill); if it is kubectl / psql / cloud-CLI invocations specific to one service, it is a per-service runbook stored next to that service.

Severity Schema

The framework uses a four-band schema. Bands are mutually exclusive — a single incident has exactly one severity at any moment, and changes to severity are logged events with a timestamp and reason.

Band	Trigger conditions	Initial response	Update cadence	Auto-escalation
SEV1	Total outage of a tier-1 service; suspected data loss; suspected breach; pay-flow halted	Page primary; declare within 5 min	Every 15 min, even with no change	Doubling impact → already SEV1
SEV2	Partial outage affecting >25% of a tier-1 surface; degraded SLO; one paying tenant down hard	Page primary; declare within 15 min	Every 30 min	Cross 50% impact → upgrade SEV1
SEV3	Single feature broken with workaround; non-paying tenant impact; fast-recoverable degradation	Acknowledge in 1 hour	Every 2 hours during waking hours	Workaround fails → upgrade SEV2
SEV4	Cosmetic / docs / known-issue with no user-visible impact; signal worth tracking but not paging	Next business day	At triage only	n/a — escalation means re-classify

Three rules govern severity:

1. Severity is set by impact, not by guesswork about cause. "We don't know yet" maps to "use the worst plausible band consistent with what you do see." A 5xx spike that might be a deploy regression is at least SEV2 until proven otherwise.
2. Customer-paying-account impact is a hard floor of SEV2. Non-paying accounts can sit at SEV3 by default.
3. Severity downgrades happen only after mitigation is confirmed and a verification window has elapsed. Never downgrade because "we think we know what's going on now."

Roles During an Active Incident

These four roles are non-overlapping. One person holds each at a time. Role transitions are announced in the incident channel and timestamped.

Incident Commander (IC)

Owns the incident. Decides what gets tried, what gets stopped, when to escalate, when to call it resolved. Does NOT type commands into a terminal during an active SEV1/SEV2 — that is the Technical Lead's job. The IC is the single decision-maker; everything else is advisory.

Communications Lead

Owns external + internal updates. Drafts the status-page entry. Posts cadence updates. Briefs leadership when the severity warrants. Knows the audience for each channel: customers want impact + ETA, leadership wants business consequence, engineers want technical state.

Technical Lead

Owns the keyboard. Runs diagnostic queries, executes mitigation steps, proposes hypotheses to the IC. Does NOT broadcast updates directly to users; routes them through the Communications Lead so cadence and tone stay consistent.

Scribe

Owns the timeline. Logs every command, every observation, every decision, with timestamps. The post-incident review depends on this. Without a scribe, the timeline is reconstructed from memory three days later, which is when the systemic causes get rationalized away.

For SEV3+ a single engineer may hold IC + Tech Lead + Scribe simultaneously provided the timeline is logged in real time. SEV1/SEV2 require role separation.

CORRECT vs WRONG — declaring an incident

# CORRECT
[14:02] alert: api-error-rate > 5% for 90s
[14:03] declare: SEV2, IC=alex, comms=jordan, tech=mei, scribe=sam
[14:03] impact: ~25% of /checkout requests failing in EU-west-1
[14:03] hypothesis: TBD; investigation starting; first update at 14:18
[14:04] @comms: post initial status-page entry now

# WRONG
[14:02] alert fires
[14:04] "let me see what's going on real quick"
[14:11] "wait, also the dashboard is loading slow"
[14:18] "should we maybe declare this?"

The wrong pattern wastes the first 16 minutes before the response structure exists. The correct pattern spends the first 90 seconds installing the structure, then investigates inside it.

Communication Cadence

Cadence is contractual, not aspirational. Stakeholders learn to trust the update interval and stop interrupting the response when they can predict the next message.

Internal (incident channel)

• SEV1: every 15 min, no exceptions, "no change" is a valid update
• SEV2: every 30 min
• SEV3: every 2 hours

A skipped update is itself an incident-process defect logged in the post-incident review.

External (status page + customer comms)

• SEV1: initial post within 10 min of declaration; cadence matches internal
• SEV2: initial post within 30 min
• SEV3: post only if customer-visible AND duration >1 hour

Leadership

• SEV1: brief at declaration + at every cadence beat
• SEV2: brief at declaration + at resolution
• SEV3/SEV4: include in weekly digest

Update template

# [SEV<N> UPDATE — <service>] <one-line status>
- State: <investigating | identified | mitigating | recovering | resolved>
- Impact: <what users see, scope, geography>
- Working theory: <best current hypothesis or "still narrowing">
- Action in flight: <what we're doing right now>
- Next update: <timestamp>

The five fields are mandatory. Filling "working theory" with "investigating" twice in a row is a signal that the hypothesis loop is stuck and the IC should consider widening the response team.

Mitigation Bias: Stop the Bleeding First

The IC's first question is not "what is the cause?" but "what is the fastest action that ends user impact?" That action is usually one of:

Mitigation	When it's right	Risk
Rollback	Onset within minutes of a deploy / config-push	None if last-known-good is verified clean
Feature-flag off	A specific feature is implicated and the flag exists	Blast radius depends on flag granularity
Failover / shed	A specific dependency is implicated and a fallback exists	Fallback path must be tested in advance
Scale up	Saturation symptoms (CPU, queue depth, connection pool) without underlying logic regression	Masks a real bug; pair with an action item
Restart	State corruption suspected; service is otherwise healthy in code	Restart cycles can cascade; cap retries

Investigation of root cause is a parallel track to mitigation, not a prerequisite. A SEV1 mitigated by rollback at minute 8 with root cause identified at minute 45 is a successful response. A SEV1 still in investigation at minute 30 with no mitigation attempted is a process failure regardless of how interesting the root cause turns out to be.

Hypothesis time-boxing

A diagnostic hypothesis gets a budget. Default is 15 minutes. When the budget elapses without confirmation, the hypothesis is set aside (logged, not erased) and the next one starts. The IC enforces this — engineers naturally over-invest in their first guess.

Ending an Incident

Recovery is a state, not a vibe. Three conditions, all required:

1. Symptoms gone in observed metrics. Error rates, latency p99, queue depth, whatever the page paged on, back inside SLO with a margin.
2. User-facing verification. Someone (Tech Lead, on-call, or the Communications Lead via a customer ping) confirms the broken surface actually works again. Not "the dashboard says 200 OK" — actually exercise the endpoint or the flow.
3. Sustained observation window. SEV1 = 30 min watching; SEV2 = 15 min; SEV3 = 5 min. No fresh alerts in that window. If anything pops, the window restarts.

When all three pass, the IC declares "resolved" in the channel with the final timeline and the link to the (yet-to-be-written) post-incident review.

Post-Incident Review (blame-free)

The review happens within 48 hours of resolution while context is still recoverable. Attendance is the response team plus anyone whose system was implicated. Format is a structured document, not a freeform retro.

Required sections

# Post-Incident Review: <service> — <YYYY-MM-DD>

- **Severity:** SEV<N>
- **Duration:** <start UTC> → <end UTC> (<minutes>)
- **Detection latency:** alert-to-declaration
- **Mitigation latency:** declaration-to-symptoms-clear
- **Customer impact:** <quantified — accounts affected, requests failed, $ if known>
- **SLO budget consumed:** <% of monthly>

## Timeline (UTC)
<table from scribe — every state change, decision, and observation>

## What broke (system, not people)
<the failure chain in technical terms; no individual names except in
"who was in IC role" sense>

## Why the safeguards we had didn't catch it earlier
<this is the point of the review>

## Action items
| ID | Action | Type    | Owner | Priority | Due date | State |
|----|--------|---------|-------|----------|----------|-------|
| 1  | …      | detect  | …     | P1       | …        | open  |
| 2  | …      | prevent | …     | P1       | …        | open  |
| 3  | …      | recover | …     | P2       | …        | open  |

## What we got right
<the response patterns to keep>

Action-item discipline

Action items are tagged by type — detect (we'd see it sooner), prevent (it can't happen again), recover (we'd recover faster) — and tracked to completion outside the review document. A repeat incident whose action item from the previous review is still open is itself a finding in the new review. This is how the review loop becomes load-bearing rather than ceremonial.

Blame-free framing

Blame-free means findings are framed in terms of system gaps. "Engineer X typo'd a config" is the wrong framing. "There is no integration test for config validation; the manual review process is the only safeguard, and manual review missed a one-character typo" is the right framing. The people involved are not absent from the document — they wrote the timeline and they own action items — but their decisions are not characterized as the cause. The system that allowed their decision to reach production is the cause.

Anti-Patterns to Reject

Anti-Pattern	Why it's wrong	Correct approach
Declaring severity after the response is over	Loses the trigger for cadence + escalation; updates never get written	Declare in the first 5 min; downgrade later if warranted
Single person doing IC + tech + comms during SEV1	Decision-maker is also typing; nobody is reading the metrics	Force role split at SEV1 even if it slows the first 5 min
"Let's not page leadership at 3am, it's not that bad"	Severity is set by impact, not by hour; under-paging trains leadership to ignore the channel	If the band says page, page; calibrate the band, not the page
Closing an incident on "metrics look better"	The mitigation may be cosmetic; users may still be broken	Three-condition close (metrics + user-facing + sustained window)
Skipping the post-incident review for "small" SEV3	Patterns are learned across small incidents; only writing reviews for SEV1 misses the systemic signal	Every SEV1/SEV2 + every repeat SEV3 gets a review
Action items without owners or due dates	The list grows, nothing closes	Owner + due date are mandatory fields; missing fields fail review
Same finding in three consecutive reviews	Action item discipline has broken; the system is teaching you nothing	Halt feature work for the affected component until the action item lands
Status-page silence during a confirmed customer-visible SEV1	Users learn from each other on social media first; trust collapses	Initial post within 10 min even if the only content is "investigating"
Treating an alert as the trigger to investigate the alert quality	Real alert is happening; tune the alert AFTER, in a follow-up	Action item: "tune <alert-name> threshold" — handled in the review, not during
Letting the tech lead push code mid-incident without review	Mid-incident hot-fixes are how the recovery becomes its own incident	Rollback first; new code goes through normal review even if expedited

On-Call Hygiene

On-call rotations are the input to this whole process — a degraded rotation produces degraded incidents. The framework's stance:

• Minimum rotation size of four engineers. Three rotates engineers through primary every third week, which corrodes sleep and judgment.
• Two-week cap on consecutive primary weeks. Hand off, rest, return.
• Shadow before primary. A new engineer rides along for at least one full rotation cycle before holding the pager alone.
• Pages-per-shift health metric. Above five pages per engineer per week is a signal that alerts are noisy or the system is degraded; either way it's a remediation target, not a steady state.
• Handoff during business hours. Rotation transitions at 10:00 local, not midnight. Engineers should hand context across to a refreshed successor, not wake one up to receive it.
• Pager compensation is non-optional. This is a stance, not a HR policy — when carrying the pager is unpaid, recruitment and retention for the senior end of the rotation deteriorate within two quarters.

These items are not enforced by a hook in this framework. They are review checklist items the Incident Commander archetype consults when reviewing on-call program proposals.

Cross-Validation with Other Disciplines

Release-gate intersection

A live SEV1 freezes new releases of the affected surface until the post-incident action items are at least scoped (not necessarily landed). This ties directly to the framework's release discipline (ADR-103 calendar-gate-final-purge — release.yml RC-hold mechanics). Mitigation deploys are exempt; feature deploys are not.

Security-and-auth intersection

A suspected breach at any band auto-elevates to SEV1. The Incident Commander coordinates with whichever archetype holds the security VETO floor for the framework (ADR-052 multi-model dispatch by role). The IC does not unilaterally decide a breach is over; the security archetype signs that off.

Chaos-and-resilience intersection

core/chaos-and-resilience lists known failure scenarios and their recovery posture. During investigation, the Tech Lead consults that catalog as the first hypothesis source — many incidents are re-runs of already-mapped scenarios. After resolution, the post-incident review updates the catalog with anything new.

Brainstorm-gate intersection

The post-incident review is a Pass-2 adversarial-framing exercise per ADR-058 (brainstorm-gate-and-two-pass-review). The reviewer's job is to disagree with the timeline's framing — to look for the systemic cause the responders rationalized around. A review that produces zero pushback on the responders' explanation is a review that hasn't done its job.

Operating-Mode Selector

Situation	Mechanism the framework expects
Active SEV1/SEV2 happening now	This skill loaded into the responding archetype's context
Authoring a per-service runbook	This skill loaded into a Senior Engineer archetype as the meta-doctrine reference; the runbook lives next to the service
On-call program review	This skill + core/observability-and-ops loaded into the operations archetype
Post-incident review facilitation	This skill + code-review-checklist loaded into the facilitator; ADR-058 Pass-2 framing applied
Incident-readiness audit / game day	This skill + core/chaos-and-resilience loaded into the auditor; outputs feed the chaos-scenario catalog

The matrix above is the basis for the Operating-Mode rationale recorded in mechanism-selection-rationale.md alongside this skill — the work is persistent doctrine plus role-bearing pattern reference, which the mechanism-selection guide maps to a SKILL with a thin task-chain companion, not a slash-command and not a runbook.

Success Signals

A working incident-management practice produces:

• Detection-to-declaration latency under 5 min for tier-1 services
• Sustained reduction in median time-to-mitigate quarter over quarter
• Zero repeat incidents whose previous review's action items shipped
• Pages-per-engineer-per-week trending down or stable, not up
• Post-incident review action-item completion >85% within stated due date
• A culture where engineers escalate freely and re-classify upward without asking permission

The metrics are lagging indicators. The leading indicator is whether the roles, cadence, and three-condition close are actually used during real incidents — that is what the SKILL.md exists to govern.

References

• core/observability-and-ops — health-check + metrics design, the input to detection
• core/chaos-and-resilience — failure-scenario catalog, the input to hypothesis generation
• core/security-and-auth — auth + breach handling intersection
• core/code-review-checklist — adversarial-framing methodology applied to the post-incident review
• ADR-052 — multi-model dispatch by role (VETO-floor invariants this doctrine plugs into)
• ADR-058 — brainstorm gate + two-pass review (Pass-2 framing for post-incident review)
• ADR-103 — calendar-gate-final-purge (release.yml RC-hold mechanics intersected by SEV1 freezes)
• docs/MECHANISM-SELECTION.md — the flowchart this skill resolves to a skill (not a runbook, not a slash-command)