Runbook writing

Runbook writing

A runbook is written for the worst version of your reader: half-awake, paged, under pressure, possibly not the person who wrote it. It succeeds when they can execute it without understanding the whole system, and it is honest about where judgment is still required.

Method

1. Open with trigger, impact, and severity. When this runbook applies (the alert or symptom that led here), what is affected, and how urgent: so a responder confirms they are in the right place in ten seconds (see alerting-design: the alert links here). A runbook you have to read fully to know if it applies is a document, not a runbook.
2. Write steps as copy-pasteable actions. Exact commands (not "restart the service" but the command, with placeholders clearly marked), expected output, and what each does; link to dashboards and consoles directly. Prose that describes what to do forces translation the stressed reader gets wrong; executable steps do not.
3. Verify after every consequential step. "Run X; confirm Y shows Z before continuing": so a wrong turn is caught immediately, not three steps later (see script-idempotency's check-then-act ethic, human edition). Steps without verification let small errors compound into a second incident.
4. Make decision points explicit branches. "If the queue depth is still rising, go to section B; if it is draining, monitor and stop": name the observable that decides, not "use judgment". Where judgment truly is required, say so and give the factors (see incident-commander-role for escalation decisions): false certainty is worse than honest ambiguity.
5. Flag the dangerous and the irreversible. Destructive steps (data deletion, traffic cutover) get a warning, the blast radius, and a confirmation gate (see automation-guardrails); include the rollback for each risky action (see rollback-strategy). The runbook that walks someone off a cliff at step 7 is worse than none.
6. Keep runbooks alive. Test them (in game-days, and the first time on the real incident, fix what was wrong: see chaos-gameday); date them and assign an owner; every incident postmortem checks whether the runbook helped and updates it (see incident-postmortem). A stale runbook confidently gives wrong instructions, which is the most dangerous failure mode of all.

Boundaries

• Runbooks encode known, repeatable responses; novel incidents need the runbook's diagnostic starting points plus human reasoning (see production-debugging, scientific-debugging): do not pretend a runbook covers the unforeseen.
• A frequently-run runbook is a candidate for automation (auto-remediation): but automate only what is safe and well-understood, keeping the manual runbook as the fallback (see automation-guardrails).
• Runbooks live where they are reachable during an incident (not only in a wiki that might be down); linked from alerts and mirrored somewhere resilient (see cloud-disaster-recovery's reachability rule).