Postmortem debugging

Postmortem debugging

By the time you are called, the fire is out. The bad node was recycled, the process that hung is gone, and rerunning proves nothing because the conditions have passed. All that survives is what the system happened to record. The job is to reconstruct a causal story from cold artifacts, knowing you cannot ask the system a single new question.

Method

1. Build the timeline before forming any theory. Pull logs, metrics, deploy events, and alerts into one time-ordered view, all in a single timezone, usually UTC. Fix the moment of first impact from the earliest symptom, not the first alert, which often fires minutes late. The sequence constrains cause: nothing after the first symptom caused it.
2. Separate the trigger from the cause. The deploy at 14:02 may be the trigger, but the latent bug it exposed is the cause. Ask what changed just before first impact, config, traffic, a dependency, and what condition made that change fatal when the same change was harmless yesterday.
3. Mine the artifacts for the state you cannot query live. A heap or core dump captured during the incident still yields a backtrace and variable values under gdb or the runtime's dump tool. Metrics show the shape: memory climbing to a ceiling, a thread pool pinned at max, error rate stepping up at a specific minute. Read them as the frozen state they are.
4. Correlate across signals to place the failure. Line up the latency spike with the GC log, the error burst with the deploy marker, the saturation with the traffic curve. A cause leaves a coincident signature in more than one place; a single suspicious line that nothing else corroborates is usually a symptom, not the origin.
5. Distinguish absence of evidence from evidence of absence. A missing log can mean the event did not happen or that the process died before flushing, or that sampling dropped it. Note which signals you would expect and did not get, and treat a silent gap right before the crash as itself a clue rather than a dead end.
6. Write the reproduction hypothesis the artifacts support, then test it offline. State the specific sequence you believe occurred and the one condition that made it fatal. Try to reproduce it in a controlled environment; a story that cannot be reproduced from the stated preconditions is not yet a root cause, only a plausible narrative.

Checks

• Does the timeline place first impact before every candidate cause?
• Does the root cause show a coincident signature in at least two signals?
• Could someone reproduce the failure from the preconditions you wrote down?

Boundaries

Postmortem work is bounded by what was recorded: if the logging level was too coarse or the dump was never captured, the answer may be permanently out of reach, and the honest output is a ranked set of hypotheses plus the instrumentation to add before next time. Live faults you can still touch belong to interactive debugging, which can ask new questions this one cannot.