Systematic debugging

Systematic Debugging

Random fixes waste time and breed new bugs; symptom patches mask the cause until it resurfaces somewhere worse. The protocol: prove the cause, then fix it once.

This is a rigid skill. The bright line: no fix is proposed before the root cause is identified with evidence. It holds hardest under time pressure — guess-and-check thrashes for hours where the four phases take minutes — and it applies to simple-looking bugs too, which have root causes like any other.

Phase 1 — investigate

1. Read the error completely. The full stack trace, line numbers, error codes. The exact answer is often already in it.
2. Reproduce reliably — observe before you theorize. Exact steps, every time; not reproducible yet means gather more data, not guess. For a behavior or regression question, run the failing path and read its real output (logs, stack, emitted query) before forming hypotheses from source — dynamic observation precedes static theory. (Exception: when running the path is destructive, irreversible, or not yet buildable, read and instrument first — don't trigger a payment or drop a table to "observe" it.) And confirm the code that ran is the code you read — resolve the imported module's __file__ and installed version; an editable checkout and an installed release of the same library diverge silently. (A data-engineering skill, when one is installed, states this canonically as an axiom — e.g. data-engineering-discipline Axiom 2.)
3. Check recent changes. Diff, recent commits, new dependencies, config and environment differences.
4. Instrument boundaries in multi-component systems. Before theorizing about a chain (CI → build → signing; API → service → database), log what enters and exits each component and whether config propagates; run once; read off which layer breaks. Then investigate that layer.
5. Trace bad values to their origin. Where does the value come from, what called this with it, all the way up — see root-cause-tracing.md. Fix at the source, not where the error surfaced.

Phase 2 — pattern-match

Find working code of the same shape in the codebase and compare. Read reference implementations completely rather than skimming — partial understanding of a pattern produces partial copies of it. List every difference between working and broken, including the ones that look like they can't matter, and pin down the dependencies and assumptions the working version relies on.

Phase 3 — hypothesize and test

State one hypothesis precisely: "X is the root cause because Y." Test it with the smallest change that could confirm it — one variable at a time. If it fails, form a new hypothesis; stacking a second fix on top of an unconfirmed first one destroys your ability to attribute anything. When something is not understood, say so and investigate it — pretending to understand blocks the protocol exactly where it's needed.

Phase 4 — fix

1. Failing test that reproduces the bug — test-driven-development owns the cycle from here.
2. One fix, addressing the proven cause. No bundled refactoring, no "while I'm here": an edit's diff is its scope. (This is the scope-bounding rule a data-engineering skill states canonically when one is installed — e.g. data-engineering-discipline Principle 17.)
3. Verify: the new test passes, nothing else broke, and the original symptom is actually gone. Hand the claim to verification-before-completion before reporting it fixed.
4. Three failed fixes mean the architecture is in question, not the next patch. The telltale pattern: each fix reveals new coupling or shared state somewhere else, or needs a rewrite to land. Stop and raise the pattern with the user before a fourth attempt — that conversation is about the design, not the symptom.

Common shortcuts and what they miss

Shortcut	What it misses
"Quick fix now, investigate later"	The patch masks the cause; later never comes.
"Just try changing X and see"	Untraced changes can't be attributed; thrashing begins.
"It's probably X"	Recognizing a symptom is not understanding a cause.
"Several fixes at once saves time"	Nothing can be isolated; regressions ride along.
"Emergency — no time for process"	The protocol is the fast path; thrashing is the slow one.
"One more attempt" (after two failures)	The third failure is an architecture signal, not bad luck.
"I read the code, so I know what it does"	The code you read may not be the code that runs — resolve module.__file__ + version.
"I'm pretty sure it's X" (no run yet)	A causal claim with zero confirming observation is a guess; produce the observation first.

When investigation finds no root cause

Genuinely environmental, timing-dependent, or external issues exist. Having completed the phases: document what was ruled out, implement appropriate handling (retry, timeout, clear error message), and add the monitoring that would catch the next occurrence. Most "no root cause" conclusions, though, are incomplete investigation — check that each phase actually ran before settling for one.

Quick reference

Phase	Activity	Done when
1 Investigate	Read, reproduce, recent changes, instrument	The failing layer is known, with evidence
2 Pattern	Compare against working examples	The differences are listed
3 Hypothesis	One precise theory, smallest test	Confirmed, or replaced
4 Fix	Failing test, single fix, verify	Symptom gone, tests green

Supporting references

• root-cause-tracing.md — backward tracing to the original trigger
• defense-in-depth.md — validation at every layer once the cause is fixed
• condition-based-waiting.md — replacing arbitrary timeouts with condition polling