Debug detective

Debug Detective

Overview

Debugging fails when it becomes random poking — changing lines, re-running, hoping. This skill enforces a scientific, evidence-first process: you never change code to "see what happens" until you have a falsifiable hypothesis backed by an observation. The goal is to find the root cause, not to make the symptom disappear.

Keywords: debug, root cause, bug, regression, crash, stack trace, flaky test, intermittent, heisenbug, deadlock, memory leak, race condition, bisect, repro, reproduce, isolate, hypothesis, works on my machine.

Core principle: A bug you cannot reproduce is a bug you cannot confirm fixed. Spend your effort earning a reliable reproduction first; everything else gets faster after that.

Workflow

Follow these phases in order. Do not skip ahead — a fix found by skipping is usually a coincidence, not a cure.

1. Capture & Reproduce

• Collect the exact symptom: error message, stack trace, wrong vs. expected output, exit code, screenshot, logs.
• Record the environment: OS, runtime/language version, dependency versions, config, env vars, data, concurrency, time/locale.
• Build a minimal reliable reproduction — the smallest command/input that triggers the bug. Note how often it fails (100%? 1 in 20?).
• If it is intermittent, quantify the rate and try to raise it (loop it, add load, shrink timeouts) before proceeding. See references/intermittent-bugs.md.
• Gate: Do not continue until you can trigger the bug on demand (or have a measured failure rate). Write the repro down.

2. Isolate

• Shrink the surface area. Remove inputs, disable features, stub dependencies until the bug still reproduces with the least machinery. (Delta-debugging mindset.)
• Establish the boundary: which layer/module/function is the last place behavior is correct, and the first place it is wrong?
• Add observation points (logging, breakpoints, assertions, print of invariants) — observe, don't fix.

3. Hypothesize

• State a single, falsifiable hypothesis: "The crash happens because users is None when the cache misses." A good hypothesis predicts an observation you haven't made yet.
• Rank candidate hypotheses by likelihood × cheapness-to-test. Test the cheapest discriminating one first.
• For each, write the prediction before you run the experiment. See references/hypothesis-log.md for the format and use templates/investigation-log.md to track them.

4. Experiment / Bisect

• Run the smallest experiment that confirms or refutes the hypothesis. One variable per experiment.
• If a regression ("worked before"), use git bisect to find the offending commit. Run scripts/git_bisect_helper.sh to automate it with a test command.
• Update the log with the actual result. Refuted hypotheses are progress — they shrink the search space.

5. Diagnose Root Cause

• Confirm you can explain the full causal chain from root cause to symptom. If there's a gap ("and then somehow..."), you're not done.
• Distinguish the root cause from contributing factors and the trigger. Ask "why" until you reach something actionable (see 5-Whys in references/root-cause-frameworks.md).

6. Fix & Verify

• Fix the root cause, not the symptom. Prefer the narrowest correct change.
• Verify: run the original repro — it must now pass. Then run it the way that previously failed (loop the flaky case, original environment, original data).
• Regression-guard: add a test that fails without the fix and passes with it. This is non-negotiable for a real bug.
• Check for siblings: the same root cause often manifests elsewhere. Search the codebase for the same pattern.

7. Document

• Record root cause, fix, and how it was verified. Note any latent issues found. Use templates/investigation-log.md.

Decision Framework: which technique by symptom

Symptom	First move	Key reference
Consistent crash / exception	Read the full stack trace bottom-up; find the deepest frame in your code	references/techniques.md
"Worked before this change"	git bisect	scripts/git_bisect_helper.sh
Flaky / 1-in-N failure	Raise failure rate, look for shared state / ordering / time / concurrency	references/intermittent-bugs.md
Wrong output, no error	Binary-search the data pipeline; assert invariants at each stage	references/techniques.md
Hang / deadlock	Get a thread dump / py-spy dump; find who holds what lock	references/techniques.md
Memory growth	Snapshot heap over time; diff allocations; check unbounded caches/listeners	references/techniques.md
Works on my machine	Diff the two environments systematically (versions, config, data, locale, TZ)	references/techniques.md
Performance regression	Profile, don't guess; compare flamegraphs before/after	references/techniques.md

Best Practices

• Change one thing at a time. Multiple simultaneous changes destroy your ability to attribute cause.
• Write the prediction before the experiment. Forces honesty and prevents post-hoc rationalization.
• Trust observations over intuition. When the data contradicts your mental model, the model is wrong.
• Read the actual error. Most "mysterious" bugs are explained in a log line nobody read fully.
• Keep the repro fast. Tighten the loop; a 2-second repro lets you run 100 experiments, a 5-minute one lets you run 6.
• Version-control your investigation. Commit experimental changes on a scratch branch so you can revert cleanly.
• Binary search everything — commits, inputs, config, time ranges. It turns linear searches into logarithmic ones.

Common Pitfalls

• Fixing the symptom: wrapping a null in a guard without asking why it was null. The real bug moves downstream.
• Confirmation bias: running only experiments that support your favorite theory. Design experiments that could refute it.
• Premature fixing: editing code before you can reproduce. You'll never know if it worked.
• Heisenbug denial: adding logging makes it disappear → that's a clue (timing/race), not bad luck.
• Skipping the regression test: the bug will return and nobody will know it regressed.
• Tunnel vision on recent changes: the trigger may be new code, but the root cause can be old latent code newly exercised.
• Ignoring "it can't be that": the bug is in the place you're sure is correct. That's why you haven't found it.

Bundled Resources

• references/techniques.md — detailed per-symptom techniques, tooling, and language-specific tips.
• references/intermittent-bugs.md — playbook for flaky tests, races, and heisenbugs.
• references/root-cause-frameworks.md — 5 Whys, fault trees, delta debugging, and the scientific method applied to code.
• references/hypothesis-log.md — how to write and rank falsifiable hypotheses.
• scripts/git_bisect_helper.sh — automated git bisect run wrapper for regression hunting.
• templates/investigation-log.md — fill-in template to track a debugging session end to end.
• examples/null-pointer-investigation.md — a full worked example from report to verified fix.