Systematic debugging

Systematic Debugging

Overview

Follow a structured 4-phase methodology to diagnose and fix issues. Never guess. Never shotgun-debug. Understand the root cause before changing code.

---

Phase 1: Root Cause Investigation

Goal: Understand what is actually happening versus what should happen.

Step 1: Reproduce the Issue

• Run the failing test or trigger the error condition.
• Confirm the issue is reproducible, not intermittent.
• If intermittent, identify the conditions under which it occurs (timing, data, environment).
• Record the exact error message, stack trace, and output.

Step 2: Read the Error Message

The answer is often in the stack trace. Read it carefully, line by line.

• Identify the exact file, line number, and function where the error originates.
• Distinguish between the error origin and the error manifestation. The root cause is often several frames up the stack.
• Look for the first frame that belongs to the project code, not library code.

Step 3: Check Recent Changes

git log --oneline -10
git diff HEAD~3

• Identify what changed recently. Regressions are usually caused by recent commits.
• Compare the current state with the last known working state.
• Use git bisect for complex regressions to pinpoint the exact commit.

Step 4: Trace the Data Flow

• Follow the data from input to the point of failure.
• Log or inspect intermediate values at each transformation step.
• Identify where the actual value diverges from the expected value.

---

Phase 2: Pattern Analysis

Goal: Use existing working code as a reference to identify what went wrong.

Step 1: Find Working Examples

• Search the codebase for similar functionality that works correctly.
• Compare the working code with the broken code side by side.
• Note structural differences, missing steps, or different patterns.

Step 2: Compare Working vs. Broken

Ask these questions:

• What is present in the working version but absent in the broken version?
• What was changed or added that the working version does not have?
• Are the types, signatures, or interfaces different?
• Are the imports, configurations, or dependencies different?

Step 3: Check Environment

Rule out environmental causes:

• Are dependencies at the expected versions?
• Is the configuration correct for the environment?
• Are environment variables set correctly?
• Is the database/cache in the expected state?
• Does the issue reproduce on a clean checkout?

---

Phase 3: Hypothesis and Testing

Goal: Form a single, testable hypothesis and validate it.

Step 1: Form One Hypothesis

Based on the investigation, state a clear hypothesis:

"The bug occurs because [specific cause] which results in [observed behavior]."

Be specific. "Something is wrong with the data" is not a hypothesis. "The date parser receives a null string because the API response omits the field when the record has no creation date" is a hypothesis.

Step 2: Design a Minimal Test

Create the smallest possible test that validates or invalidates the hypothesis:

• If the hypothesis is correct, the test fails in a predictable way.
• If the hypothesis is wrong, the test passes (or fails differently), and a new hypothesis is needed.

Step 3: Validate

Run the test. Only two outcomes matter:

• Hypothesis confirmed: Proceed to Phase 4.
• Hypothesis rejected: Return to Phase 1 or 2 with the new information gained. Form a new hypothesis.

The One-Hypothesis Rule

Test one hypothesis at a time. Do not change multiple things simultaneously. Each change should test exactly one idea. This ensures clear cause-and-effect understanding.

---

Phase 4: Implementation

Goal: Fix the issue with a test-driven approach and verify no regressions.

Step 1: Write a Failing Test

Write a test that reproduces the bug. This test must:

• Fail in the current codebase.
• Fail for the same reason as the reported bug.
• Describe the expected correct behavior in its name and assertion.

it('handles missing creation date in API response', () => {
  const response = { id: 1, name: 'Test' }; // no createdAt field
  const result = parseRecord(response);
  expect(result.createdAt).toBeNull();
});

Step 2: Fix the Code

Apply the minimal fix to make the failing test pass. Do not fix additional issues or refactor while fixing the bug.

Step 3: Verify the Fix

• Run the bug reproduction test. It must pass.
• Run the full test suite. All tests must pass.
• Check for regressions in related functionality.

Step 4: Assess for Related Issues

• Consider whether the same bug pattern exists elsewhere in the codebase.
• Search for similar code that might have the same vulnerability.
• If found, create separate issues for those instances rather than fixing them in the same change.

---

Escalation Rule

If three or more fix attempts fail, stop and question the architecture.

This means the mental model of the system is incorrect. Step back and:

• Re-read the relevant code without assumptions.
• Trace the full execution path from entry point to error.
• Check whether the architectural assumptions are correct.
• Consider whether the component boundaries are in the right place.
• Ask whether the fix is being attempted at the right layer.

Continuing to apply fixes at the wrong level of abstraction wastes time and introduces new bugs.

---

Anti-Patterns

Fix and Pray

Making a change and hoping it works without understanding why. Every fix must be explainable: "This fixes the bug because [root cause] and the fix [addresses it by]."

Suppressing Errors

Catching and silencing exceptions, adding null checks without understanding why the value is null, or hiding symptoms instead of fixing causes. This creates silent failures that are harder to debug later.

Skipping Reproduction

Attempting to fix a bug that has not been reliably reproduced. Without reproduction, there is no way to verify the fix works.

Fixing Without Understanding

Applying a fix found on the internet or suggested by AI without understanding why it works. If the root cause is not understood, the fix might be masking the real problem or introducing new ones.

Shotgun Debugging

Changing multiple things at once to "see if something works." This prevents learning which change actually fixed the issue and often introduces new bugs.

---

Debugging Checklist

Before declaring a bug fixed:

• Root cause is identified and understood.
• A failing test reproduces the bug.
• The fix addresses the root cause, not a symptom.
• The reproduction test now passes.
• The full test suite passes with no regressions.
• Similar code has been checked for the same pattern.

---

Quick Reference

Phase	Goal	Output
1. Root Cause	Understand what is actually happening	Error trace, data flow analysis
2. Pattern Analysis	Compare with working code	Differences identified
3. Hypothesis & Testing	Form and test one hypothesis	Confirmed or rejected
4. Implementation	Fix with test-driven approach	Passing test, no regressions

---

References

• For the condition-based waiting pattern (replacing arbitrary delays with polling), read references/condition-based-waiting.md. See references/condition-based-waiting-example.ts for a TypeScript implementation.
• For defense-in-depth validation (multi-layer bug prevention), read references/defense-in-depth.md.
• For root cause tracing (backward trace through call chain), read references/root-cause-tracing.md.
• For finding which test creates unwanted files/state, use scripts/find-polluter.sh.

---

Dependencies

Skill	Reason
tdd-discipline	Phase 4 uses TDD (write failing test, implement fix, verify) to implement the bug fix

Used By

Component	Reason
Agent: debugger	Implements the 4-phase methodology defined here
Agent: code-reviewer	Dispatches the debugger agent when critical bugs are found