Issue debugging

Issue Debugging

Systematic methodology for debugging that prevents the "first error fix and declare done" anti-pattern. Ensures root cause analysis, impact assessment, and verified fixes.

---

When to Use This Skill

Load this skill when the user request matches ANY of these patterns:

Trigger Pattern	Example User Request
Bug investigation	"Why is X failing?", "Debug this error", "Investigate this issue"
Test failures	"Tests are failing", "Fix the broken tests", "Why did CI fail?"
Error analysis	"Getting this error...", "Exception in...", "Stack trace shows..."
Regression hunting	"This used to work", "Something broke after...", "Regression in..."
Fix validation	"Did the fix work?", "Verify this change", "Make sure it's fixed"
Root cause requests	"What's causing this?", "Find the root cause", "Why does this happen?"

DO NOT use for:

• New feature implementation (no existing bug)
• Refactoring without reported issues
• General code questions

---

Overview

The Problem

AI (and developers) often fix the first visible error and declare victory without verifying end-user functionality works. This leads to:

• Symptom fixes instead of root cause fixes
• Regressions in seemingly unrelated areas
• False confidence in fix completeness

The Solution

This skill provides:

1. Root Cause Analysis - Find the actual cause, not just symptoms
2. Impact Analysis - Map affected areas and dependencies
3. Validation Planning - Tiered test priorities (P1/P2/P3)
4. Confidence Assessment - Objective fix reliability rating
5. Escalation Triggers - When manual testing is required

---

Root Cause Analysis

The 5 Whys Method

Iteratively ask "Why?" to drill past symptoms to root cause:

Problem: Login fails with 500 error
├─ Why? → API returns null for user profile
├─ Why? → Database query found no records
├─ Why? → Migration didn't run on deploy
├─ Why? → Deployment script skipped migration step
└─ Why? → Environment variable missing

Root Cause: Missing environment variable (not the 500 error)

Key Principle: Focus on HOW and WHY the defect occurred, not just WHERE.

Hypothesis-Driven Debugging

Apply scientific method:

Step	Action
Observe	Gather data: errors, logs, stack traces, reproduction steps
Hypothesize	Form a falsifiable hypothesis about the cause
Experiment	Design tests to prove/disprove hypothesis
Conclude	If confirmed, proceed. If rejected, form new hypothesis
Document	Write down hypotheses tested (prevents circular investigation)

Binary Search / Bisection

For isolating bugs in time or space:

1. Place checkpoint at midpoint
2. Determine if bug is before or after
3. Repeat on remaining half
4. Continue until exact location found

Efficiency: O(log n) - approximately 7 tests for 100 commits.

---

Impact / Dependency Analysis

Mapping Procedure

For each affected file, identify:

Affected Code: src/auth/login.ts

Upstream (What calls this?):
├─ src/api/auth-routes.ts → POST /login
├─ src/middleware/session.ts → validateSession()
└─ src/pages/login.tsx → handleSubmit()

Downstream (What does this affect?):
├─ User dashboard → fetches profile on load
├─ Settings page → assumes profile exists
└─ API responses → include user context

Risk Scope Assessment

Scope	Definition	Validation Approach
Isolated	Self-contained, no affected areas	Unit tests only
Medium	1-5 affected areas	Integration tests
Broad	>5 affected areas, cross-cutting	Integration + E2E + manual

---

Debug Report Format

Location

Debug reports are written to: logs/debug-reports/{issue-id}-{timestamp}.yaml

This location is separate from standard agent logs to:

• Enable easy reference by downstream pipeline stages
• Keep debug context grouped by issue
• Allow FixValidator to read validation plans

Schema Overview

debug_report:
  metadata:
    issue_id: "{id}"
    timestamp: "{ISO-8601}"
    analyzer: "bulwark-issue-analyzer"
    complexity: low | medium | high

  analysis:
    symptom: "{user-visible problem}"
    root_cause: "{underlying reason}"
    fix_approach: "{recommended approach}"

  impact_analysis:
    affected_files: [...]
    upstream_dependencies: [...]
    downstream_effects: [...]
    risk_scope: isolated | medium | broad

  validation_plan:
    tests_to_execute:
      - path: "{test file}"
        reason: "{why this test}"
        priority: 1  # P1=must, P2=should, P3=nice-to-have
    functionalities_to_validate: [...]

  confidence_criteria:
    high: [...]
    medium: [...]
    low: [...]

  debug_journey:  # Required for medium/high complexity
    hypotheses_tested: [...]

Full schema: See references/debug-report-schema.md

---

Validation Plan

Tiered Test Priorities

Priority	Definition	Action
P1 (Must)	Direct tests of affected function	Always run
P2 (Should)	Integration tests for upstream/downstream	Run if time/tokens allow
P3 (Nice-to-have)	E2E tests, edge cases	Run if available

Selection Criteria

P1 Tests (always run):

• Tests in the same file/module as the fix
• Tests named after the affected function
• Tests that were failing before the fix

P2 Tests (run if possible):

• Tests for upstream callers
• Tests for downstream consumers
• Tests for related functionality

P3 Tests (run if available):

• E2E tests that include the affected flow
• Performance tests
• Edge case tests

Functionality Validation

Beyond tests, list user-level verifications:

• "User can log in successfully"
• "Dashboard displays correct data after login"
• "Session persists across page refresh"

---

Confidence Rubric

Assessment Criteria

Level	Criteria
High	All P1-P2 tests pass, no regression, at least one functionality verified
Medium	P1 tests pass, some P2-P3 skipped or not applicable
Low	Tests cannot reliably validate, broad risk scope with untested paths

Determining Complexity

Complexity	Indicators
Low	Single file, isolated change, clear cause
Medium	2-5 files, some dependencies, requires investigation
High	>5 files, cross-cutting, unclear cause, multiple hypotheses needed

---

Escalation Triggers

Manual testing is required when ANY of:

• Confidence level is low
• Risk scope is broad AND confidence is not high
• Any functionality cannot be validated via automated tests
• Security-related fix without security test coverage

Escalation Message Format

When escalation is triggered, the Orchestrator must inform the user:

Manual testing required for:
- [functionality 1]
- [functionality 2]

Reason: [why automated validation insufficient]

Recommended manual test steps:
1. [step]
2. [step]

---

Debug Journey Logging

When Required

Complexity	Debug Journey Required?
Low	Optional
Medium	Required
High	Required

Format

debug_journey:
  started_at: "{timestamp}"
  completed_at: "{timestamp}"

  hypotheses_tested:
    - hypothesis: "Database connection timeout causing failure"
      tested_at: "{timestamp}"
      method: "Checked DB logs and connection pool stats"
      result: rejected
      evidence: "DB logs show successful queries, pool not exhausted"

    - hypothesis: "Null profile object when user has no data"
      tested_at: "{timestamp}"
      method: "Added logging at profile access point"
      result: confirmed
      evidence: "Stack trace shows NPE at profile.name access"

  investigation_path:
    - "Started with error log analysis"
    - "Ruled out infrastructure issues"
    - "Narrowed to application code"
    - "Identified profile access as failure point"

---

Anti-Patterns

What NOT to Do

Anti-Pattern	Description	Correct Approach
Shotgun Debugging	Random changes hoping bug disappears	Hypothesis-driven approach
Fix Without Verify	Declaring "fixed" without running tests	Always verify at code + user level
Symptom Fixing	Adding workarounds instead of root cause fix	Use 5 Whys
Blind AI Trust	Accepting suggestions without verification	Test the failing scenario
Full Regression	Running 4000+ tests after every fix	Use tiered validation
Circular Investigation	Repeating tested hypotheses	Document debug journey

Full checklist: See references/anti-patterns.md

---

Quick Reference

Before Fixing

• Symptom identified and documented
• Root cause analysis completed (not just symptom)
• Impact analysis completed (upstream/downstream)
• Risk scope assessed (isolated/medium/broad)
• Complexity determined (low/medium/high)

After Fixing

• Fix implemented
• P1 tests pass
• P2 tests pass (or documented why skipped)
• Confidence level assessed
• Manual testing escalated if required
• Debug journey documented (if medium/high complexity)
• Debug report written to logs/debug-reports/

Complexity → Actions

Complexity	Debug Journey	Validation	Escalation
Low	Optional	P1 tests	Only if low confidence
Medium	Required	P1 + P2 tests	If broad scope
High	Required	P1 + P2 + P3	Almost always

---

Integration

Pipeline Integration

This skill is used by the Fix Validation pipeline:

IssueAnalyzer (loads this skill, produces debug report)
|> FixWriter (reads debug report for context)
|> TestWriter (reads validation plan)
|> FixValidator (executes validation plan, assesses confidence)
|> CodeReviewer (reviews all, makes approval decision)

Agent Integration

Agent	Usage
bulwark-issue-analyzer (P1.2)	Loads via skills: issue-debugging, produces debug report
bulwark-fix-validator (P1.3)	Loads via skills: issue-debugging, executes validation plan

Artifact Flow

IssueAnalyzer
    └─> logs/debug-reports/{issue-id}.yaml
              │
              ├─> FixWriter (reads root cause, fix approach)
              ├─> TestWriter (reads validation plan)
              ├─> FixValidator (executes tests, assesses confidence)
              └─> CodeReviewer (reviews everything)

---

Related Skills

• subagent-prompting (P0.1) - 4-part template for agent invocation
• subagent-output-templating (P0.2) - Output format for logs
• pipeline-templates (P0.3) - Fix Validation pipeline definition