Systematic debugging

Systematic Debugging

Overview

Random fixes waste time. Investigate root cause first.

Core principle: ALWAYS find root cause before attempting fixes.

Violating the letter of this process is violating the spirit of debugging.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Phase 1: Gather Evidence

1. Read error output — Read the failed stage's output and error files completely. Don't skip.
2. Read progress.log — What was attempted? What happened before the failure?
3. Read stage spec — What was this stage supposed to do? What are the inputs, parameters, success criteria?
4. Reproduce — Can you trigger the failure again with the same inputs?

Phase 2: Diagnose Failure Category

Category	Symptoms	Common Causes
Input error	Missing files, wrong format, bad parameters	Typo in paths, wrong output from prior stage
Convergence failure	Iterative solver not converging, optimization stalled	Bad initial conditions, too tight criteria, wrong method
Numerical instability	NaN values, diverging quantities, unphysical outputs	Step size too large, ill-conditioned inputs, problematic initial state
Resource limit	OOM, timeout, disk full	System too large, too many k-points, long trajectory
Wrong results	Converged but incorrect answer	Wrong method/model, missing correction, inappropriate approximation
Analysis/plotting error	Missing data, wrong column, library issue	Wrong file format, missing dependency, encoding
Environment issue	Missing binary, wrong version, import error	Wrong conda env, missing package, version mismatch
Dependency conflict	Software missing feature, incompatible libs	Software missing required module, incompatible library versions

Phase 3: Hypothesis and Test

1. Form single hypothesis — "I think X is the root cause because Y"
2. Test minimally — Smallest possible change. One variable at a time.
3. Verify — Did it work? Yes → Phase 4. No → new hypothesis.
4. If 3+ fixes failed — STOP. Question whether the approach is fundamentally wrong. Discuss with user before continuing.

Phase 4: Fix and Retry

1. Apply the fix
2. Update progress.log with diagnosis and fix applied
3. Never remove the failed attempt from the log — append the retry
4. Retry the stage via superscientist:executing-workflows
5. retry_count is incremented in workflow-state.json

DPDispatcher Failures (HPC Stages)

When an HPC stage fails (any stage using a remote backend profile), always start with dpdispatcher.log — it is available locally without any download.

Step 1: Read dpdispatcher.log

cat {workflow_root}/dpdispatcher.log

Extract for the failed stage's submission:

• Job IDs: Slurm/PBS job IDs (e.g., job_id: 23085)
• Status: terminated (non-zero Slurm exit) or finished (exit 0 but DPDispatcher found issues)
• fail_count: Number of times DPDispatcher internally retried the job
• Remote path: The hash-based remote working directory (e.g., 959691370546a346be2a4770ce9b93789f42ce6a)

Step 2: Classify the failure

DPDispatcher status	Meaning	Root cause category	Next step
terminated	Scheduler killed the job	Scheduler/resource error: wrong partition, OOM, walltime exceeded, invalid GPU request, duplicate --gres	Fix resources or custom_flags in submission parameters
finished + backward_files missing	Job ran to completion but expected outputs not created	Command-level error: wrong input script, missing software package, wrong binary path	Read stage-N/log and stage-N/err if downloaded; if not, request Level 2 diagnostic reproduction run from orchestrator
DPDispatcher error (Python traceback)	DPDispatcher itself failed	SSH/network issue, remote_root misconfigured, $ref config invalid	Check machine config path, network connectivity; retry

Step 3: Check for custom_flags issues

If the job was terminated by the scheduler, check dpdispatcher.log or the original submission.json for common custom_flags problems:

• Missing #SBATCH prefix: Entries like "--qos=4gpus" instead of "#SBATCH --qos=4gpus" are silently ignored by Slurm. They appear as bare shell lines in the .sub script.
• Duplicate directives: gpu_per_node: 1 auto-generates #SBATCH --gres=gpu:1. Adding "#SBATCH --gres=gpu:1" to custom_flags creates a duplicate that some schedulers reject.

Wrong Results: The Hardest Category

When the calculation converges and files look correct, but the answer is physically wrong:

1. Check against known reference values (published data, databases, analytical solutions)
2. Check domain-appropriate bounds (values within physically/mathematically meaningful ranges)
3. Check internal consistency (outputs internally consistent — quantities derived from the same source agree)
4. If no reference exists — propose a validation calculation against a well-known test case as a new stage (via amendment protocol)

Red Flags

Thought	Reality
"I know what's wrong"	Investigate first. You might be wrong.
"Quick fix, let me try"	Root cause first. Phase 1 → 2 → 3 → 4.
"Let me try one more fix" (after 2+)	Stop. Question the approach. Talk to user.
"Results look close enough"	Check against reference. "Close" might be wrong.
"The error is in the simulation code" (HPC stage)	Read dpdispatcher.log first. The failure may be scheduler-level, not code-level.