Simulation failure triage

Simulation Failure Triage

Goal

Classify common simulation failure signatures and return immediate actions, retry ladders, and stop conditions.

Requirements

• Python 3.10+
• No external dependencies
• Works on Linux, macOS, and Windows

Inputs to Gather

Input	Description	Example
Code	Simulation code	LAMMPS, VASP, MOOSE, QE
Stage	Setup, runtime, postprocess	runtime
Symptoms	Failure signs	nan,pressure-blowup
Log text or file	Error evidence	Lost atoms, ZBRENT
Recent change	Last modified setting	larger timestep

Decision Guidance

• First preserve evidence: logs, inputs, executable version, and scheduler output.
• Separate setup errors from numerical instability and physical model issues.
• Retry with a single controlled change.
• Stop retrying when the result becomes scientifically meaningless or a required model input is missing.

Script Outputs

scripts/failure_triage.py emits:

• likely_causes
• immediate_actions
• retry_ladder
• stop_conditions
• evidence

Workflow

python3 skills/robustness/simulation-failure-triage/scripts/failure_triage.py \
  --code LAMMPS \
  --stage runtime \
  --symptoms nan,pressure-blowup \
  --recent-change "increased timestep" \
  --json

Error Handling

Invalid stages or oversized log files stop with exit code 2. Unknown symptoms are retained as custom evidence.

Limitations

This skill gives first-response triage. It does not guarantee that a failed simulation can be repaired.

Verification checklist

• Recorded the preserved-evidence set BEFORE any rerun: copied logs, input decks, scheduler output, and the executable/version string (per the first immediate_actions item) so the original failure is reproducible.
• Quoted the FIRST warning/error from the run, not just the final crash line, and confirmed evidence.log_excerpt (first 500 chars, original casing) captures it; re-run with --log-file/--log-text if the excerpt misses the root signature.
• Logged every likely_causes entry with its symptom, category, and first_action, and verified no real signature landed in the category: custom bucket because its keyword was absent from LOG_HINTS/PATTERNS.
• Confirmed the failure category is correct rather than mislabeled: a segfault/SIGSEGV/signal 11/core-dumped maps to crash (not corrupted-output), and an OOM/bad_alloc/oom-kill maps to out-of-memory (not incomplete-run).
• Applied the retry_ladder strictly one change at a time, recording the single parameter changed and its effect at each rung (including the memory rung when memory-bound).
• Checked every stop_conditions entry and halted the retry loop instead of stacking arbitrary stabilizing tweaks when any condition is met (e.g. unverifiable potential/pseudopotential, results dependent on arbitrary stabilizers).
• After any numerical recovery, handed off to the simulation-validator skill for conservation / physical-bounds / "can I trust these results" checks rather than declaring success here.

Common pitfalls & rationalizations

Tempting shortcut	Why it's wrong / what to do
"The last line of the log is the error, so I'll fix that."	The final crash line is often a downstream symptom; the script's second immediate_actions item exists because the FIRST warning/error is the real cause. Capture it in evidence.log_excerpt and triage from there.
"It segfaulted, so it's a corrupted-output / disk issue."	A crash (SIGSEGV/signal 11/core dumped) is a memory-fault category pointing at out-of-bounds/null-pointer/ABI bugs and a gdb/valgrind/ASan repro — not I/O. Do not treat it as corrupted-output.
"The job got killed, so I'll just bump the walltime."	killed alone is genuinely ambiguous and stays incomplete-run; an OOM signature (bad_alloc, oom-kill, "out of memory") is a distinct out-of-memory cause needing memory reduction / --mem increase, not more walltime.
"I'll lower dt AND change the preconditioner AND adjust the barostat to get it running."	The retry ladder is one controlled change per rung; stacking fixes makes results "depend on arbitrary stabilizing changes" — an explicit stop_conditions trigger. Change one parameter, record its effect, then proceed.
"I added the missing PAIR coeff path and it ran, so the physics is fine."	A clean run after fixing missing-potential/bad-pseudopotential resolves setup, not validity; species mapping, valence, or functional may still be wrong. Run completion is not correctness.
"It runs now, triage is done."	Triage only restores execution. Conservation, physical bounds, and convergence are out of scope here — defer to simulation-validator before trusting any number.

Security

Input Validation

• --stage is checked against a fixed allowlist (setup, runtime, postprocess, unknown); any other value exits with code 2.
• --code must be non-empty and at most 100 characters; an empty or oversized value exits with code 2.
• --symptoms are split on commas and capped at 50 entries, each at most 100 characters; exceeding either limit exits with code 2.
• Unknown symptoms are not rejected; they are retained as custom evidence rather than silently dropped.
• There are no numeric arguments, so no finite/positive checks apply.
• All input-validation failures (and log read errors) are caught and exit with code 2 and a stderr message.

File Access

• The only file read is the optional --log-file path; when omitted, the script reads no files.
• Before reading, the log file's size is checked via stat; if it exceeds the 10 MB cap (MAX_LOG_SIZE) the script raises and exits with code 2. Both file content and --log-text are then truncated to 10 MB.
• The script writes no files; all results go to stdout (JSON with --json, otherwise plain text).
• Path-sandboxing caveat: the log path is not restricted to a working directory, so the script can read any file the invoking process has permission to read; only pass trusted paths.

Tool Restrictions

• allowed-tools is Read, Bash, Write, Grep, Glob.
• Bash runs only the bundled scripts/failure_triage.py.
• Read opens the skill's own files and a user-provided log to inspect failure evidence; Grep/Glob locate log files, inputs, and prior output; Write records triage notes or preserved evidence at the user's direction. The bundled script itself does not write files.

Safety Measures

• No eval/exec and no dynamic code execution; log text is treated as data and never run.
• The script spawns no subprocesses and runs no external solvers.
• Output is structured JSON (--json), making results machine-checkable.
• DoS caps bound resource use: 10 MB log size/text limit, 50 symptoms, and 100-character limits on code and each symptom.

References

• See references/failure_patterns.md for common failure signatures and retry ladders.

Version History

• 1.1.3: Add Verification checklist (evidence-based checks tied to failure_triage.py outputs — preserved-evidence set, first-error excerpt, category-mislabel guards, one-change retry ladder, stop conditions, simulation-validator handoff) and Common pitfalls & rationalizations table covering last-line-only triage, segfault-as-I/O, OOM-vs-walltime, stacked stabilizing changes, and "it ran so it's valid".
• 1.1.1: Strengthen evals to be discriminating — each case now pins the exact failure_triage.py JSON output via script_checks (specific category/first_action strings, log-hint symptom inference, fixed retry-ladder/stop-condition text, recent-change immediate action), so the suite measures the skill's actual output rather than generic triage knowledge.
• 1.1.0: Add dedicated out-of-memory and crash (segfault) classifications; segfaults and OOM are no longer mislabeled as I/O or interrupted-execution. Preserve original-case log excerpt. Add input-validation caps (symptoms, code length). Sharpen description to defer deep validation to simulation-validator. Fix missing-potential eval token.
• 1.0.0: Initial cross-code simulation failure triage skill.