Training pipeline debugging

Skill yeaight7/agent-powerups/plugins/machine-learning-ops/skills/training-pipeline-debugging

Curated power-ups for coding agents: skills, slash commands, MCP configs, hooks, AGENTS.md templates, and workflows for serious software engineering. Claude Code, Codex, Antigravity CLI, Cursor and more

Install

npx -y skills add yeaight7/agent-powerups --skill training-pipeline-debugging

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

6 stars6 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use when an ML training run fails or misbehaves -- NaN losses, out-of-memory errors, shape mismatches, or a model that cannot fit even a single batch.

SKILL.md

2.3 KB, 476 tokens by cl100k_base, as published. Nobody here has run it

Purpose

ML training bugs are often silent mathematical errors rather than explicit code crashes. This protocol localizes the standard failure classes quickly.

When to Use

Loss becomes NaN or diverges
Training crashes with out-of-memory errors
Shape/dimension errors at matrix multiplications or loss computation
The pipeline runs but the model never learns

Inputs

The failing training script and its error output or loss curve

Workflow

Run the overfit test first: fit a single batch of ~10 examples. If the model cannot achieve near-zero loss on a single batch, the pipeline is fundamentally broken — do not debug full runs until the single-batch test passes.
NaN losses — check:
- Learning rate too high?
- Missing data (NaNs in input)?
- Log/Exp/Divide-by-zero in custom loss functions?
- Exploding gradients (clip gradients)?
OOM (out of memory):
- Reduce batch size.
- Check for memory leaks in the training loop (e.g., accumulating history across epochs without .detach()).
Shape mismatches:
- Add temporary print statements or assertions on tensor.shape before matrix multiplications and loss calculations.

Output

The identified failure class and its specific fix, or a single-batch reproduction showing exactly where the pipeline breaks

Verification

Single-batch overfit test run before any full-run debugging
For NaN: learning rate, input NaNs, unsafe math, and gradient explosion all checked
For OOM: batch size and training-loop accumulation both checked
For shapes: assertions placed before matmul/loss sites
Fix validated by re-running the previously failing case

Failure Modes

Debugging full runs first — hours per iteration; the single-batch test gives answers in minutes.
Treating NaN as random — NaN losses have a small set of causes; check all four systematically instead of restarting with a lower LR.
Batch-size-only OOM fixes — shrinking the batch hides a loop leak that will OOM again later; check accumulation too.
Leftover instrumentation — remove temporary shape prints/assertions once the fix is validated.