agentsclimarketplace

Training loop debugger

Skill maximussthegreat/ml-researcher-os/skills/training-loop-debugger

Agent skills and workflows for reproducible ML research.

Install
npx -y skills add maximussthegreat/ml-researcher-os --skill training-loop-debugger

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use when reviewing ML training code, suspicious metrics, unstable loss, broken evaluation, or a model that appears too good to be true.

SKILL.md

1.5 KB, as published. Nobody here has run it

Training Loop Debugger

Use this skill to review training and evaluation code before trusting results.

Goal

Find the smallest code or protocol issue that could invalidate reported model metrics.

Review order

  1. Data loading
  2. Split creation
  3. Preprocessing fit and transform
  4. Model initialization
  5. Loss and optimizer
  6. Training mode and eval mode
  7. Metric computation
  8. Checkpoint selection
  9. Test set use
  10. Logging and reproducibility

High-risk bugs

Immediately flag:

  • fitting scalers, tokenizers, imputers, or feature selectors on validation or test data
  • using test data for early stopping
  • computing metrics on logits when probabilities or labels are required
  • forgetting model.eval() during evaluation
  • forgetting torch.no_grad() during evaluation
  • using different preprocessing at train and eval time
  • reporting the best validation run as test performance
  • changing seeds until a good result appears

Required output

AreaFindingSeverityEvidenceFix

Severity must be one of:

  • blocker
  • high
  • medium
  • low

Rules

  • Do not claim the model is fixed until a rerun is available.
  • Do not rewrite the whole project when a small patch explains the issue.
  • If logs are missing, ask for logs or mark the conclusion as provisional.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.