agentsclimarketplace

Experiment planner

Skill maximussthegreat/ml-researcher-os/skills/experiment-planner

Agent skills and workflows for reproducible ML research.

Install
npx -y skills add maximussthegreat/ml-researcher-os --skill experiment-planner

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use when turning an ML hypothesis, paper claim, or model idea into a reproducible experiment plan with baselines, metrics, seeds, risks, and expected artifacts.

SKILL.md

1.7 KB, as published. Nobody here has run it

Experiment Planner

Use this skill after claim extraction and before writing training code.

Goal

Convert a research claim into the smallest controlled experiment that can produce useful evidence.

Required inputs

  • Hypothesis or paper claim
  • Dataset or proposed dataset
  • Target metric
  • Compute budget
  • Existing baseline, if any

If any input is missing, ask for it or mark it as unknown and make a conservative plan.

Required output

Write an experiment plan with:

  1. Hypothesis
  2. Minimum viable experiment
  3. Baselines
  4. Ablations
  5. Data split strategy
  6. Metrics
  7. Seed policy
  8. Failure modes
  9. Logging and artifacts
  10. Stop criteria

Baseline rules

Every plan needs at least one baseline. Prefer:

  • simplest non-neural baseline
  • standard library baseline
  • previously reported baseline from the source paper
  • ablation of the proposed method

Failure modes to check

  • data leakage
  • train/validation split mismatch
  • metric mismatch
  • hidden preprocessing fit on validation or test data
  • seed sensitivity
  • tiny test set
  • weak baseline
  • cherry-picked run
  • missing negative result

Rules

  • Do not plan an experiment without at least one meaningful baseline.
  • Do not compare a heavily tuned model against an untuned baseline.
  • Do not claim reproduction when the dataset, split, or metric differs from the source claim.
  • Prefer three small controlled runs over one expensive ambiguous run.

Output style

Be concrete. Prefer a small experiment that can run today over an impressive plan that cannot be verified.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.