Ml experiment design

Skill pengzhangzhi/superpowers-ml/skills/ml-experiment-design

Skills that make your AI agent do ML research like a skeptical scientist, not a hype machine.

Install

npx -y skills add pengzhangzhi/superpowers-ml --skill ml-experiment-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

5 stars5 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use when turning a rough ML idea into a runnable experiment - define the experiment card (question, hypothesis, baseline, primary metric, budget, cheapest verification rung) before writing training code.

SKILL.md

3.3 KB, as published. Nobody here has run it

ML Experiment Design

Overview

An ML idea is not runnable until it is a CARD. This skill turns a rough idea into a compact, one-screen experiment card — rung R0, the precondition for planning or launching anything. Fill it WITH your human partner; do not write training code until the card is settled.

Core principle: One change under test, one locked metric, one decision rule. If you cannot state in advance what result would change your mind, you are not ready to run.

Upstream: the idea comes from superpowers-ml:brainstorming. Downstream: the card drives superpowers-ml:ml-feedback-ladder (how to verify it cheaply) and superpowers-ml:writing-plans (the tasks).

The Experiment Card

Settle every field. Keep each to a line or two — a card, not a document.

Research question — the one question this run answers.
Hypothesis — specific and falsifiable: what changes, and the direction you expect.
Baseline — the exact, runnable comparison (config / commit, not "the usual setup").
Variant(s) — the SINGLE change under test. Anything else is a separate experiment.
Primary metric — one metric, locked now, that decides the outcome.
Guardrail metrics — what must NOT regress while the primary metric moves (cost, latency, memory, a quality/safety metric).
Dataset / split — train / val / test, and the leakage risk you checked for.
Seed policy — how many seeds, fixed or swept; how you tell seed noise from a real effect.
Budget — the compute / wall-clock ceiling you will spend before stopping to decide.
Cheapest useful rung — the smallest rung that tells you something real, to start at (see superpowers-ml:ml-feedback-ladder).
Success criterion — the threshold on the primary metric that CONFIRMS the hypothesis, set before running.
Exploratory-only — what this run may suggest but can never confirm (anything the locked primary metric does not measure).

One Change At A Time

A variant is ONE change, or you cannot attribute the result — two changes is two experiments. If your human partner wants to vary many things, that is a study: write a card per change, then design a ladder for each.

Template

Question:         ...
Hypothesis:       changing X will <increase|decrease> <metric> because ...
Baseline:         <config / commit>
Variant:          <the single change>
Primary metric:   <one, locked>     Success: <threshold set in advance>
Guardrails:       <must-not-regress>
Data / split:     <train / val / test>     Leakage check: <...>
Seeds:            <n, fixed | swept>
Budget:           <GPU-hours / wall-clock ceiling>
Start rung:       <cheapest useful rung>
Exploratory-only: <what will not count as confirmation>

Red Flags - STOP

No locked primary metric, or the metric is chosen after seeing results.
Success criterion not written down before the run.
More than one change under test in a single variant.
"We'll know it when we see it" — no decision rule.

A card you can fill in five minutes saves a burned cluster job. Write it first.