Experimental design
Skill thada2402/AutoResearchClaw/researchclaw/skills/builtin/experiment/experimental-design
Best practices for designing reproducible ML experiments. Use when planning ablations, baselines, or controlled experiments.From its SKILL.md
npx -y skills add thada2402/AutoResearchClaw --skill experimental-designAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
1.1 KB, 157 tokens by cl100k_base, as published. Nobody here has run it
Experimental Design Best Practice
- ALWAYS include meaningful baselines (not just random):
- At least one classical method baseline
- At least one recent SOTA method baseline
- A simple-but-strong baseline (e.g., linear probe, k-NN)
- Use MULTIPLE random seeds (minimum 3, ideally 5)
- Report mean +/- std across seeds
- Design ablations that isolate EACH key component:
- Remove one component at a time
- Each ablation must be meaningfully different from baseline
- Control variables: change only ONE thing per comparison
- Use standard splits (train/val/test) — never test on training data
- Report wall-clock time and memory usage alongside accuracy
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.