agentsclimarketplace
Sign in

Experiment design

Skill gioviat/research-toolkit/skills/experiment-design

Use when designing a new experiment, planning an ablation, choosing baselines or metrics, or deciding how many seeds/runs are needed before any code is written. Trigger on "design an experiment", "how should I test", "what's the right baseline", "how many seeds".From its SKILL.md

Install
npx -y skills add gioviat/research-toolkit --skill experiment-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

3.1 KB, as published. Nobody here has run it

Experiment design

Before any implementation, produce a written design covering the items below. If an item cannot be satisfied given the project's constraints, say which one and why, rather than silently omitting it.

Required before writing code

  • Hypothesis: the specific, falsifiable claim being tested. Not "does X help" but "does X improve metric M by at least an amount that matters, under condition C."
  • Metric: the exact formula, not just its name. State how it's computed, over what set, with what aggregation.
  • Baselines: what the result is compared against, and why those are the right baselines (not just the most convenient ones).
  • Independent variables: what is being varied; everything else must be held fixed and stated explicitly.
  • Hardware environment: specify the exact target infrastructure (e.g., local single-GPU workstation, SLURM-based HPC cluster, cloud multi-node TPUs). This dictates scaling constraints, expected execution time, and parallelization strategy.
  • Compute/data budget: what's affordable given the stated hardware, and whether it's sufficient to detect an effect of the size being claimed. If underpowered, say so up front.
  • Seeds / repetitions: minimum 3 seeds for any result that will be reported as a comparison; state the number and justify it if fewer.
  • Statistical test: which test will be used to compare conditions (e.g., paired t-test, bootstrap CI, Wilcoxon), decided before seeing results — not chosen post hoc to fit the data.
  • Failure criteria: what result would count as "the hypothesis is not supported." State this before running anything.

During implementation and while running

  • One directory per run under experiments/results/<run-id>/, holding the full configuration (all hyperparameters, seed, code version/commit, and specific hardware topology used) and the raw metrics. A result with no config attached cannot be reported.
  • Write outputs only to experiments/results/, never edited by hand afterward.
  • Launch long runs in the background and check them periodically rather than blocking on them. Ensure checkpointing is implemented to handle preemptions on shared hardware (e.g., HPC clusters).
  • A run that crashed or turned out misconfigured is recorded with its config and failure, not deleted. Runs that silently disappear are a form of cherry-picking.

Common design errors to check for explicitly

  • Comparing against a weak or outdated baseline
  • Metric that doesn't match the stated hypothesis
  • Cherry-picking the seed or checkpoint with the best result
  • Reporting a single run as if it were representative
  • Choosing the statistical test after looking at the data
  • Overlooking hardware bottlenecks (e.g., being I/O bound on network storage in an HPC, or failing to account for multi-GPU communication overhead)

Keep looking

Skills are one crate of 627,801. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.