agentsclimarketplace

Experiment design

Skill jacazjx/AI-Research-Orchestrator/skills/experiment-design

Design experiments at pilot or full scale. Scope adapts to current phase: pilot phase produces a minimal validation experiment (< 24hr), experiments phase produces a comprehensive matrix with ablations, baselines, and statistical rigor. Use when user says "design pilot", "design experiments", "设计 Pilot", "设计实验".From its SKILL.md

Install
npx -y skills add jacazjx/AI-Research-Orchestrator --skill experiment-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.5 KB, as published. Nobody here has run it

Experiment Design

Purpose

Design experiments at the appropriate scale for the current phase. The agent decides scope, structure, and detail level based on context.

Mode Selection

PhaseModeInputOutputTime Budget
PilotMinimal Pilot Designdocs/pilot/problem-analysis.mddocs/pilot/pilot-design.md< 24 hours
ExperimentsFull Matrix Designdocs/pilot/pilot-validation-report.mddocs/experiments/experiment-spec.mdDays-Weeks

Statistical Analysis Guide

Use these standards when designing the evaluation protocol:

Significance Tests

ScenarioRecommended Test
Two models, same test setPaired t-test (alpha = 0.05)
Multiple models comparedANOVA + Bonferroni correction
Non-normal distributionsWilcoxon signed-rank test
Multiple hypothesesApply Bonferroni or Holm-Bonferroni correction

Effect Size

  • Cohen's d for pairwise comparisons: small (0.2), medium (0.5), large (0.8)
  • Report alongside p-values -- statistical significance without practical significance is misleading

Result Reporting Format

Mean +/- std across N seeds (N >= 3)
95% confidence intervals
p-values for key comparisons
Effect size (Cohen's d)

Power Analysis (Full Matrix)

Before committing resources, estimate required sample size:

  • Desired power: 0.8 (standard) or 0.9 (for key claims)
  • Expected effect size: estimate from pilot results
  • Significance level: 0.05

Key Rules

  1. Pilot must complete in < 24 hours -- scope ruthlessly
  2. Success criteria must be unambiguous -- measurable, with clear thresholds
  3. Minimum 3 seeds per configuration for statistical validity
  4. Must include both strong AND weak baselines in full matrix
  5. Resource estimates need 20% buffer minimum
  6. Multiple comparison corrections required when testing multiple hypotheses
  7. Include rollback plan if pilot fails

Keep looking

Skills are one crate of 608,666. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.