agentsclimarketplace
Sign in

Design model experiment

Skill gaelic-ghost/socket/plugins/model-lab-skills/skills/design-model-experiment

Design a reproducible model experiment with hypotheses, controls, baselines, metrics, budgets, provenance, artifacts, and stop conditions. Use before model work whose results must support a decision.From its SKILL.md

Install
npx -y skills add gaelic-ghost/socket --skill design-model-experiment

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 6 stars6 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.8 KB, 314 tokens by cl100k_base, as published. Nobody here has run it

Design Model Experiment

Outcome

Produce an experiment manifest that another operator can run, audit, and compare without reconstructing unstated choices.

Workflow

  1. Write one falsifiable hypothesis and the decision it informs.
  2. Define the experimental unit: model revision, adapter, prompt set, intervention, runtime, or harness.
  3. Name exactly one primary controlled variable per comparison. Record every intentional difference.
  4. Select a baseline and explain why it is a fair comparator.
  5. Pin model, tokenizer, dataset, code, dependency, template, evaluator, and seed provenance.
  6. Define primary metrics, guardrail metrics, uncertainty treatment, and failure thresholds before the run.
  7. Estimate compute, storage, time, and paid cost. Set smoke-test and full-run stop conditions.
  8. Define raw and derived artifacts, retention, and sensitive-data handling.
  9. Copy assets/experiment-manifest.yaml, fill it, then run:
python3 scripts/validate_experiment_manifest.py path/to/experiment.yaml
  1. Run the smallest experiment capable of detecting configuration or pipeline failure before spending the full budget.

Evidence Contract

Keep configuration validation, smoke-run evidence, and final experimental evidence separate. A successful process exit proves execution, not model quality. Report deviations from the manifest before interpreting results.

Resources

  • assets/experiment-manifest.yaml: portable experiment template.
  • references/experiment-design.md: field semantics and comparison rules.
  • scripts/validate_experiment_manifest.py: deterministic structural validation.

What ships with it: 4 files

7.7 KB alongside SKILL.md, 1 of them executable

agents/

assets/

references/

scripts/

Keep looking

Skills are one crate of 326,782. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.