Experiment rigor

Skill grimaldost/craft-collection/plugins/experiment-discipline/skills/experiment-rigor

A Claude Code plugin marketplace that codifies engineering craft: disciplined Python and data-engineering practice alongside deliberate session-knowledge workflows, packaged as reusable skills.

Install

npx -y skills add grimaldost/craft-collection --skill experiment-rigor

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Structure an experiment and its write-up so the Methods reconstruct without the conversation and the uncertainty is declared, not hidden — a typed record.yaml across a probe / measurement / decision tier ladder, every load-bearing rule a script that exits non-zero rather than a line of prose. Use when you pre-register an A/B, freeze a plan before running it, write up an experiment or a comparison, add error bars or a confidence interval to a rate, reconcile declared cells against the runs that happened, separate confirmatory from exploratory outcomes, name the threats to validity, design the 2x2, or ask whether a skill, tier, model, or strategy actually helps and how you would show it rigorously. Covers the small-n refusal (no CLT below 30 — Wilson, Clopper-Pearson, or a within-experiment Beta-Binomial), the design-arithmetic reconciliation (declared cells == disposition == denominators), the plan freeze and its drift gate, the rate that needs both a numerator and a denominator, probe self-labeling, and record-is-source with the report derived. Not for running the scenario matrix or scoring a bank — that is fathom's fathom-eval; not for measuring one skill's trigger activation — that is evaluate-skill; not for judging whether a spec is ready to build — that is keel's Definition of Ready; and not for a throwaway spot-check you will decide nothing on.

SKILL.md

8.6 KB, as published. Nobody here has run it

Experiment Rigor

Turn an experiment into a typed record.yaml whose report.md is derived, not written. This is a rigid skill: the scripts under scripts/ are the mechanism, and each bright line below is a gate that exits non-zero with a stable error code (validate.py), not advice to remember. The dividing line the discipline draws: methods uncertainty is disqualifying, effect uncertainty is declarable — a reader who cannot reconstruct what was manipulated, where, and how it was measured has found a defect; a wide confidence interval is an honest result.

The tier ladder

tier is a field, not a second skill. It selects the required fields and which gates apply. Tier-0 is the exception: it names no artifact, so it never appears as a tier: value.

check (tier-0) — the structured check. An evaluation act answered inline in the response: no file, no record, no gate. An evaluation act asks a question whose correct answer is a valuative claim ("is this effective", "which is better"); an execution or lookup request asks for an action or a fact and owes nothing here. The five-element shape — method, metric, result(s) with denominators, conclusion, and a one-line "what this updates" — the boundary, and two worked examples live in references/report-skeleton.md. This rung is guidance, not a bright line, and entry above it is unchanged: a record is owed when a decision rides on the result.
probe — cheap and exploratory. Refuses a confirmatory verdict or any posterior; a probe that wants either graduates to measurement.
measurement — a frozen pre-registration and a reported interval on every rate.
decision — adds the comprehension gate: fresh-context readers reconstruct the Methods before the result is allowed to move a decision.

The loop

Copy the tier skeleton from templates/<tier>.yaml; the field guide is templates/SCHEMA.md (generated from templates/schema.json, the canonical schema).
Author the record. While drafting, validate.py <record> --schema-only checks shape without the context gates (they are skipped and listed).
Freeze the plan: commit the pre-registration and record plan_frozen_at.commit before the first run.
Run the experiment; fill results. from_fathom.py maps a fathom ledger into the run-derived fields.
validate.py <record> runs the full gate. render.py <record> derives report.md; render.py --check is the drift gate over a committed pair. Never hand-edit report.md.

The activation line

Whenever the frame engages, one plain line opens the work product, naming the tier and the artifact behind it. At probe and above it is generated, not typed: render.py --activation-line <record> prints it, and --check-activation-line "<line>" <record> exits non-zero when the tier or the path disagrees — the line is a claim tied to the artifact, not a badge. At tier-0 the artifact reference is the literal inline, which nothing resolves.

[experiment-rigor | check -> inline]
[experiment-rigor | measurement -> experiments/retry-backoff/record.yaml]

Bright lines

Each is a gate in validate.py; the error code is named so a failure points at the rule.

The freeze (ER-PREREG). The pre-registration subset — design.cells, each outcome's role, operationalization, and verifier hash, and analysis_plan — is reconstructed with git show on plan_frozen_at.commit and compared to the analyzed record. Any drift fails. A confirmatory verdict is legal only on an outcome whose frozen role is confirmatory.
Declared-cells reconciliation (ER-RECON). N_expected is the sum of design.cells[].planned_n, and it must equal the disposition total and every outcome's sum of arm denominators. The model tier is one named factor level, not a separate multiplier.
The confirmatory / exploratory partition. outcomes[].role belongs to the frozen plan. An outcome added after the freeze carries added_after_freeze: true and role: exploratory — the quarantine — and may report only an exploratory_signal or inconclusive verdict, never a confirmatory one.
The small-n CI refusal (ER-STATS). No CLT / normal method below a cell denominator of 30. Allowed: wilson, clopper_pearson, beta_binomial. Every stated interval is recomputed from stats.py and must match to four decimals. See references/small-n-stats.md.
A rate needs both a numerator and a denominator (ER-SCHEMA). A rate lives under results.<outcome>.arms.<arm> with both; a loose rate at the outcome level fails.
Record is source, report is derived (ER-PARITY, render.py --check). The record is the single source of truth; report.md is regenerated and must not contradict it.
Probe self-labeling (ER-PROBE). A probe carrying a confirmatory verdict or a posterior fails; the message names the graduation path to measurement.
Threat coverage (ER-THREAT). Every core threat in the closed enum carries a row with a status and a statement; silence on one fails. The enum and what each key means are in references/threats-catalog.md.

Statistics and threats

Small-n interval methods, clustered and paired standard errors, the Beta(1, 1) prior and its sensitivity, and the within-experiment-only pooling boundary: references/small-n-stats.md.
The nine-key closed threat enum, and when each is controlled versus residual: references/threats-catalog.md.

What binds, role-generically

Readiness before build. When a spec-readiness gate is installed (for example keel's Definition of Ready), a decision-tier experiment that will inform a build defers spec readiness to it rather than re-deciding it here. Absent one, the experiment stands alone.
Fresh-context readers. The decision-tier comprehension block records independent readers who reconstruct the Methods with no access to this conversation. Any fresh-context reader tooling that is installed can produce those reads; the block records the transcript path and the four verbatim answers, and their genuineness is ceded to review. The gate checks presence, resolution, and unanimity, not sincerity.

Boundaries

Running the scenario matrix, scoring a bank, or driving the paid run is fathom's fathom-eval; this skill structures and gates the record that results. Measuring one skill's trigger activation is evaluate-skill. Judging whether a spec is ready to decompose is a readiness gate (keel's Definition of Ready). A throwaway spot-check you will decide nothing on needs none of this.

Correct-usage check

The record passes validate.py at its declared tier (exit 0).
The Methods reconstruct from the record alone — what was manipulated, where it was placed, how each outcome was operationalized, whether execution was real — without the conversation.
Every reported rate carries a recomputable interval; no CLT below 30.
report.md is derived (render.py), not hand-edited (render.py --check clean).
The work product opens with the activation line — inline at tier-0, and at probe and above the generated line, --check-activation-line clean.