agentsclimarketplace

Experiment design

Skill ramanbamba/10x-pm/skills/experiment-design

Prompts don't compound. Skills do. The open-source AI toolkit for product managers — 13 Claude Code skills + 3 red-team agents for the full PM workflow.

Install
npx -y skills add ramanbamba/10x-pm --skill experiment-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Design an A/B test or product experiment with a falsifiable hypothesis, guardrail metrics, sample-size sanity check, and pre-committed decision rules. Use when the user says "design an experiment", "A/B test for", "how should we test this", "is this result significant", or wants to validate a product change with data.

SKILL.md

3.7 KB, as published. Nobody here has run it

Experiment Design

Design experiments that can actually change a decision: falsifiable hypothesis, feasible sample size, and the ship/kill rule written down before the data arrives.

Before designing

  1. Ask what decision the experiment gates. If the team will ship regardless of the result, say so plainly and recommend skipping the experiment — measure post-launch instead.
  2. Ask for baseline numbers: current conversion/metric value and weekly eligible traffic. Without these, sample-size math is impossible — help estimate from analytics if needed.
  3. Ask the smallest effect worth acting on (MDE). "Any improvement" is not an answer; detecting a 0.1% lift may need more traffic than the product has.

Workflow

  1. Write the hypothesis falsifiably: "Because we observed [evidence], we believe [change] for [segment] will move [primary metric] by ≥[MDE]. We are wrong if [observable outcome]."
  2. One primary metric. Multiple primaries = p-hacking with extra steps. Secondaries are for learning, not deciding, and are labeled as such.
  3. Set guardrails: the 2–4 metrics that must not degrade (revenue, latency, unsubscribe, support contacts), each with a tolerance threshold.
  4. Sanity-check the sample size. Compute required n per arm (α=0.05, power=0.8) with the standard approximation; run the numbers in code for precision. Then the honest conversation: at current traffic, how many weeks? If >8 weeks, propose alternatives — bigger change, coarser MDE, a more sensitive proxy metric, or a non-experimental method (holdout, pre/post with controls, user tests).
  5. Pre-commit the decision rules, all four branches: significant win → [action]; significant loss → [action]; flat/inconclusive → [action — usually kill, and say so]; guardrail breach → [stop rule]. Also fix the run length up front: no peeking-and-stopping on a good day, run full weeks to capture weekly cycles.
  6. List validity threats relevant to this setup: novelty effect, seasonality, contamination between arms, segment mix shifts — with a mitigation or an acknowledgment.

Output format

# Experiment — [name]
**Decision it gates:** [what ships or dies] · **DRI:** [name]

## Hypothesis
Because [evidence], we believe [change] for [segment] will move [metric] by ≥[MDE]%.
We are wrong if: [falsifiable outcome].

## Design
Arms: [control / variant(s)] · Split: [%] · Unit: [user/session/account]
Targeting: [who's eligible] · Exclusions: [who's not, why]

## Metrics
**Primary:** [metric — exact definition]
**Guardrails:** [metric · threshold] × [2–4]
**Secondary (learning only):** [metrics]

## Power
Baseline: [X]% · MDE: [Y]% (relative) · n per arm: [computed] · Est. runtime: [Z] weeks at current traffic
[If >8 weeks: the recommended alternative]

## Decision rules (pre-committed)
- Win (p<0.05, ≥MDE): [action]
- Loss: [action]
- Flat after [N] weeks: [action]
- Guardrail breach ([threshold]): stop, [procedure]

## Validity threats
[Threat → mitigation]

Quality bar — self-check

  • Hypothesis is falsifiable and cites the evidence that motivated it.
  • Exactly one primary metric.
  • Sample size was computed, not asserted — show the numbers.
  • Runtime is honest. An underpowered experiment recommended anyway is the cardinal sin — never do it silently.
  • All four decision branches are pre-committed, including the uncomfortable "flat → kill" one.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.