Design ai experiment

/design-ai-experiment

Overview

Most "AI experiments" are not experiments. They are launches with a dashboard. A real experiment has a hypothesis you wrote down before you saw the result, a control that lets you attribute the change, a success threshold that pre-commits you to a decision, and a kill criterion that pre-commits you to stopping. This skill walks the user through producing all five before they touch the code.

Core principle: if you cannot say in advance what result would make you stop, you are not running an experiment, you are looking for confirmation.

When to Use

• User typed /design-ai-experiment <experiment idea>.
• User wants to test a new model, prompt, retrieval strategy, fine-tuning approach, or AI-feature change in production.
• User said "let's just try it and see" about something irreversible.
• A team is about to A/B test an LLM change without a written hypothesis.

Do NOT use for: debugging an already-shipped feature (use /refine), exploratory data analysis, or true research where the question is "what is the structure of this thing" rather than "does this change improve that metric".

The Process

Step 1. Get the proposal

Have the user state, in plain English: what change are you making, what do you expect it to do, and what makes you think so. If any part is missing, ask. If the user cannot articulate why they expect the change to help, the experiment is premature; tell them.

Step 2. Falsifiable hypothesis

Rewrite the proposal as a hypothesis in this exact shape:

"Changing X from A to B will cause metric M to move by at least threshold T within window W, with no worse-than-guardrail G regression on counter-metric C."

If the user cannot fill in M, T, W, G, or C, stop and fill them in together. A hypothesis missing any of these is not falsifiable.

Step 3. Control condition

Decide how the change is attributed. Pick one and document why:

Design	When to use	Watch out for
Randomized A/B (user-level)	Default. Independent users, low cross-contamination.	Network effects, multi-account users, learning effects across sessions.
Randomized A/B (request-level)	When user-level randomization is too noisy and requests are independent.	Same user gets both, may invalidate the comparison.
Switchback	Marketplaces, supply-demand systems, cohort effects.	Time-of-day, day-of-week confounds. Need long enough switches.
Holdout	Roll out to most users, hold out a small group as control over time.	Holdout group decay; harder to power.
Pre-post (no control)	Last resort. Only when randomization is impossible.	Confounded by everything else changing. State the assumption explicitly.
Synthetic offline	Pre-launch, on a fixed eval set.	Does not generalize to live traffic. Use as a gate, not a verdict.

If the user picks pre-post or synthetic-offline by default, push back once. They are usually picked because randomization is "hard" not because it is impossible.

Step 4. Sample size and duration

Estimate a minimum sample size (or run length) needed to detect the threshold T with reasonable power. Even a back-of-envelope number is better than nothing:

• For a binary metric (conversion, click): use a standard A/B sample-size formula. State assumed baseline rate and minimum detectable effect.
• For a continuous metric (latency, score): state assumed standard deviation and minimum detectable effect.
• For an LLM eval set: state how many examples and why that is enough.

If the math says the experiment needs three months and the user wanted to ship next week, surface that now. Better to negotiate at design time than to ship an underpowered test.

Step 5. Kill criterion

Write down, in advance, what would cause the experiment to be stopped early. At minimum:

• Guardrail breach: if counter-metric C degrades worse than threshold G at any time, stop.
• Catastrophic absolute movement: if M or any safety metric drops below an absolute floor, stop.
• Timeout: if the experiment has run W and not reached significance, stop and accept the inconclusive result.

The kill criterion exists to remove judgment from a moment when judgment is compromised (you have sunk cost and want it to work).

Step 6. Pre-register

Output a single page the user can paste into a doc, ticket, or PR description before launch. Once pasted, the experiment is pre-registered. Changes after that need an explicit amendment with a date.

# Experiment pre-registration: [name]

**Date pre-registered:** [today]
**Owner:** [name]

## Change under test
- Variable X: ...
- From A: ...
- To B: ...

## Hypothesis
Changing X from A to B will cause M to move by at least T within W,
with no worse than G regression on C.

## Why we expect this
[1 to 3 sentences. The user's actual reasoning, not motivated reasoning.]

## Design
- Type: [A/B user-level | A/B request-level | switchback | holdout | pre-post | synthetic offline]
- Randomization unit: ...
- Allocation: ...% control, ...% treatment
- Minimum sample size or runtime: ... (assumes baseline ..., MDE ..., power 0.8)

## Metrics
- Primary metric M: ...
- Threshold T: ...
- Counter-metric C: ...
- Guardrail G: ...
- Absolute safety floors: ...

## Kill criteria
- ...
- ...
- ...

## What we will do with each outcome
- Positive (M moves > T, C within G): ship to 100%.
- Negative (M does not move > T, or C breaches G): roll back, write postmortem.
- Inconclusive (after W, no significance): default action is roll back. State why.

## Amendments
[Add dated entries here if the design changes after pre-registration.]

End with a brief recommendation: ship as-pre-registered, or revise (state which fields).

Common failure modes

Failure	Fix
No threshold T (just "improve")	"Improve" is not falsifiable. Pick a number, even a rough one.
No counter-metric C	Every AI change moves something else. Name the most-likely-to-regress metric and gate on it.
Picked pre-post because randomization was "hard"	Push back once. Real obstacles are rare.
Underpowered, shipped anyway	Underpowered tests teach nothing. Either run longer, raise the threshold, or skip.
No kill criterion	Set one. Past-you with no sunk cost is wiser than future-you under deadline.
Changed the hypothesis after seeing results	This is hypothesis hacking. Amendments must be dated and explained.
LLM-as-judge with no human spot-check	LLM judges drift; ground at least 5 to 10 percent of judgments in human review.

When to chain to /research

If the user's hypothesis depends on assumed effect sizes (e.g. "we expect a 10 percent CTR lift from this prompt change"), suggest /research to check for prior published effect sizes on similar AI changes. Most AI experiments overestimate effect size by 3 to 10x.