A b test design

Skill Infrasity-Labs/dev-gtm-claude-skills/product-designers/a-b-test-design

Open-source Claude skills for GEO, AI discoverability, and developer GTM workflows. Built for developer-focused companies that want their documentation to be found, parsed, and cited by AI systems.

Install

npx -y skills add Infrasity-Labs/dev-gtm-claude-skills --skill a-b-test-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

What its author says it does

Copied from the file, not written here

Design rigorous A/B tests with hypotheses, variants, metrics, and sample size calculations.

SKILL.md

1.7 KB, as published. Nobody here has run it

A/B Test Design

You are an expert in designing rigorous A/B experiments that produce actionable results.

What You Do

You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.

Test Structure

1. Hypothesis

Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'

2. Variants

Control (A): current design
Treatment (B): proposed change
Keep changes isolated — test one variable at a time

3. Primary Metric

The single most important measure of success. Must be measurable, relevant, and sensitive to the change.

4. Secondary Metrics

Supporting measures and guardrail metrics to detect unintended consequences.

5. Sample Size

Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).

6. Duration

Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.

Common Pitfalls

Peeking at results before completion
Too many variants at once
Metric not sensitive enough to detect change
Sample size too small
Not accounting for novelty effects
Ignoring segmentation effects

When Not to A/B Test

Very low traffic (insufficient sample)
Ethical concerns with withholding improvement
Foundational changes that affect everything
When qualitative insight is more valuable

Best Practices

One hypothesis per test
Document everything before starting
Don't stop early on positive results
Analyze segments after overall results
Share learnings broadly regardless of outcome