Ab testing

A/B Testing

Experiment Design

Hypothesis Template

We believe that [change] will cause [metric] to [increase/decrease]
because [reasoning based on data/insight].
We'll know this is true when we see [statistical significance at X% confidence].

Example:

"We believe that adding customer logos above the signup CTA will increase trial signups by ≥10% because users cite trust as their #1 objection in exit surveys. We'll know this is true when we see 95% confidence with ≥500 conversions per variant."

Variable Isolation Rule

Test ONE change at a time per experiment. Multi-variable tests need multivariate setup (MVT) with much larger sample sizes.

Control vs Treatment

	Control	Treatment
Definition	Current experience	Modified experience
Traffic split	50% (typical)	50% (typical)
Changes	None	One specific change

Statistical Significance

Key Concepts

• Significance level (α): Probability of false positive. Standard: α = 0.05 (5%)
• Confidence: 1 - α = 95% confidence
• Power (1-β): Probability of detecting a real effect. Target: 80%
• p-value: Probability that result is due to chance. Need p < 0.05 to call a winner
• MDE (Minimum Detectable Effect): Smallest improvement worth detecting

Sample Size Calculator (Python)

import math

def sample_size_per_variant(baseline_cr, mde_relative, alpha=0.05, power=0.80):
    """
    baseline_cr: current conversion rate (e.g. 0.05 for 5%)
    mde_relative: minimum detectable effect as relative change (e.g. 0.10 for 10% lift)
    """
    p1 = baseline_cr
    p2 = baseline_cr * (1 + mde_relative)
    
    z_alpha = 1.96  # for 95% confidence (two-tailed)
    z_beta = 0.842  # for 80% power
    
    p_avg = (p1 + p2) / 2
    n = ((z_alpha * math.sqrt(2 * p_avg * (1 - p_avg)) + 
          z_beta * math.sqrt(p1 * (1-p1) + p2 * (1-p2))) ** 2) / (p2 - p1) ** 2
    
    return math.ceil(n)

# Example: 5% baseline CR, want to detect 10% relative lift
n = sample_size_per_variant(0.05, 0.10)
print(f"Need {n} visitors per variant ({n*2} total)")
# Output: ~3,842 per variant (7,684 total)

Quick Sample Size Reference Table

Baseline CR	Detect 5% lift	Detect 10% lift	Detect 20% lift
2%	~38,000/variant	~10,000/variant	~2,700/variant
5%	~15,000/variant	~3,900/variant	~1,000/variant
10%	~7,400/variant	~1,900/variant	~490/variant
20%	~3,500/variant	~900/variant	~230/variant

Duration Calculation

Required duration (days) = Sample size per variant / (Daily traffic × traffic_split%)

Example:
  Need: 4,000 per variant
  Daily traffic: 1,000 visitors
  Traffic split: 50%
  Duration = 4,000 / (1,000 × 0.5) = 8 days → round up to 14 days (full 2 weeks)

Always run for at least 2 full business cycles (typically 14 days) to account for day-of-week effects.

Test Prioritization

ICE Framework

Criterion	Question	Score 1–10
Impact	If this wins, how much does the metric move?
Confidence	How confident are we the change will work?
Ease	How easy is it to implement and run?

ICE Score = (Impact + Confidence + Ease) / 3

PIE Framework (alternative)

Criterion	Definition
Potential	How much can this page be improved? (based on analytics)
Importance	How much traffic / revenue does this page generate?
Ease	How easy is implementation?

Test Backlog Template

Test	Hypothesis	ICE	Metric	Sample needed	Status
Logos above CTA	Trust → more signups	8.3	Trial CR	8,000	Running
Shorter form	Less friction	7.7	Lead CR	15,000	Queued
Video testimonial	Social proof	6.0	Demo CR	4,000	Backlog

Common Mistakes to Avoid

Mistake	Why It's Wrong	Fix
Stopping test early when winning	Peeking problem inflates false positives	Pre-commit to sample size; don't check daily
Running overlapping tests	Interaction effects confound results	Serialize tests on same page/flow
Testing during unusual periods	Sales events, holidays skew results	Avoid peak seasons unless testing those specifically
Low traffic test	Underpowered — can't reach significance	Calculate sample size first
Multiple metrics	Higher chance of false positive	Pre-specify ONE primary metric
Not segmenting results	Winning average hides losing segments	Always check results by device, source, user segment

Multivariate Testing (MVT)

Use only when you need to test COMBINATIONS of changes:

MVT vs A/B

	A/B	MVT
Variables	1	2+ simultaneously
Variants	2	4–16+
Traffic needed	Low	10–50× more
Insight	Which version wins	Which combination wins AND interaction effects

Full Factorial MVT Example

Testing: 2 headlines × 2 images = 4 variants:

• A: Headline 1 + Image 1
• B: Headline 1 + Image 2
• C: Headline 2 + Image 1
• D: Headline 2 + Image 2

Declaring a Winner

Decision Criteria

1. Reached pre-specified sample size ✓
2. p-value < 0.05 ✓
3. Ran for minimum 2 business cycles ✓
4. No anomalies in data (traffic spike, tracking issue) ✓
5. Effect consistent across major segments (mobile/desktop, organic/paid) ✓

When to Ship Without Significance

• Revenue impact is very large (directionally positive, even if not significant)
• Test is directionally consistent across all segments
• Cost of not shipping exceeds cost of being wrong

Result Documentation

## Test: [Name]
- Hypothesis: [stated hypothesis]
- Start: [date] | End: [date]
- Traffic: [N control] / [N treatment]
- Primary metric: [metric]
  - Control: [value] | Treatment: [value] | Δ: [%] | p-value: [p]
- Result: Winner / No significant difference / Inconclusive
- Decision: Ship / Rollback / Extend
- Learnings: [1–3 sentences on what this tells us]
- Next test: [what to test next based on this result]