Experiment analysis

Experiment analysis

Designing an experiment well is half the job; reading its results without fooling yourself is the other half. The analysis traps are famous and costly: declaring a win from noise, peeking early, reading segments that were not pre-planned, and Simpson's paradox. Honest analysis is what makes the experiment worth running.

Method

1. Check the experiment's health before the result. Sample-ratio mismatch (a 50/50 split arriving 48/52 means broken assignment and invalidates everything), instrumentation working, and the test having run its planned duration. A result from a broken experiment is worse than no result (see ab-test-design, data-quality-checks).
2. Report the effect with a confidence interval, not just significance. "+2.1% conversion, 95% CI [0.4%, 3.8%]" tells you the size and the uncertainty; "p < 0.05" hides both. A significant result with an interval spanning trivial-to-large is a weak conclusion. Lead with the effect size and its range (see statistical-inference).
3. Respect the pre-registered analysis. Analyze the primary metric you named before the test, at the planned time. Do not peek and stop when significant (repeated looks manufacture false positives), and do not hunt through metrics and segments for something that "worked" after a flat primary. Post-hoc findings are hypotheses for the next test, not conclusions.
4. Check the guardrails before celebrating. A lift in the primary metric that tanks a guardrail (a checkout-conversion win that spikes refunds, an engagement win that burns opt-outs) is often a net loss. Read the guardrails first (see ab-test-design, product-metrics).
5. Watch for Simpson's paradox in segments. An effect can reverse direction between the aggregate and every subgroup (or vice versa) when groups are unbalanced. Pre-planned segment analysis is informative; unplanned segment-slicing until something is significant is fishing. When aggregate and segments disagree, understand why before trusting either (see correlation-causation).
6. Interpret a flat result as a real result. "No detectable effect" is a finding: the change did not move the metric, which updates the roadmap. Do not torture the data until a subgroup shows a win, and do not read non-significance as proof of no effect (it may be underpowered; see statistical-inference).

Boundaries

• Analysis is only as valid as the design: randomization, sample size, and pre-registration set before the test are what make the analysis trustworthy (see ab-test-design, experiment-design-basics). Good analysis cannot rescue a broken design.
• Statistical significance measures signal versus noise, not business value or long-term effect; a significant short-term win can hide long-term harm that only a holdout reveals (see product-metrics' Goodhart warning).
• Some situations cannot support a clean experiment (low traffic, network effects, big redesigns); there, honest observational analysis with stated caveats beats a fake A/B test (see correlation-causation, canary-analysis).