Statistical analysis

Statistical Analysis

Get the statistics of a question right — the test, its assumptions, the uncertainty, and the traps — so a decision rests on evidence rather than on a lone p-value. The deliverable is a decision-ready answer: effect size, uncertainty, assumptions, caveats.

Workflow

1. Frame the question as a hypothesis

State the null and alternative in plain words before touching data ("the new flow's conversion equals the old flow's"). Identify the unit of analysis (user? session? order?) — mismatching it with the randomization unit silently breaks independence.

2. Design before data (when the experiment hasn't run yet)

• Choose the primary metric and pre-register it — one primary, the rest explicitly secondary/exploratory.
• Compute sample size from power: minimum detectable effect worth acting on, α (typically 0.05), power (typically 0.8). Run until n is reached — don't stop early on a good-looking result.
• Randomize at the unit you'll analyze; check the split is balanced on key covariates.

3. Choose the test and state its assumptions

Match the test to the data (two proportions → two-proportion z-test / χ²; means → t-test or its nonparametric fallback; counts → Poisson/χ²; paired data → paired tests). Always state the assumptions and check the ones you can: independence of observations (the usual casualty — repeated measures per user, network effects, shared fixtures), adequate counts for normal approximations (rule of thumb: ≥10 successes and failures per arm), variance assumptions for t-tests. If an assumption fails, switch tests rather than pretending.

4. Report effect size and interval, not a verdict

Work the actual numbers. Report the effect (absolute and relative difference) with a confidence interval, and the p-value as supporting detail — never a bare "significant/not significant". The CI is what the decision needs: a CI of [+0.1%, +1.7%] and a CI of [+0.8%, +1.0%] are both "significant" and mean very different things.

5. Run the pitfall gate

Before reporting anything, check:

• Peeking / optional stopping: was n fixed in advance? Repeatedly testing as data arrives inflates false positives badly. If peeking happened, say so and treat the result as suggestive (or use sequential methods designed for it).
• Multiple comparisons: k tests at α=0.05 ≈ k/20 false positives expected. Correct (Holm/Bonferroni for strict control, Benjamini-Hochberg FDR when many metrics), and label post-hoc findings as exploratory — they are hypotheses for the next experiment, not wins from this one.
• Subgroup fishing: subgroup effects found after the fact need the same exploratory label and a confirmatory follow-up.
• Survivorship / selection: does the analyzed sample still represent the population the decision is about?

6. Translate to the decision

Distinguish statistical from practical significance: a real +0.05% lift may not pay for the complexity; a non-significant result with a wide CI is "we don't know yet", not "no effect". Give the ship/don't-ship framing with the caveats attached, and say what additional data would resolve remaining uncertainty.

Principles

• KISS: a well-checked z-test beats a fancy model with unexamined assumptions.
• Pre-register or label: every metric is either declared up front or reported as exploratory — no third category.
• Uncertainty is the product: any answer without an interval is opinion with extra steps.

Cross-skill boundaries

• Live A/B testing of an AI/LLM app — defining answer quality, LLM-as-judge, human feedback loops, eval gates → ai-evaluation (this skill still owns the underlying experiment statistics if asked).
• Generating the hypotheses to test → exploratory-data-analysis.
• Tracking and comparing model-training runs → ml-experiment-tracking.
• Defining what business metric to move → metrics-and-okrs.