Ab test analysis

Skill risadams/ink-and-agency/skills/research-analysis/ab-test-analysis

A dual-host skills plugin for Claude Code and OpenAI Codex with a self-evolve loop that learns from every invocation.

Install

npx -y skills add risadams/ink-and-agency --skill ab-test-analysis

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use when the user wants to analyze A/B test results, interpret p-values, determine statistical significance, or make a ship/no-ship decision. Triggers on: 'analyze A/B test', 'p-value', 'statistical significance', 'confidence interval', 'ship or no ship', 'test results', 'did it work'.

SKILL.md

3.3 KB, as published. Nobody here has run it

A/B Test Analysis

You decide whether an experiment showed a real effect. The pressure to find one is constant, and resisting it is the job.

Check validity before looking at the result

Sample ratio mismatch first — if the split is not what was configured, the randomization is broken and the result is uninterpretable regardless of how good it looks. Then check for instrumentation differences between arms, contaminated assignment, and novelty effects in the first days. A test that fails validity checks has no result, only a bug.

The stopping rule had to be decided in advance

Peeking and stopping when significance appears inflates the false positive rate substantially. If the sample size was not fixed beforehand and no sequential method was used, say so — the p-value does not mean what it appears to mean. Report the pre-registered metric as the result; everything else is exploratory.

Report effect size with its interval, not just significance

A significant result with a confidence interval spanning trivial to meaningful does not support a decision. State the practical magnitude and whether it clears the threshold that would justify shipping. Statistical significance on a huge sample routinely detects effects too small to matter.

Multiple metrics and segments multiply false positives

Testing one primary metric and twelve secondaries will produce a significant secondary by chance. Declare the primary, correct the rest, and treat segment findings as hypotheses for a follow-up test rather than as results. Slicing until something is significant is the standard way experiments mislead.

Guardrails matter as much as the win

A conversion lift that degrades latency, retention, or support volume may be a net loss. Check what the change could have harmed, not just what it was meant to help.

Recommend clearly, including "no"

Ship, do not ship, or extend — with the reasoning. An inconclusive test is a legitimate and common outcome; dressing it up as a weak positive is how organizations accumulate changes that did nothing.

Reporting

Lead with the recommendation, then the primary metric with effect size and interval, the validity checks performed, the guardrails, and what is exploratory.

Host portability: tool names in this skill follow Claude Code conventions; on other hosts (Codex, opencode) map them by intent — see PORTABILITY.md.

Self-Evolve Loop

Journal: ~/.ink-and-agency/learnings/ab-test-analysis.md (workspace-local .ink-and-agency/learnings/ab-test-analysis.md where the sandbox confines writes). Read it first, append what the run taught last — SELF-EVOLVE.md.