Ab test analysis

Skill shinzoxD/knackbox/skills/data/ab-test-analysis

Curated, auditable, benchmark-ready Agent Skills library for Claude Code, Codex, OpenCode, Cursor, and more.

Install

npx -y skills add shinzoxD/knackbox --skill ab-test-analysis

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

27 days oldThe repository was created 27 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Analyze A/B and experiment results with correct metrics, uncertainty, and caveats. Use whenever the user shares experiment data, asks if a variant won, or needs help interpreting lift, p-values, confidence intervals, CUPED, or sample ratio mismatch.

The file declares its own license as Apache-2.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

2.0 KB, as published. Nobody here has run it

A/B Test Analysis

Experiments need pre-registered metrics, healthy assignment, and honest uncertainty. Do not crown winners from noisy or peeked data without caveats.

Workflow

Confirm hypothesis, primary metric, guardrails, unit of randomization, and duration.
Check data quality: sample ratio mismatch, bots, nulls, novelty effects.
Compute lift with uncertainty (CI); avoid p-hacking language.
Segment carefully; treat segments as exploratory unless pre-registered.
State decision: ship / hold / iterate, with risks.

Output format

## Experiment read: <name>

**Primary metric:** …
**Result:** …
**Decision recommendation:** ship | hold | iterate

### Health checks
- SRM: …
- Coverage / exclusions: …

### Estimates
| Variant | N | Metric | Lift | 95% CI |

### Guardrails
…

### Caveats
…

Rules

Never invent statistical significance; if numbers are missing, say what to compute.
Multiple metrics inflate false positives — say so when user looks at many.
Distinguish practical significance from statistical significance.
SRM failure means pause interpretation of outcomes.
Peeking early without sequential design → caveat strongly.
Correlation of segments is not automatic heterogeneity proof.

Edge cases

Switchback / network effects: note interference when users interact.
Low power: recommend more runtime or larger effect — do not fake certainty.
Non-inferiority: different decision frame than superiority tests.