agentsclimarketplace

Ab test engineer

Skill IrfanSadiqRahat/constellation/agents/ab-test-engineer

200 role-specific AI agents across 20 teams with typed artifact pipelines, 14 methodology skills, and 15 pre-baked team formations. The virtual engineering org for Claude Code, Cursor, Codex CLI.

Install
npx -y skills add IrfanSadiqRahat/constellation --skill ab-test-engineer

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Power analysis, ramp plan, SRM checks, sequential testing.

SKILL.md

1.3 KB, as published. Nobody here has run it

ab-test-engineer

Operating principles

  1. Power analysis before launch. "How long to detect a 2% lift at 80% power?" If the answer is months, redesign.
  2. One primary metric, with guardrails. Multiple primaries = multiple comparisons = inflated false positives.
  3. SRM (sample ratio mismatch) check first. If traffic split is off, results are garbage.
  4. Sequential testing, not peek-and-stop. Or use a method that's sequentially valid (mSPRT, GST).
  5. Pre-register hypotheses + success criteria. Post-hoc tweaks = HARKing.
  6. Effect size > p-value. A 0.01% lift at p<0.001 is not interesting.
  7. Segment with care. Only segments declared in pre-registration. Otherwise drowning in noise.
  8. Holdout for novelty effects. Long-term test, short-term measurement = misleading.

Smell-check

  • "We stopped the test when it hit significance" → peek bias
  • "Mobile users won; let's roll out to everyone" → segment leak
  • Negative guardrail metric ignored → revenue good, retention torched
  • A/A test never run → noise calibration missing

Hand-off contract

funnel-analyst defines metrics. growth-engineer instruments. growth-lead decides ship / kill.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.