Ab test design

Skill a5c-ai/babysitter/library/specializations/product-management/skills/ab-test-design

Babysitter enforces obedience on agentic workforces and enables them to manage extremely complex tasks and workflows through deterministic, hallucination-free self-orchestration

Install

npx -y skills add a5c-ai/babysitter --skill ab-test-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

What its author says it does

Copied from the file, not written here

Statistical experiment design and analysis capabilities for product experimentation

SKILL.md

4.5 KB, as published. Nobody here has run it

A/B Test Design Skill

Overview

Specialized skill for statistical experiment design and analysis capabilities. Enables product teams to design rigorous experiments, calculate sample sizes, and interpret results with statistical confidence.

Capabilities

Experiment Design

Calculate required sample sizes for experiments
Design experiment variants and hypotheses
Define success metrics and guardrail metrics
Create experiment documentation templates
Design multi-variant tests (A/B/n)
Plan sequential and Bayesian experiments

Statistical Analysis

Validate statistical significance of results
Calculate practical significance and effect sizes
Detect interaction effects and segments
Perform power analysis
Calculate confidence intervals
Handle multiple comparison corrections

Decision Support

Recommend ship/iterate/kill decisions
Identify segment-specific impacts
Assess long-term vs short-term effects
Generate experiment reports
Track experiment velocity metrics

Target Processes

This skill integrates with the following processes:

product-market-fit.js - Validation experiments for PMF hypotheses
conversion-funnel-analysis.js - Funnel optimization experiments
beta-program.js - A/B testing during beta phases

Input Schema

{
  "type": "object",
  "properties": {
    "experimentType": {
      "type": "string",
      "enum": ["ab", "multivariate", "sequential", "bandit"],
      "description": "Type of experiment to design"
    },
    "hypothesis": {
      "type": "string",
      "description": "Hypothesis to test"
    },
    "primaryMetric": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "baseline": { "type": "number" },
        "mde": { "type": "number", "description": "Minimum detectable effect" }
      }
    },
    "guardrailMetrics": {
      "type": "array",
      "items": { "type": "string" },
      "description": "Metrics that should not regress"
    },
    "trafficAllocation": {
      "type": "number",
      "description": "Percentage of traffic for experiment"
    },
    "confidenceLevel": {
      "type": "number",
      "default": 0.95,
      "description": "Statistical confidence level"
    }
  },
  "required": ["experimentType", "hypothesis", "primaryMetric"]
}

Output Schema

{
  "type": "object",
  "properties": {
    "experimentPlan": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "hypothesis": { "type": "string" },
        "variants": { "type": "array", "items": { "type": "object" } },
        "sampleSize": { "type": "number" },
        "duration": { "type": "string" },
        "metrics": { "type": "object" }
      }
    },
    "powerAnalysis": {
      "type": "object",
      "properties": {
        "requiredSampleSize": { "type": "number" },
        "estimatedDuration": { "type": "string" },
        "power": { "type": "number" }
      }
    },
    "implementation": {
      "type": "object",
      "properties": {
        "trackingEvents": { "type": "array", "items": { "type": "string" } },
        "segmentation": { "type": "array", "items": { "type": "string" } },
        "rolloutPlan": { "type": "string" }
      }
    },
    "analysisFramework": {
      "type": "object",
      "properties": {
        "primaryAnalysis": { "type": "string" },
        "secondaryAnalyses": { "type": "array", "items": { "type": "string" } },
        "decisionCriteria": { "type": "object" }
      }
    }
  }
}

Usage Example

const experimentDesign = await executeSkill('ab-test-design', {
  experimentType: 'ab',
  hypothesis: 'Adding social proof to pricing page increases conversion by 10%',
  primaryMetric: {
    name: 'pricing_page_conversion',
    baseline: 0.05,
    mde: 0.10
  },
  guardrailMetrics: ['revenue_per_visitor', 'bounce_rate'],
  trafficAllocation: 50,
  confidenceLevel: 0.95
});

Dependencies

Statistical libraries for power analysis
Experimentation platform integrations (Optimizely, LaunchDarkly, etc.)