agentsclimarketplace

Ab testing

Skill supportersimulator/3-surgeons/skills/ab-testing

Full A/B test lifecycle with safety constraints and human veto windowsFrom its SKILL.md

Install
npx -y skills add supportersimulator/3-surgeons --skill ab-testing

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

5.7 KB, as published. Nobody here has run it

A/B Testing

When to Propose an A/B Test

A/B testing is for controlled experiments where you have two plausible approaches and want empirical evidence for which performs better. Use it when:

  • Optimizing prompts -- "Does system prompt A produce better results than system prompt B?"
  • Comparing approaches -- "Is caching strategy X faster than strategy Y?"
  • Parameter tuning -- "Does temperature=0.3 produce more consistent output than temperature=0.7?"
  • Validating improvements -- "Does the refactored code actually perform better than the original?"

Do NOT use for: safety-critical parameters, authentication settings, cost/rate limits, or anything that could compromise system integrity. These are enforced as forbidden parameters (see Safety Constraints below).

The Full Lifecycle

An A/B test moves through these states:

PROPOSED -> GRACE_PERIOD -> ACTIVE -> MONITORING -> CONCLUDED
                |                                       |
                v                                       v
             VETOED                                 REVERTED
                                                (auto: cost/time exceeded)

Step 1: Propose

3s ab-propose "temperature" "0.3" "0.7" "Lower temperature produces more consistent classification output"

Or via MCP:

ab_propose(
    param="temperature",
    variant_a="0.3",
    variant_b="0.7",
    hypothesis="Lower temperature produces more consistent classification output"
)

This creates the test in PROPOSED status and returns a test ID (UUID). The test is not yet running.

Step 2: Grace Period (Human Veto Window)

ab_start(test_id="<uuid>")

Moves the test to GRACE_PERIOD. This is a deliberate pause (default 30 minutes) that allows a human to review the proposal and veto it before any resources are spent. The grace period duration is informational -- enforcement is the caller's responsibility.

Why a grace period? Autonomous A/B testing without human oversight can waste budget on ill-conceived experiments. The grace period is a safety valve.

Step 3: Activate

ab_start -> (wait for grace period) -> activate(test_id="<uuid>")

Activation requires the test to be in GRACE_PERIOD. If the test was vetoed during the grace period, activation will fail. Once active, the experiment is running.

Step 4: Measure

ab_measure(test_id="<uuid>", metric_a=0.85, metric_b=0.92)

Records a measurement pair. Returns a comparison:

{
    "test_id": "<uuid>",
    "metric_a": 0.85,
    "metric_b": 0.92,
    "delta": 0.07,
    "variant_b_better": true
}

You can record multiple measurements over the test duration.

Step 5: Conclude

ab_conclude(test_id="<uuid>", verdict="Variant B (0.7) produced 7% higher accuracy; adopting as default")

Records the verdict in the evidence store for future reference. The test moves to CONCLUDED status.

Alternative: Veto

At any point before conclusion, a test can be vetoed:

# Not yet exposed via CLI -- use MCP or programmatic API

Vetoed tests are marked with the reason and move to terminal VETOED status.

Safety Constraints

Forbidden Parameters

The following parameters cannot be A/B tested. Attempting to propose a test on them raises a ValueError:

ParameterReason
safety_gateCould disable safety checks
corrigibilityCould weaken safety invariants
evidence_retentionCould erase institutional memory
cost_limitCould remove spending guardrails
rate_limitCould allow resource exhaustion

Additionally, any parameter containing "security" or "auth" (case-insensitive) is automatically forbidden.

Budget Caps

Each test has two automatic limits:

LimitDefaultWhat Happens
max_cost_usd$2.00 (from budgets.autonomous_ab_usd)Test auto-reverts to REVERTED status
max_duration_hours48 hoursTest auto-reverts to REVERTED status

These limits are checked via check_safety(test_id). When exceeded, the test is automatically moved to REVERTED with a verdict explaining why.

Daily Budget

The overall daily external spend cap (budgets.daily_external_usd, default $5) applies across all A/B tests and other external LLM calls.

Querying Active Tests

# Programmatic -- returns all non-terminal tests
engine.get_active_tests()

# Check a specific test
engine.get_test(test_id)

Active tests are those not in CONCLUDED, VETOED, or REVERTED status.

Evidence Integration

When a test is concluded, the result is automatically recorded in the evidence store:

  • ab_results table: experiment ID, param, variants, verdict
  • cost_tracking table: total cost attributed to the test
  • Searchable: future evidence queries can surface past A/B results

This means if you later ask "What temperature setting works best?", the evidence store can return past A/B test results on that parameter.

Example: Full Workflow

# 1. Propose
3s ab-propose "prompt_style" "concise" "detailed" "Concise prompts reduce token usage without quality loss"
# Output: A/B test proposed: a1b2c3d4-...

# 2. Start grace period (via MCP tool)
# ab_start(test_id="a1b2c3d4-...")
# Wait 30 minutes for human review

# 3. Activate (via MCP tool)
# activate(test_id="a1b2c3d4-...")

# 4. Run both variants, measure results
# ab_measure(test_id="a1b2c3d4-...", metric_a=0.82, metric_b=0.79)
# ab_measure(test_id="a1b2c3d4-...", metric_a=0.85, metric_b=0.80)

# 5. Conclude with verdict
# ab_conclude(test_id="a1b2c3d4-...", verdict="Variant A (concise) 4% better on avg; adopting")

Keep looking

Skills are one crate of 609,689. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.