agentsclimarketplace
Sign in

Experiment metric audit

Skill sisodiabhumca/agent-skills/skills/experiment-metric-audit

Vendor-neutral skill for auditing experiment metrics definitions for common analytics pitfalls and inconsistencies.From its SKILL.md

Install
npx -y skills add sisodiabhumca/agent-skills --skill experiment-metric-audit

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.3 KB, 247 tokens by cl100k_base, as published. Nobody here has run it

When to invoke

  • You have an A/B test plan with metric definitions captured in JSON.
  • You want to validate that metric formulas, unit of analysis, and guardrails are internally consistent.
  • You need vendor-neutral checks before implementing metrics in any analytics stack.

Inputs needed

  • JSON file describing an experiment and its metrics (primary/secondary/guardrail).

Workflow

  1. Validate schema (experiment name, variants, metrics list).
  2. Check each metric for:
    • missing unit of analysis (user, session, order)
    • missing time window
    • unclear numerator/denominator for ratio metrics
    • guardrail metrics present (e.g., error rate) when risky changes are described
  3. Detect common inconsistencies:
    • metrics mixing units (user-level denominator with event-level numerator)
    • duplicate metric names
  4. Emit a structured audit report with actionable recommendations.

Output format

  • JSON report with findings and a summary score.
  • Human-readable markdown printed to stdout.

Guardrails

  • Do not claim statistical validity; this is a definition audit only.
  • Treat output as a checklist; analyst review required.

Reference code

  • experiment_metric_audit.py

What ships with it: 2 files

7.9 KB alongside SKILL.md, 1 of them executable

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.