agentsclimarketplace
Sign in

Experiment metric audit

Skill sisodiabhumca/agent-skills/skills/experiment-metric-audit

Vendor-neutral skill for auditing experiment metrics definitions for common analytics pitfalls and inconsistencies.From its SKILL.md

Install
npx -y skills add sisodiabhumca/agent-skills --skill experiment-metric-audit

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.3 KB, 247 tokens by cl100k_base, as published. Nobody here has run it

When to invoke

  • You have an A/B test plan with metric definitions captured in JSON.
  • You want to validate that metric formulas, unit of analysis, and guardrails are internally consistent.
  • You need vendor-neutral checks before implementing metrics in any analytics stack.

Inputs needed

  • JSON file describing an experiment and its metrics (primary/secondary/guardrail).

Workflow

  1. Validate schema (experiment name, variants, metrics list).
  2. Check each metric for:
    • missing unit of analysis (user, session, order)
    • missing time window
    • unclear numerator/denominator for ratio metrics
    • guardrail metrics present (e.g., error rate) when risky changes are described
  3. Detect common inconsistencies:
    • metrics mixing units (user-level denominator with event-level numerator)
    • duplicate metric names
  4. Emit a structured audit report with actionable recommendations.

Output format

  • JSON report with findings and a summary score.
  • Human-readable markdown printed to stdout.

Guardrails

  • Do not claim statistical validity; this is a definition audit only.
  • Treat output as a checklist; analyst review required.

Reference code

  • experiment_metric_audit.py

What ships with it: 2 files

7.9 KB alongside SKILL.md, 1 of them executable

Gives 0 of the 12 instructions most audit compliance skills give in 247 tokens

Counted across 937 of the 1,487 authors here whose files we hold, read 2026-08-07

  • Fetch latest guidelines before each reviewin 43 of 937, across 3 files
  • Group findings by severityin 43 of 937
  • Check files against all fetched rulesin 42 of 937, across 2 files
  • Output findings in terse file:line formatin 41 of 937, across 3 files
  • Ask user which files to review if none specifiedin 41 of 937, across 3 files
  • Read specified files or prompt user for filesin 39 of 937, across 1 file
  • Generate the audit reportin 33 of 937, across 30 files
  • Assign a severity to every findingin 25 of 937
  • Run automated accessibility scansin 23 of 937, across 13 files
  • Output a markdown audit reportin 22 of 937
  • Map findings to WCAG criteriain 20 of 937, across 10 files
  • Confirm audit scopein 19 of 937, across 9 files

Said here and by no other author read

  • validate the experiment metric schema
  • check each metric for missing unit of analysis
  • check each metric for missing time windows
  • check each metric for unclear numerators or denominators
  • verify guardrail metrics are present for risky changes
  • detect metrics mixing user-level and event-level units

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.

Keep looking

Skills are one crate of 326,790. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.