Eval audit
Agent evals on autopilot: find quality bugs in your AI agent, ship a targeted fix, and prove it on a held-out set. Zero-dependency Agent Skill + CLI.
npx -y skills add avnath13/evalpilot --skill eval-auditAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- 20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Diagnose an EXISTING eval pipeline and produce findings ranked by business impact. Use when the user inherited or already has evals (traces, judges, dashboards, labeled data) and wants to know what's wrong or what to fix first. Do NOT use to build evals from scratch (use prepare-data / user-simulator) or to grade (use grade), this only audits.
SKILL.md
1.9 KB, as published. Nobody here has run it
eval-audit: diagnose an existing eval pipeline
Router skill. Gather artifacts, run checks across six areas, emit findings ordered by business impact, then route to the build skills to remediate.
Six diagnostic areas (with red flags)
- Error analysis, did systematic failure-mode ID happen? Red flag: generic labels borrowed from papers instead of categories observed in THIS app's traces.
- Evaluator design, binary Pass/Fail (not Likert); one judge per failure mode (not holistic); code checks for objective criteria; similarity metrics NOT used as primary evaluators.
- Judge validation, measured vs human labels with TPR/TNR (not raw accuracy);
proper train/dev/test split; no few-shot leakage. → route to
validate-evaluator/calibrate. - Human review, domain experts involved; reviewers see the FULL trace, not just
outputs; data rendered human-readable. → route to
review-interface. - Labeled data, ~100 traces for error-analysis saturation; ~50 Pass / ~50 Fail for validation; strategic sampling when sparse.
- Pipeline hygiene, error analysis re-run after changes; judges periodically re-validated on fresh labels.
Procedure
- Inventory artifacts (traces, configs, judge prompts, labels, dashboards).
- Run each area's checks; record red flags with evidence.
- Emit findings ranked by business impact, each with the sibling skill that fixes it.
Anti-patterns
- Auditing without reading real traces first.
- Trusting judge scores whose TPR/TNR were never measured.