Analyze failures
Agent evals on autopilot: find quality bugs in your AI agent, ship a targeted fix, and prove it on a held-out set. Zero-dependency Agent Skill + CLI.
npx -y skills add avnath13/evalpilot --skill analyze-failuresAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- 20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Read grading verdicts, cluster failures by ROOT-CAUSE failure mode (open→axial coding), and rank by frequency × severity into a report. Use when the user wants to understand WHY their agent fails or which problem to fix first. Do NOT use to produce verdicts (use grade) or to apply a fix (use optimize). Let categories emerge from the data, do not start from a pre-defined failure list.
SKILL.md
1.8 KB, as published. Nobody here has run it
analyze-failures: verdicts → clusters + report
Outputs: report.md (human) and .evalpilot/clusters.json (machine, feeds optimize).
Procedure (open → axial coding)
- Read
runs/<runId>/verdicts.jsonl. Keep only failing verdicts (plus low scores). - Root-cause first. Annotate the FIRST thing that went wrong in each trace, errors cascade, so downstream symptoms vanish once the root cause is fixed.
- Let categories emerge (open coding). Do NOT start from a pre-defined failure list. Split when root causes differ (fabricating external facts vs fabricating user intent); merge when one cause has different surface manifestations. Target 5-10 distinct, actionable categories, each pointing to a specific fix.
- Rank by
frequency × severity. Tag each cluster with afix_kind(code|prompt|tool|judge) so optimize knows the lever. - Write
report.md(ranked list first, then per cluster: count/%, severity, 2-3 examples with evidence quote, one-line root-cause hypothesis) andclusters.json[{name, count, severity, rank, fix_kind, example_case_ids, hypothesis}].
Anti-patterns
- Brainstorming failure categories before reading traces.
- Treating this as one-time, re-run after prompt rewrites, model swaps, or incidents.
- 40 singleton clusters. A hypothesis is a hypothesis, optimize will test it.