Agent evaluation
Skill charlieviettq/awesome-agent-skill/.cursor/skills/ai-agent-systems/agent-evaluation
Curated skill pack for LLM agents in engineer and science workflow (Cursor & Claude ready).
npx -y skills add charlieviettq/awesome-agent-skill --skill agent-evaluationAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 22 stars22 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Evaluate LLM agents and tool-using workflows—task success, tool accuracy, latency/cost, safety, and regression suites. Use when shipping agent features, comparing prompts/models, or debugging agent failures. Triggers: "agent eval", "benchmark agent", "tool accuracy", "agent regression".
SKILL.md
1.7 KB, as published. Nobody here has run it
Agent evaluation
What to measure
| Dimension | Examples |
|---|---|
| Task success | End state matches spec (binary or rubric) |
| Tool use | Correct tool, valid args, no spurious calls |
| Safety | No policy violations, no secret leakage |
| Efficiency | Tokens, latency, tool call count |
| Stability | Same input -> consistent outcome across runs |
Workflow
- Define tasks — realistic user intents with clear pass/fail or scored rubric.
- Build dataset — golden set + edge cases (errors, ambiguous input, empty context).
- Run baseline — fixed model/settings; log traces (inputs, tools, outputs).
- Score — automated checks first; human review for ambiguous cases.
- Compare — A/B prompts, models, or tool schemas; report deltas with confidence notes.
- Gate — block release on regression in must-pass tasks.
Automated checks
- Schema validation on tool arguments.
- Assert final answer contains required fields or avoids forbidden content.
- Snapshot tests for deterministic sub-steps where possible.
Human rubric (when needed)
Score 1-5 on: correctness, completeness, tone, safety. Document disagreements.
Anti-patterns
- Eval only on cherry-picked happy paths.
- Changing task and model simultaneously without isolation.
- No trace logs when debugging tool failures.
Output
Summary table: variant | success rate | avg tools | avg latency | notes.