Agent evaluation
Skill payals/ai-skills-engine/dot_cursor/skills/ai-research/agent-evaluation
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.From its SKILL.md
npx -y skills add payals/ai-skills-engine --skill agent-evaluationAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
2.0 KB, 345 tokens by cl100k_base, as published. Nobody here has run it
Agent Evaluation
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.
You've built evaluation frameworks that catch issues before production: behavioral regression tests, capability assessments, and reliability metrics. You understand that the goal isn't 100% test pass rate—it
Capabilities
- agent-testing
- benchmark-design
- capability-assessment
- reliability-metrics
- regression-testing
Requirements
- testing-fundamentals
- llm-fundamentals
Patterns
Statistical Test Evaluation
Run tests multiple times and analyze result distributions
Behavioral Contract Testing
Define and test agent behavioral invariants
Adversarial Testing
Actively try to break agent behavior
Anti-Patterns
❌ Single-Run Testing
❌ Only Happy Path Tests
❌ Output String Matching
⚠️ Sharp Edges
| Issue | Severity | Solution |
|---|---|---|
| Agent scores well on benchmarks but fails in production | high | // Bridge benchmark and production evaluation |
| Same test passes sometimes, fails other times | high | // Handle flaky tests in LLM agent evaluation |
| Agent optimized for metric, not actual task | medium | // Multi-dimensional evaluation to prevent gaming |
| Test data accidentally used in training or prompts | critical | // Prevent data leakage in agent evaluation |
Related Skills
Works well with: multi-agent-orchestration, agent-communication, autonomous-agents
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.
Gives 0 of the 12 instructions most context ai engineering skills give in 345 tokens
Counted across 1,328 of the 2,349 authors here whose files we hold, read 2026-09-06
- Dispatch a fresh subagent for each taskin 76 of 1328, across 59 files
- Perform spec compliance review before code quality reviewin 44 of 1328, across 34 files
- Dispatch a final code reviewer after all tasksin 38 of 1328, across 26 files
- Answer subagent questions before allowing implementationin 36 of 1328, across 26 files
- Use the least powerful model capable of the taskin 33 of 1328, across 26 files
- Create a TodoWrite list for all tasksin 32 of 1328, across 22 files
- Perform a task review after each implementationin 31 of 1328, across 24 files
- Extract all tasks and context from the planin 29 of 1328, across 20 files
- Provide full task text to subagentsin 28 of 1328, across 20 files
- Use git worktrees for isolated workspacesin 25 of 1328, across 20 files
- Specify the model explicitly when dispatching a subagentin 23 of 1328, across 18 files
- Execute all tasks from the plan without stoppingin 21 of 1328, across 16 files
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.