Evaluate agent quality

Skill m00kk/agent-skills-playbook/skills/evaluate-agent-quality

Creates regression suites and golden-path evaluations for AI agents: task success, tool correctness, citation faithfulness, and safety refusals. Use when testing agents, CI eval gates, or benchmarking prompt or tool changes.From its SKILL.md

Install

npx -y skills add m00kk/agent-skills-playbook --skill evaluate-agent-quality

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.6 KB, 360 tokens by cl100k_base, as published. Nobody here has run it

Evaluate Agent Quality

Eval types

Type	Measures
Task success	Did the agent complete the user goal?
Tool accuracy	Right tool, right args, right order
RAG faithfulness	Answer supported by retrieved chunks
Safety	Refusal on harmful or out-of-scope requests
Regression	No worse than baseline on golden set

Golden dataset format

{
  "id": "rag-001",
  "input": "What is our refund policy for EU customers?",
  "expected_tools": ["search_knowledge"],
  "expected_citations": ["policy-eu-2024.md"],
  "rubrics": ["mentions 14-day window", "cites EU doc"]
}

Store under evals/golden/ in the project (not in skill repo).

Workflow

- [ ] Collect 20–50 real user tasks (anonymized)
- [ ] Label expected outcome + forbidden behaviors
- [ ] Automate runner: invoke agent, capture trace
- [ ] Score with rules + optional LLM-as-judge
- [ ] Block deploy if success rate drops >5% vs baseline

LLM-as-judge cautions

Use separate model or fixed rubric prompt
Judges fail on subjective tone — prefer binary checks
Always keep deterministic checks (tool called, file exists)

CI integration

Run subset (smoke 5 cases) on every PR
Full suite nightly
See github-ci-agents for Actions wiring

Metrics to track over time

pass@1, pass@3
average steps per task
cost per successful task

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.