Eval harness design
Skill jukrap/ai-agent-playbook/skills/delivery/eval-harness-design
Use when agent evals need graders, pass@k, regression gates, or reliability evidence.From its SKILL.md
npx -y skills add jukrap/ai-agent-playbook --skill eval-harness-designAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
1.0 KB, 185 tokens by cl100k_base, as published. Nobody here has run it
Eval Harness Design
Use this as the primary delivery skill for eval-driven changes to agent workflows, prompts, MCP tools, skills, or automation surfaces.
Workflow
- Define the target behavior, risk class, baseline, failure modes, and release decision before changing the harness.
- Choose deterministic code, schema, or rule graders before model or human graders.
- Separate capability evals from regression evals, then set pass@k, pass^k, cost, latency, and repeatability expectations.
- Store eval definitions and run reports as runtime evidence until reviewed and promoted.
Reference
Read references/eval-artifact-contract.md for eval definition, run report, evidence envelope, and storage boundaries.
Read references/grader-and-metric-rubric.md for grader choice, metric thresholds, pass@k usage, and anti-overfitting checks.
What ships with it: 2 files
5.7 KB alongside SKILL.md