Eval harness design
Skill jukrap/ai-agent-playbook/skills/delivery/eval-harness-design
Reusable AI agent skills, project templates, and guardrails for safer software maintenance and delivery.
npx -y skills add jukrap/ai-agent-playbook --skill eval-harness-designAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Use when agent evals need graders, pass@k, regression gates, or reliability evidence.
SKILL.md
1.0 KB, as published. Nobody here has run it
Eval Harness Design
Use this as the primary delivery skill for eval-driven changes to agent workflows, prompts, MCP tools, skills, or automation surfaces.
Workflow
- Define the target behavior, risk class, baseline, failure modes, and release decision before changing the harness.
- Choose deterministic code, schema, or rule graders before model or human graders.
- Separate capability evals from regression evals, then set pass@k, pass^k, cost, latency, and repeatability expectations.
- Store eval definitions and run reports as runtime evidence until reviewed and promoted.
Reference
Read references/eval-artifact-contract.md for eval definition, run report, evidence envelope, and storage boundaries.
Read references/grader-and-metric-rubric.md for grader choice, metric thresholds, pass@k usage, and anti-overfitting checks.