Build evals
Generate a runnable eval suite (tasks, graders, judges, targets) for a Claude Code plugin BEFORE it is built. Use for "build evals", "eval suite for my plugin", "test my plugin", "retrofit evals", "mine traces into tasks". In the forge pipeline this is the evals phase; retrofit and mine modes run standalone against any existing plugin.From its SKILL.md
npx -y skills add Aznatkoiny/plugin-forge --skill build-evalsAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
12.5 KB, as published. Nobody here has run it
build-evals — the eval builder
!cat .forge/state.json 2>/dev/null || echo NO_STATE
Generate the complete evals/ tree for a target plugin. In new mode the plugin does not
exist yet: tasks derive from design/CONTRACT.md (the PDR), not from code. The suite this
skill emits becomes the definition of "done" for the entire build — treat every task as a
contract clause, not a test.
This skill drafts and lints the suite. It never arms, freezes, or scores it — those are
/plugin-forge:arm-evals and bin/forge-eval. It is model-invocable by design (decision D2):
the forge conductor chains it via the Skill tool; safety comes from the state guard below,
not from invocation control.
State guard (new mode only)
Apply this table to the injected state line above before doing anything else in new mode
(also the default when no argument is given). retrofit and mine skip this guard entirely —
they run standalone against any plugin, with or without a forge pipeline (see their sections).
| Injected state | Action |
|---|---|
NO_STATE | REFUSE. Say: "No forge pipeline in this project. Run /plugin-forge:forge to start one, or use /plugin-forge:build-evals retrofit <plugin-path> for an existing plugin." Stop. |
phase is interview or contract | REFUSE. Say: "The contract is not approved yet — evals derive from it. Run /plugin-forge:forge resume." Stop. |
phase is evals | Proceed with new mode. |
phase is armed, building, smoke, verify, or ship | REFUSE edits. The suite is frozen (sha256 in .forge/freeze.json); guard hooks will block writes anyway. Say: "The suite is frozen. Append the desired change to .forge/eval-change-requests.md, then run /plugin-forge:arm-evals to apply it, re-gate, and re-freeze as a new version." Stop. |
Mode: new (pipeline phase)
Work through the steps in order. Load the referenced file before executing its step — each reference is binding, not advisory.
1. Map the surface from the contract
Read design/CONTRACT.md. Extract, per capability row:
- the chosen composition (skills, hooks, agents, MCP servers, scripts) and its archetype
- the C5 Testability column: grader type, observable signal, and planned task IDs — this column IS the task backlog; build-evals exists to make those IDs real
- the C1 Trigger column: every auto-invoked component (model-invoked skill, hook, agent with proactive description) needs positive AND negative trigger tasks
- external dependencies (MCP servers, APIs, corpora) and whether the contract chose live or mock for each
Also read design/INTENT.md for the runtime-environment answers (headless/cloud/local)
— they constrain what graders can observe.
2. Interview — propose clusters, recommend, WAIT
Never one-shot the suite (interviewing beats one-shot generation — LangChain's central empirical finding). Use AskUserQuestion to:
- Propose 3–5 testable ability clusters derived from the PDR rows, each with a one-line description, its archetype, and a rough task count. Recommend one ordering and say why. WAIT for the user's choice before generating anything.
- For every external dependency, ask live-vs-mock. Default recommendation: mock fixture stubs for write-heavy, paid, or rate-limited tools; live only for cheap read-only calls the user explicitly accepts. Record the decision per dependency — the harness stores which mode each result came from.
For scientific or business domains where fixture corpora need domain-real content, offer
to run the /plugin-forge:skill-forge research pipeline to ground fixtures in real sources instead of
inventing plausible-looking data (decision D10).
3. Scaffold
Copy the scaffold from ${CLAUDE_PLUGIN_ROOT}/templates/evals/ into the target plugin as
evals/, and create the sibling evals-holdout/ next to it (NOT inside evals/). The
scaffold provides registry.yaml, targets/{default,no-plugin,crowded}.yaml,
suites/{capability,regression,triggers}.yaml, judges/{fact,rubric,battle}.yaml,
labels/labels.jsonl, and tasks/_example/. Delete tasks/_example/ after real tasks
exist.
Then vendor the harness runner (stdlib-only, so the suite outlives plugin-forge):
mkdir -p <plugin_dir>/evals/bin && cp "${CLAUDE_PLUGIN_ROOT}/scripts/harness/run.py" <plugin_dir>/evals/bin/run.py && chmod +x <plugin_dir>/evals/bin/run.py
4. Author tasks
Follow references/task-authoring.md for everything about tasks: the exact task.yaml
schema, the 20–50 task budget, the two-experts bar, the REQUIRED reference solution, the
goal-condition grammar for instruction.md, fixture rules, the holdout split, and
elicitation_answers. Every task ID planned in the contract's C5 column must exist in
evals/tasks/ or evals-holdout/tasks/ when this step ends.
5. Choose graders
Follow references/grader-menu.md: deterministic-first hierarchy (workspace → transcript
→ state_check → judge → trigger, plus the simulate_user driver), archetype routing, the
mandatory numeric tolerance, the never-assert-tool-ORDER rule, and the crowded-listing
requirement for trigger tasks.
6. Author judges
Only where the grader menu genuinely runs out of deterministic options. Follow
references/judge-authoring.md: modelgraded-spec YAML shape, one rubric dimension per
isolated judge call, Unknown always legal, pinned cheap judge model different from the
target model, anti-echo phrasing. Seed evals/labels/labels.jsonl with the unlabeled
sample skeleton — calibration itself is arm-evals' gate, not this skill's.
7. Generate the elicitation auto-answer hook (when applicable)
If the target plugin bundles or drives an MCP server that uses elicitation, headless eval
runs will hang or die — no dialog exists under claude -p. For every such server:
- set
elicitation_answersin each task.yaml that exercises it (see task-authoring.md) - emit
evals/hooks/elicitation-auto-answer.sh(bash,set -euo pipefail, executable) andevals/hooks/elicitation-settings.jsonregistering it as an exec-form Elicitation hook:"command": "${CLAUDE_PLUGIN_ROOT}/evals/hooks/elicitation-auto-answer.sh"— harness runs merge this settings file; production users never load it. The script reads the elicitation request JSON from stdin, looks up the requested fields in.forge-eval/elicitation-answers.json(staged into the workspace by the harness from the task'selicitation_answers), and prints the matching answers object to stdout with exit 0. No match → one-line stderr message and exit 2, so the trial fails loudly instead of hanging.
8. Lint the suite
Run every mechanical lint in references/antipatterns.md (L01–L16) and fix all findings
before proceeding. L01 (numeric tolerance) and L02 (task-text/grader consistency) are the
builder meta-eval contract: plugin-forge's own dogfood suite feeds this skill adversarial
specs engineered to reproduce the CORE-Bench and METR grader bugs, and passes only if the
emitted suite provably cannot contain them. Do not hand a suite to arm-evals with any lint
red.
9. Smoke-run drafts and hand off
An eval that has never run is not delivered:
- Run
forge-eval reference --task <ID>for at least 3 representative tasks (one per archetype used) and show the user the graded output. Useforge-eval run --dry-runfor a cost estimate of the full suite; state it plainly — trials × tasks × judges is real money, and--max-cost-usdexists for a reason. - Present the suite summary: task count per suite (capability/regression/triggers), holdout count, judge dimensions, live-vs-mock map, estimated full-run cost.
- Tell the user the exact next command:
/plugin-forge:arm-evals— the four arming gates (reference-green, red baseline, judge calibration, freeze) all run there.
Do not modify .forge/state.json. The phase advances to armed only when arm-evals
freezes the suite.
Mode: retrofit <plugin-path>
Standalone — no forge pipeline, no state guard, works on any installed or in-repo plugin.
- Map the plugin surface from files (there is no PDR): every
skills/*/SKILL.mdand command frontmatter (descriptions = trigger contracts),agents/*.md,hooks/hooks.json(matchers = firing contracts),.claude-plugin/plugin.json,.mcp.json. Derive one ability cluster per user-facing capability and one trigger axis per auto-invoked component. MCP tool checks must use plugin-scoped names:mcp__plugin_<plugin>_<server>__<tool>. - Then run steps 2–9 above unchanged, writing
evals/andevals-holdout/into the plugin's own directory. Where step 1 of new mode says CONTRACT.md, use the surface map; flag any capability whose observable signal is unclear as "unverifiable — needs a design decision" instead of inventing a grader for it. - Recommend arming: retrofit suites get the same four gates via
/plugin-forge:arm-evals <plugin-path>before their scores mean anything.
Mode: mine
Standalone — converts captured production failures into tasks (the flywheel).
- Read traces from the generated plugin's trace-capture hook output:
${CLAUDE_PLUGIN_DATA}/traces/*.jsonl(tool args, results, errors per session). If no traces exist, say so and stop — mine has no other input; suggest confirming the trace-capture hook is installed. - Cluster failures (repeated errors, user corrections, wrong outputs) and propose each cluster as a candidate task via AskUserQuestion — same interview discipline: propose, recommend, WAIT.
- For each accepted failure, author a task per
references/task-authoring.mdwith the real failing input asinstruction.mdseed, a fixture reconstructing the state, and a reference solution showing the correct behavior. New tasks go toevals/tasks/in the regression suite (they protect against a now-known failure). - A frozen suite that gains tasks is a NEW suite: finish by instructing the user to run
/plugin-forge:arm-evalsto re-gate and re-freeze (registry version bumps, scores stay comparable). If guard hooks block the writes, that is the freeze working — route the change through.forge/eval-change-requests.mdas the guard message says.
Standing rules (all modes)
- Deterministic-first, always. A judge where a file check would do is a bug.
- Every task ships a reference solution. No exceptions — 0% pass usually means a broken task, and only the reference proves otherwise.
- Positive AND negative trigger cases for every auto-invoked component; one-sided suites train over-triggering.
- Golden answers, key-fact lists, and reference solutions live under
reference/, which the harness stages OUTSIDE the agent-visible worktree. Never copy any of it intofixtures/orinstruction.md. - Holdout tasks (
evals-holdout/) are never named, quoted, or summarized in any session the builder loop can read. - Track cost honestly:
tracked_metricsare recorded, never graded; suite cost estimates go to the user before, not after, an expensive run. - Never emit deprecated shapes: no legacy decision/approve hook JSON, no legacy mount-path
skill references, and MCP tool names always in the full plugin-scoped form
mcp__plugin_<plugin>_<server>__<tool>— never the short server-only form.
References
references/task-authoring.md— task budget, schema, instruction grammar, fixtures, holdout, elicitation fixturesreferences/grader-menu.md— grader hierarchy, archetype routing, trigger + crowded target, simulate_userreferences/judge-authoring.md— modelgraded spec shape, judge rules, calibration handoffreferences/antipatterns.md— reward-hacking catalog with the mechanical lints L01–L16