Benchmark
Measure whether a harness component (a skill, agent, or rule) actually earns its keep — run the same task k times with and without it, in isolated git worktrees, and report pass@k (works once) and pass^k (works consistently). Opt-in; not wired into the default pipeline. Use to decide keep-vs-cut on evidence, or to fill the R5 benchmark for a /extract proposal.From its SKILL.md
npx -y skills add vasuag09/harness-claude --skill benchmarkAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
4.4 KB, as published. Nobody here has run it
/benchmark — does this component earn its keep?
Goal: replace opinion with evidence. Run a task k times with a named component enabled and
k times without it, each trial in its own throwaway git worktree, then compare reliability.
Report pass@k (passed in ≥1 trial — "can it work") and pass^k (passed in all k trials —
"does it work reliably"), and a verdict.
Opt-in. No default-pipeline skill or hook invokes
/benchmark. It forks git worktrees and (when you supply a real task) spawns work k×2 times — always an explicit, deliberate run. Git boundary: it onlygit worktree add/remove(an isolated sandbox); it never commits, pushes, or branches the working tree.
When to run
- You want to decide whether a skill / agent / rule is worth keeping (keep-vs-cut on data).
- A
/extractproposal sits at R5 MANUAL and you want a real empirical signal — benchmark it with--component <draft-name>so the result lands where R5 looks.
The harness contract
scripts/eval/benchmark.js is generic: it knows nothing about your component. It toggles the
component only through env vars the task command honors, for each trial:
| env | meaning |
|---|---|
HARNESS_COMPONENT | the component name (label) |
HARNESS_COMPONENT_ENABLED | 1 (with) · 0 (without) |
HARNESS_TRIAL | 0-based trial index |
A trial passes iff its score command exits 0. Default score = the task's own exit code;
recommended = node scripts/eval/checkpoint.js <spec> so a trial is scored against a spec's
acceptance criteria (reusing the v0.4 gate).
Do this
- Define the task. Decide what "doing the task" means and write a command that performs it
in the current worktree, reading
HARNESS_COMPONENT_ENABLEDto enable/disable the component under test (e.g. include the skill's guidance only when=1). For a model-driven task, the command drives a subagent; for a deterministic check, any shell script works. - Define scoring. Either let the task's exit code be the signal, or pass
--score(e.g.checkpoint.jsagainst the task's spec) so PASS/FAIL is judged against acceptance criteria. - Run the benchmark:
Exit 0 = pass verdict (component earns its keep) · 1 = fail verdict (no reliability gain) · 2 = usage/setup error (nonode scripts/eval/benchmark.js --component <name> --task '<cmd>' [--score '<cmd>'] [--k 5]--task, not a git repo, no HEAD commit). - Read the result. The run prints both cohorts' pass@k / pass^k / success-rate and the
verdict, and writes
.claude/eval/benchmarks/<slug>.json(<slug>= the component name). The verdict is: earns its keep iffwith.successRate > without.successRateor (with.passCaret && !without.passCaret). - Act on it. Recommend keep or cut with the numbers. For an
/extractproposal, the artifact now feeds R5: re-runnode scripts/eval/extract-rubric.js <draft>and R5 reports PASS/FAIL from the benchmark instead of MANUAL. Promotion intoskills/is still a human call (R4 + your approval) — a passing benchmark never auto-promotes.
Security & trust
--task/--scoreare author-authored shell, same trust boundary as the test scripts and theevalfences in checkpoint.js. Don't pass untrusted commands.- The result artifact records metrics only (counts, rates, the commands you supplied) — no trial output or secrets are captured.
- Worktrees are always removed after the run, including on task/score failure.
Notes
- Self-contained: pure Node +
git+ shell, no companions (mgrep/graph/context7), no network. - Keep k small for model-driven tasks (each trial is a real run); larger k for cheap deterministic tasks buys tighter pass^k confidence.
Exit criterion
A benchmark result (.claude/eval/benchmarks/<slug>.json) exists with both cohorts' pass@k /
pass^k and a verdict, and you've made an evidence-based keep-vs-cut recommendation (or fed the
artifact back into a /extract proposal's R5).