agentsclimarketplace

Rc bench

Skill juandarn/rawcode/skills/rc-bench

Measure a prompt/config change with a real pass@1 A/B on HumanEval instead of a vibe. Trigger: verifying whether a system-prompt or persona change helps or hurts. Composes with Gentle AI's sdd-verify.From its SKILL.md

Install
npx -y skills add juandarn/rawcode --skill rc-bench

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its file declares

Copied from the file, not written here

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

2.2 KB, as published. Nobody here has run it

Purpose

A rawcode extension that turns "is this prompt change better?" into a number. It runs the repo's bench/ — a paired HumanEval pass@1 A/B graded by executable unit tests (no LLM judge) — and reports correctness and token deltas as separate axes. Use it when a persona/prompt/config change needs evidence, not an opinion.

When to use

  • Before claiming a system-prompt or persona change "helps".
  • As a quantitative step inside sdd-verify for a change that alters agent behavior.
  • To sanity-check that a conciseness tweak did not lower correctness (it can — see the Anthropic conciseness postmortem).

How to run

bench/run.sh 40        # 40 HumanEval problems, baseline vs the rawcode prompt

It downloads HumanEval on first run, generates one greedy completion per problem per arm via claude -p, runs each problem's own tests, then prints:

  • Correctness (pass@1) per arm, the paired difference, a 95% CI, and a McNemar exact p-value. p > 0.05 = no significant change.
  • Output tokens per arm and the paired reduction with a bootstrap 95% CI.

Reading the result honestly

  • A win = tokens down with pass@1 not significantly down. Report both.
  • Never blend the two into one score — length and correctness are different axes.
  • Watch for grading artifacts: an aggressive persona can make the model return a terse confirmation ("Done") or act agentically instead of printing code, which a naive extractor miscounts as a failure. If pass@1 drops, inspect the raw outputs before believing it — the drop may be format, not quality.
  • Small N (40, single sample) → wide correctness CI. Raise N or samples for pass@k.

Compose with Gentle AI

  • Inside sdd-verify: cite the bench/ numbers (with CI) as the quantitative evidence, alongside the spec-conformance check.
  • Persist the run summary under the active artifact store's verify-report key — no new backend.

Keep looking

Skills are one crate of 608,475. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.