Eval

Measure non-deterministic behavior — LLM features, agents, prompts, or a skill itself — with repeatable evals instead of one-shot checks. Use when building or tuning AI/LLM functionality (ranking, extraction, generation, agent loops), when a feature could pass once by luck, or when validating that a prompt or skill actually changes behavior.From its SKILL.md

Install

npx -y skills add vasu-devs/Forge --skill eval

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

4.3 KB, ~1.0k tokens by cl100k_base, as published. Nobody here has run it

███████╗██╗   ██╗ █████╗ ██╗
██╔════╝██║   ██║██╔══██╗██║
█████╗  ██║   ██║███████║██║
██╔══╝  ╚██╗ ██╔╝██╔══██║██║
███████╗ ╚████╔╝ ██║  ██║███████╗
╚══════╝  ╚═══╝  ╚═╝  ╚═╝╚══════╝

Eval-driven development

Deterministic code gets forge:verify — run it once, read the output, done. Non-deterministic behavior (anything LLM- or agent-driven) needs evals, because a single green run can be luck. Evals are the unit tests of AI work: a repeatable input set + expected behavior + a grader, run enough times that the result isn't luck (see "how many runs" below).

Two kinds of eval

Capability — can it do the thing? Target a pass rate (e.g. pass@k ≥ 0.90). Used while building/improving a feature.
Regression — does a known-good case still hold? For release-critical paths, demand pass^k = 1.0 (every single run passes). Each bug you fix becomes a regression eval so it can't silently return.

pass@k vs pass^k (pick to the stakes)

pass@k — succeeds in at least one of k tries. Right for capability/exploration ("can the model do this at all?").
pass^k — succeeds in all k tries. Right for reliability ("can I ship this without it flaking?"). Running once and seeing green tells you neither — you need k.

How many runs (pick k by stakes, not vibe)

"Run it a few times" is not a number. With 0 failures in n runs you can only claim the failure rate is below ~3/n at 95% confidence (the "rule of three") — so pass^k = 1.0 at k=3 proves almost nothing, and a release-critical path needs ~20 clean runs to claim <~15% failure. Defaults: capability k ≥ 10; release-path regression k ≥ 20; never k=1.

Graders — cheapest reliable one wins

Code / assertion — exact or structural check. Fast, deterministic, preferred.
Schema / constrained-output — validate structured output against a JSON schema or type contract; the best cheap grader for any LLM feature emitting structured data.
Rule / regex — pattern match on output.
Model-as-judge — an LLM grades against a written rubric. Use only for genuinely open-ended output, and treat the judge as a model under test: validate it against human labels first (target ≥0.8 agreement), pin the judge model + version, prefer pairwise/reference comparison over absolute 1-5 scores, and randomize answer order to cancel position bias.
Human — last resort, for subjective quality.

Building the eval set

Real inputs paired with expected behavior, covering: the common case, the edge cases, and every past failure (regression). Keep a held-out slice you never tune against.

Anti-patterns

Overfitting prompts to the eval set — always score on held-out cases, or you're memorizing, not improving.
Chasing pass-rate while ignoring cost/latency drift — track tokens and time alongside accuracy.
Evals that only exercise the happy path.

Evaluating a prompt or skill itself

Same shape as forge:tdd's watch-it-fail, applied to instructions: baseline a fresh agent WITHOUT the skill/prompt (does it fail or behave wrong?), then add it and confirm it now passes — and run that with/without comparison k times and compare pass rates, not once (a single before/after is the same luck this skill warns against). If it passes either way, the skill isn't earning its place. (This is how Anthropic's skill-creator validates skills.)

Exit

Capability evals at target pass@k and regression evals at pass^k = 1.0 → forge:verify / forge:ship. For an AI app like this one, the LLM ranking, extraction, and tailoring paths are exactly what to put behind evals.

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.

Gives 0 of the 12 instructions most evals benchmarks skills give in ~1.0k tokens

Counted across 499 of the 513 authors here whose files we hold, read 2026-09-06

Spawn with-skill and baseline runs in the same turnin 31 of 499, across 24 files
Keep SKILL.md under 500 linesin 31 of 499, across 24 files
Draft assertions while test runs are in progressin 31 of 499, across 24 files
Compare against the baseline after changesin 31 of 499, across 13 files
Define evals before codingin 26 of 499, across 17 files
Run evals frequently during developmentin 25 of 499, across 16 files
Keep evals fastin 24 of 499, across 15 files
Version evals with codein 24 of 499, across 15 files
Generate the eval viewer before evaluating outputs yourselfin 24 of 499, across 17 files
Generate an eval report after runsin 24 of 499, across 15 files
Track pass@k metrics over timein 22 of 499, across 14 files
Save a baseline before making changesin 21 of 499, across 9 files

Said here and by no other author read

run evals enough times that results aren't luck
default k to 10, or 20 for release paths
demand pass^k equals 1.0 on release-critical paths
turn each fixed bug into a regression eval
pin the judge model and version
randomize answer order to cancel position bias

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.