Grade

Grade agent traces against rubrics using deterministic checks and binary LLM-as-judge (critique-then-label), no external eval framework required. Use when the user wants to score, judge, or evaluate agent outputs against criteria. Do NOT use to author a judge or validate it against human labels (use validate-evaluator/calibrate), and do NOT use to fix failures (use analyze-failures then optimize). Prefer a deterministic check over a judge whenever the criterion is objective (fix-before-eval).From its SKILL.md

Install

npx -y skills add avnath13/evalpilot --skill grade

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

3.5 KB, 779 tokens by cl100k_base, as published. Nobody here has run it

grade: traces × rubrics → verdicts (the zero-dep heart)

Output: runs/<runId>/verdicts.jsonl, each line conforming to evalpilot/schemas/verdict.schema.json.

You, the host agent, ARE the autorater. There is nothing to install.

Every rubric answers one of two questions: correctness (did it do the right thing?) or quality/safety (was the way it did it safe and usable?). A serious suite covers both.

Procedure

For each (trace, rubric) pair, where rubrics resolve to evalpilot/rubrics/<name>.rubric.md:

Read the rubric frontmatter. kind: deterministic | judge, severity, threshold.
Deterministic rubrics → write and run a small check (regex / JSON field compare / numeric tolerance / "was tool X called"). No model call. Example: refund-hallucination cross-checks any dollar figure in output against the result of lookup_refund in steps[]. Fast, exact, reproducible.
Judge rubrics → the rubric body IS the judge prompt. Read the trace + the rubric criteria and emit critique-then-label: {critique, result: Pass|Fail} plus reason, evidence, cluster_hint. Binary only, no Likert.
Always fill reason, evidence, and cluster_hint. These are what make stage 4 clustering and stage 5 fixes targeted rather than guesswork. Quote the exact offending span in evidence.
Write one verdict object per line. Print per-rubric aggregate: correctness 0.71 (65/92) | tool-use 0.88 | refund-halluc 0.14 (13 fails).

Calibration (do not skip for judge rubrics)

An unvalidated judge is a ruler of unknown length. Validate before trusting its scores (python3 -m evalpilot validate):

Label 30-50 cases spanning good / bad / borderline, ideally with two domain labelers. Where they disagree, discuss and resolve, if you can't resolve it, the rubric is underspecified: fix the rubric and relabel.
Measure TPR/TNR vs human consensus (not raw accuracy); apply Rogan-Gladen bias correction. Gate: >0.90 trust · 0.80-0.90 iterate the prompt · <0.80 not ready.
The calibration set becomes the judge's own regression test, re-run it whenever you edit the judge prompt and keep agreement above the bar.

Judge failure modes (anti-patterns to control for)

Position bias, judges prefer the first of two outputs. Randomize order.
Length bias, longer looks more complete. Control for it in the rubric.
Over-lenience, helpful-trained models call "Acceptable" outputs "Good"; calibration surfaces this.
Phrasing sensitivity, small rubric wording changes shift scores. Version the judge prompt.
Model drift, a judge-model upgrade shifts scores unchanged rubric. Recalibrate after any judge-model change.

Guardrails

Be a harsh, specific grader. Vague "looks fine" verdicts are useless downstream.
Same trace + same rubric should grade the same way, prefer deterministic checks wherever the criterion is objective (a healthy suite is ~60-70% code-based).

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.