agentsclimarketplace

Grade

Skill avnath13/evalpilot/skills/grade

Agent evals on autopilot: find quality bugs in your AI agent, ship a targeted fix, and prove it on a held-out set. Zero-dependency Agent Skill + CLI.

Install
npx -y skills add avnath13/evalpilot --skill grade

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Grade agent traces against rubrics using deterministic checks and binary LLM-as-judge (critique-then-label), no external eval framework required. Use when the user wants to score, judge, or evaluate agent outputs against criteria. Do NOT use to author a judge or validate it against human labels (use validate-evaluator/calibrate), and do NOT use to fix failures (use analyze-failures then optimize). Prefer a deterministic check over a judge whenever the criterion is objective (fix-before-eval).

SKILL.md

3.5 KB, as published. Nobody here has run it

grade: traces × rubrics → verdicts (the zero-dep heart)

Output: runs/<runId>/verdicts.jsonl, each line conforming to evalpilot/schemas/verdict.schema.json.

You, the host agent, ARE the autorater. There is nothing to install.

Every rubric answers one of two questions: correctness (did it do the right thing?) or quality/safety (was the way it did it safe and usable?). A serious suite covers both.

Procedure

For each (trace, rubric) pair, where rubrics resolve to evalpilot/rubrics/<name>.rubric.md:

  1. Read the rubric frontmatter. kind: deterministic | judge, severity, threshold.
  2. Deterministic rubrics → write and run a small check (regex / JSON field compare / numeric tolerance / "was tool X called"). No model call. Example: refund-hallucination cross-checks any dollar figure in output against the result of lookup_refund in steps[]. Fast, exact, reproducible.
  3. Judge rubrics → the rubric body IS the judge prompt. Read the trace + the rubric criteria and emit critique-then-label: {critique, result: Pass|Fail} plus reason, evidence, cluster_hint. Binary only, no Likert.
  4. Always fill reason, evidence, and cluster_hint. These are what make stage 4 clustering and stage 5 fixes targeted rather than guesswork. Quote the exact offending span in evidence.
  5. Write one verdict object per line. Print per-rubric aggregate: correctness 0.71 (65/92) | tool-use 0.88 | refund-halluc 0.14 (13 fails).

Calibration (do not skip for judge rubrics)

An unvalidated judge is a ruler of unknown length. Validate before trusting its scores (python3 -m evalpilot validate):

  • Label 30-50 cases spanning good / bad / borderline, ideally with two domain labelers. Where they disagree, discuss and resolve, if you can't resolve it, the rubric is underspecified: fix the rubric and relabel.
  • Measure TPR/TNR vs human consensus (not raw accuracy); apply Rogan-Gladen bias correction. Gate: >0.90 trust · 0.80-0.90 iterate the prompt · <0.80 not ready.
  • The calibration set becomes the judge's own regression test, re-run it whenever you edit the judge prompt and keep agreement above the bar.

Judge failure modes (anti-patterns to control for)

  • Position bias, judges prefer the first of two outputs. Randomize order.
  • Length bias, longer looks more complete. Control for it in the rubric.
  • Over-lenience, helpful-trained models call "Acceptable" outputs "Good"; calibration surfaces this.
  • Phrasing sensitivity, small rubric wording changes shift scores. Version the judge prompt.
  • Model drift, a judge-model upgrade shifts scores unchanged rubric. Recalibrate after any judge-model change.

Guardrails

  • Be a harsh, specific grader. Vague "looks fine" verdicts are useless downstream.
  • Same trace + same rubric should grade the same way, prefer deterministic checks wherever the criterion is objective (a healthy suite is ~60-70% code-based).

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.