Review interface

Skill avnath13/evalpilot/skills/review-interface

Generate a self-contained browser tool for humans to label agent traces, producing the human labels that judge validation needs. Use when the user needs to collect Pass/Fail labels, calibrate a judge, or review traces by hand. Do NOT use to grade automatically (use grade) or to validate the judge math (use validate-evaluator/calibrate), this only collects labels.From its SKILL.md

Install

npx -y skills add avnath13/evalpilot --skill review-interface

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.0 KB, 392 tokens by cl100k_base, as published. Nobody here has run it

review-interface: human labeling UI

Output: a self-contained review.html (no external dependencies) that exports <rubric>.labels.jsonl, the input to calibrate/validate.

Fastest path: python3 -m evalpilot review builds it from the demo run. To author from scratch, follow the directives below.

Directives

Full trace, domain-native rendering. Show input, every intermediate step, and the final output. Emails look like emails; code is syntax-highlighted; collapse redundant content; promote key metadata to headers.
Tool-Call Anatomy cards, for each tool call display (1) name+icon, (2) input args, (3) output/result, (4) timing/latency. Expandable; color-code success vs error.
Trace-level labeling, Pass / Fail / Defer buttons + a notes field; auto-save (no explicit save). Never span-level.
Fast keyboard workflow, 1=Pass, 2=Fail, D=Defer, ←/→ navigate; a progress counter.
Export <rubric>.labels.jsonl ({case_id, pass, labeler, note}), excluding Defers.

Calibration labels

Collect 30-50 cases spanning good / bad / borderline, from two domain labelers.
Reconcile disagreements: if two labelers can't agree, the rubric is underspecified , sharpen the rubric and relabel, don't average the noise. The resolved set feeds validate and doubles as the judge's regression test.

Anti-patterns

Showing only the final output, reviewers can't see where the trace went wrong.
Requiring a manual save, lost labels kill a labeling session.

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.