agentsclimarketplace

Review interface

Skill avnath13/evalpilot/skills/review-interface

Agent evals on autopilot: find quality bugs in your AI agent, ship a targeted fix, and prove it on a held-out set. Zero-dependency Agent Skill + CLI.

Install
npx -y skills add avnath13/evalpilot --skill review-interface

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Generate a self-contained browser tool for humans to label agent traces, producing the human labels that judge validation needs. Use when the user needs to collect Pass/Fail labels, calibrate a judge, or review traces by hand. Do NOT use to grade automatically (use grade) or to validate the judge math (use validate-evaluator/calibrate), this only collects labels.

SKILL.md

2.0 KB, as published. Nobody here has run it

review-interface: human labeling UI

Output: a self-contained review.html (no external dependencies) that exports <rubric>.labels.jsonl, the input to calibrate/validate.

Fastest path: python3 -m evalpilot review builds it from the demo run. To author from scratch, follow the directives below.

Directives

  • Full trace, domain-native rendering. Show input, every intermediate step, and the final output. Emails look like emails; code is syntax-highlighted; collapse redundant content; promote key metadata to headers.
  • Tool-Call Anatomy cards, for each tool call display (1) name+icon, (2) input args, (3) output/result, (4) timing/latency. Expandable; color-code success vs error.
  • Trace-level labeling, Pass / Fail / Defer buttons + a notes field; auto-save (no explicit save). Never span-level.
  • Fast keyboard workflow, 1=Pass, 2=Fail, D=Defer, ←/→ navigate; a progress counter.
  • Export <rubric>.labels.jsonl ({case_id, pass, labeler, note}), excluding Defers.

Calibration labels

  • Collect 30-50 cases spanning good / bad / borderline, from two domain labelers.
  • Reconcile disagreements: if two labelers can't agree, the rubric is underspecified , sharpen the rubric and relabel, don't average the noise. The resolved set feeds validate and doubles as the judge's regression test.

Anti-patterns

  • Showing only the final output, reviewers can't see where the trace went wrong.
  • Requiring a manual save, lost labels kill a labeling session.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.