Run inference

Skill avnath13/evalpilot/skills/run-inference

Agent evals on autopilot: find quality bugs in your AI agent, ship a targeted fix, and prove it on a held-out set. Zero-dependency Agent Skill + CLI.

Install

npx -y skills add avnath13/evalpilot --skill run-inference

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Run the agent-under-test over every eval case and capture output + trajectory as trace artifacts. Use when you need fresh runs to grade or to A/B a fix. Do NOT use for cases that already carry a real production trace (skip them, the trace exists), do NOT use to build the eval set (use prepare-data), and do NOT use to score (use grade). Run the agent in a SANDBOX, never let eval runs hit production databases or live APIs.

SKILL.md

1.8 KB, as published. Nobody here has run it

run-inference: eval set → traces

Output: runs/<runId>/<case_id>.trace.json, conforming to evalpilot/schemas/trace.schema.json.

Procedure

Mint a run_id (e.g. run_2026_07_16_a). Read .evalpilot/evalset.jsonl.
Skip trace-origin cases. If a case has source_trace_id and its trace is already available, copy it into the run instead of re-calling the agent. This is the diagram's "skip if traces exist" edge, production data never gets re-run.
For every remaining case, call the agent per agent_under_test.kind:
- cli: substitute the case input into command ({input} placeholder), run it, capture stdout as output and parse any structured tool log into steps.
- http: POST input to url (as input_field), capture the JSON response.
- python: import and call the entrypoint.
- trace-only: do nothing; grading uses existing traces.
Capture the trajectory, not just the answer, tool calls, retrievals, errors, so grading can judge how the agent got there, not only the final string.
Record latency_ms and tokens when available. Run cases in parallel when safe.

Guardrails

On agent error, still write a trace with error set, a crash is a gradeable outcome.
Never fabricate an output. If the agent can't be called, stop and report the wiring bug.