Run inference

Skill avnath13/evalpilot/skills/run-inference

Run the agent-under-test over every eval case and capture output + trajectory as trace artifacts. Use when you need fresh runs to grade or to A/B a fix. Do NOT use for cases that already carry a real production trace (skip them, the trace exists), do NOT use to build the eval set (use prepare-data), and do NOT use to score (use grade). Run the agent in a SANDBOX, never let eval runs hit production databases or live APIs.From its SKILL.md

Install

npx -y skills add avnath13/evalpilot --skill run-inference

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.8 KB, 361 tokens by cl100k_base, as published. Nobody here has run it

run-inference: eval set → traces

Output: runs/<runId>/<case_id>.trace.json, conforming to evalpilot/schemas/trace.schema.json.

Procedure

Mint a run_id (e.g. run_2026_07_16_a). Read .evalpilot/evalset.jsonl.
Skip trace-origin cases. If a case has source_trace_id and its trace is already available, copy it into the run instead of re-calling the agent. This is the diagram's "skip if traces exist" edge, production data never gets re-run.
For every remaining case, call the agent per agent_under_test.kind:
- cli: substitute the case input into command ({input} placeholder), run it, capture stdout as output and parse any structured tool log into steps.
- http: POST input to url (as input_field), capture the JSON response.
- python: import and call the entrypoint.
- trace-only: do nothing; grading uses existing traces.
Capture the trajectory, not just the answer, tool calls, retrievals, errors, so grading can judge how the agent got there, not only the final string.
Record latency_ms and tokens when available. Run cases in parallel when safe.

Guardrails

On agent error, still write a trace with error set, a crash is a gradeable outcome.
Never fabricate an output. If the agent can't be called, stop and report the wiring bug.

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.