agentsclimarketplace

Prepare data

Skill avnath13/evalpilot/skills/prepare-data

Agent evals on autopilot: find quality bugs in your AI agent, ship a targeted fix, and prove it on a held-out set. Zero-dependency Agent Skill + CLI.

Install
npx -y skills add avnath13/evalpilot --skill prepare-data

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Build a versioned eval set from raw traces/logs or synthetic scenarios, dedup, sample for coverage, redact PII, emit evalset.jsonl. Use when the user wants to turn production traces into test cases or assemble an eval dataset. Do NOT use when there is no data at all (use user-simulator first to bootstrap scenarios), and do NOT use to score cases (use grade) or to run the agent (use run-inference).

SKILL.md

1.9 KB, as published. Nobody here has run it

prepare-data: traces → eval set

Output: .evalpilot/evalset.jsonl, each line conforming to evalpilot/schemas/evalcase.schema.json.

Procedure

  1. Ingest. Read data.traces_path (glob) and/or scenarios.jsonl from user-simulator. Accept common shapes: raw JSONL logs, OpenTelemetry spans, chat transcripts.
  2. Distill each trace into a case. Extract the initial input. Set origin: "trace" and carry source_trace_id so run-inference can skip re-running it. For synthetic scenarios set origin: "synthetic".
  3. Redact PII when data.redact_pii is true, emails, names, card/refund identifiers are masked in input before writing. Never write raw PII to the eval set.
  4. Dedup & sample for coverage. Drop near-duplicate inputs. Prefer diversity across tags and intent so the set isn't 80% "reset my password". Target a balanced set; log what you sampled out (no silent truncation).
  5. Attach reference signals where derivable (e.g. the tool result that reveals the correct refund amount becomes reference.expected_refund). Optional but powerful.
  6. Version. Stamp version (date + counter). Write the file. Print a summary: 147 traces → 92 cases (61 trace, 31 synthetic) across 8 tags.

Guardrails

  • If zero usable traces are found, say so and suggest data.source: synthetic.
  • Keep ids stable across versions so metrics are comparable over time.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.