agentsclimarketplace

Production monitor

Skill avnath13/evalpilot/skills/production-monitor

Agent evals on autopilot: find quality bugs in your AI agent, ship a targeted fix, and prove it on a held-out set. Zero-dependency Agent Skill + CLI.

Install
npx -y skills add avnath13/evalpilot --skill production-monitor

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Continuously grade live production traces from an OTel endpoint and alert when a quality rubric crosses threshold, grading in place without re-running the agent (extrinsic evaluation). Use when the user wants ongoing online evals or monitoring in production. Do NOT use for offline dev iteration (use the run loop / intrinsic evaluation) and do NOT re-run the agent on production inputs, grade what actually happened.

SKILL.md

1.6 KB, as published. Nobody here has run it

production-monitor: the production loop

Turns the dev flywheel into a standing online monitor. Grades in place, skips stages ① and ②.

Procedure

  1. Pull recent traces from data.otel_endpoint (or an exported trace file) on a cadence.
  2. Distill each into a trace artifact (reuse the prepare-data distillation) with origin: trace. Do NOT re-run the agent, you're grading what really happened.
  3. grade the batch with the same rubrics used in the dev loop, so dev and prod speak the same quality language. Append to a rolling verdicts.jsonl.
  4. Compute rolling per-rubric scores over a window. Alert when any rubric crosses its targets threshold (write to report.md, optionally exit non-zero for CI/cron).
  5. On a sustained regression, hand the failing slice to analyze-failures → optimize so a production regression can re-enter the dev loop and get fixed.

Guardrails

  • Read-only against production data. Never write back to the traced system.
  • Respect data.redact_pii. Never persist raw production PII into the eval store.
  • Sampling is fine at volume, log the sample rate so scores are interpreted correctly.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.