agentsclimarketplace
Sign in

Production monitor

Skill avnath13/evalpilot/skills/production-monitor

Continuously grade live production traces from an OTel endpoint and alert when a quality rubric crosses threshold, grading in place without re-running the agent (extrinsic evaluation). Use when the user wants ongoing online evals or monitoring in production. Do NOT use for offline dev iteration (use the run loop / intrinsic evaluation) and do NOT re-run the agent on production inputs, grade what actually happened.From its SKILL.md

Install
npx -y skills add avnath13/evalpilot --skill production-monitor

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.6 KB, 283 tokens by cl100k_base, as published. Nobody here has run it

production-monitor: the production loop

Turns the dev flywheel into a standing online monitor. Grades in place, skips stages ① and ②.

Procedure

  1. Pull recent traces from data.otel_endpoint (or an exported trace file) on a cadence.
  2. Distill each into a trace artifact (reuse the prepare-data distillation) with origin: trace. Do NOT re-run the agent, you're grading what really happened.
  3. grade the batch with the same rubrics used in the dev loop, so dev and prod speak the same quality language. Append to a rolling verdicts.jsonl.
  4. Compute rolling per-rubric scores over a window. Alert when any rubric crosses its targets threshold (write to report.md, optionally exit non-zero for CI/cron).
  5. On a sustained regression, hand the failing slice to analyze-failures → optimize so a production regression can re-enter the dev loop and get fixed.

Guardrails

  • Read-only against production data. Never write back to the traced system.
  • Respect data.redact_pii. Never persist raw production PII into the eval store.
  • Sampling is fine at volume, log the sample rate so scores are interpreted correctly.

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.