agentsclimarketplace

Ai eval regression tester

Skill sisodiabhumca/agent-skills/skills/ai-eval-regression-tester

Production-Ready Agent Skills : product analytics, growth experiments, CRM, research synthesis, postmortems, data contracts, SaaS spend, compliance, architecture maps, and LLM eval and many more.

Install
npx -y skills add sisodiabhumca/agent-skills --skill ai-eval-regression-tester

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use to run a regression eval suite over an LLM application — fixed test cases with deterministic graders (exact match, JSON schema, regex, embedding similarity, LLM-as-judge). Compares the candidate model/prompt against a baseline and gates a release on pass rate + per-tag thresholds.

SKILL.md

1.9 KB, as published. Nobody here has run it

AI Eval Regression Tester

When to invoke

  • "Run the eval suite for the new prompt version."
  • "Compare gpt-X vs the current baseline on our customer-support eval."
  • "Block release if eval pass rate drops below 95%."

Inputs needed

  1. Eval YAML / JSONL — list of cases with input, expected outputs, graders, tags.
  2. Candidate runner — Python callable / HTTP endpoint that takes input and returns output.
  3. Baseline run — JSONL of prior outputs (optional, for diffing).
  4. Pass thresholds — overall and per-tag.

Workflow

  1. Load cases.
  2. Run candidate over each case (parallelized).
  3. Grade with configured graders.
  4. Aggregate — pass rate overall + per tag + diff vs baseline.
  5. Gate — exit 1 if any threshold fails (CI-friendly).
  6. Report — Markdown + JSONL of every case for diffing.

Eval case schema

- id: refund_basic
  tags: [refund, policy]
  input: "How do I request a refund after 60 days?"
  graders:
    - type: contains
      values: ["return policy", "support team"]
    - type: not_contains
      values: ["sure thing", "lol"]
    - type: regex
      pattern: "policy"
    - type: json_schema
      schema:
        type: object
        required: [answer, citation]

Guardrails

  • Always require deterministic graders before LLM-as-judge.
  • LLM-judge results must include the judge's reasoning verbatim.
  • Per-case results must be logged to JSONL for forensics.
  • No silent retries on grader failures.

Reference code

run_eval.py runs cases in parallel, grades, diffs vs baseline, and exits with the right code.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.