Llm eval suite scaffolder

Skill imtiazrayhan/agentscamp-library/skills/llm-eval-suite-scaffolder

Installable library of AI coding agents, skills, commands, guides & tools — the copy-paste mirror of agentscamp.com

Install

npx -y skills add imtiazrayhan/agentscamp-library --skill llm-eval-suite-scaffolder

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Stand up an evaluation suite for an LLM feature from scratch — a representative dataset, the right metrics, a baseline score, and a CI gate — using DeepEval, promptfoo, or RAGAS. Use when a feature has no evals, before tuning a prompt, or when adding an LLM feature to CI.

SKILL.md

2.8 KB, as published. Nobody here has run it

The hardest part of LLM evaluation is starting. This skill scaffolds a complete, runnable eval suite for a feature — dataset, metrics, baseline, and CI wiring — using the framework that fits the stack (DeepEval for Python/pytest, promptfoo for config-driven CLI, RAGAS for RAG-specific metrics).

When to use this skill

An LLM feature ships with no evals and you need a gate before changing it further.
You're about to tune a prompt or swap a model and want to measure the change, not guess.
You're adding an LLM feature to CI and need a suite that fails on regressions.

Instructions

Pin the task and the unit of scoring. State exactly what the feature must produce and how one output is judged: exact match, JSON-schema valid, a numeric tolerance, or an LLM-as-judge rubric. An ambiguous success criterion is the real bug — resolve it first.
Build a representative dataset. Collect 20–50 real inputs with expected behavior, deliberately oversampling hard and adversarial cases (empty input, ambiguity, the format that broke last time, the prompt-injection attempt). Freeze it under version control. For RAG, capture the gold passages too.
Pick the few metrics that matter. Two or three the feature is actually graded on — not every metric the framework offers. Faithfulness and answer relevancy for RAG; task accuracy and format validity for extraction; a calibrated rubric (llm-as-judge-scorer) for open-ended output.
Choose the framework and scaffold it. Generate the suite: DeepEval (pytest-style assertions), promptfoo (YAML matrix), or RAGAS (RAG metrics). Wire the dataset and metrics in, with thresholds.
Record a baseline. Run the current/naive prompt over the full set and commit the score. Every later number is compared to this.
Wire the CI gate. Add a run-evals step that fails the build when a metric drops below threshold, so regressions are caught in PRs — see the Run Evals command.

[!WARNING] Don't generate hundreds of synthetic cases and call it an eval set. Twenty real, well-chosen cases — including the adversarial ones — beat a thousand bland synthetic ones. Quality and coverage of failure modes, not volume.

Output