Llm eval design

Skill Amey-Thakur/AI-SKILLS/skills/llm-engineering/llm-eval-design

Plug-and-play skills and prompts for every AI coding agent

Install

npx -y skills add Amey-Thakur/AI-SKILLS --skill llm-eval-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

19 days oldThe repository was created 19 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Build LLM evaluations from real failures with calibrated judges and regression gates that catch quality drift. Use when an LLM feature needs quality measurement or prompts change without anyone knowing what broke.

SKILL.md

3.6 KB, as published. Nobody here has run it

LLM eval design

Without evals, every prompt edit is a blind bet and every model upgrade is a gamble. The eval suite is to LLM applications what the test suite is to code: built from real failures, run on every change, trusted because it is calibrated.

Method

Harvest the eval set from production reality. Real (anonymized) inputs from logs (see llm-observability), especially the failures: user rephrases, thumbs-downs, support escalations, and the weird inputs nobody designed for; synthetic cases fill coverage gaps (edge lengths, adversarial phrasings, each language you support) but the spine is what users actually send (see ml-error-analysis: same loop, generative edition). Start small and honest: 50 well-chosen cases beat 5000 scraped ones.
Grade with the cheapest sufficient judge. Exact/ programmatic checks wherever possible (structured output validates against schema: see structured-output; retrieval cites the right doc; the SQL runs and returns the expected rows); LLM-as-judge only for genuine open-ended quality, with a rubric per criterion (correctness, groundedness, tone: separate scores, not one vibe number).
Calibrate the judge against humans. Label 50-100 cases yourself, measure judge-human agreement, iterate the rubric until agreement is strong on the dimensions that matter; recheck after judge-model upgrades. Known judge biases (position, verbosity, self-preference) need mitigations (randomize orderings, cap length credit): an uncalibrated judge is a random number generator with confidence (see model-evaluation's calibration ethic).
Gate changes on eval runs. Every prompt edit, model swap, retrieval change, or parameter tune runs the suite before shipping (CI for prompts: see deployment-pipelines); compare against the incumbent with per-case diffs, not just aggregates: a +2% average hiding a regression on the refund-request cluster is a net loss (see model-evaluation's slicing). Version eval sets alongside prompts (see experiment-tracking).
Test behaviors, not just answers. Invariance (paraphrase in, same answer out), refusal correctness (should-refuse and should-not-refuse sets: see llm-guardrails), format stability under adversarial input, and multi-turn behaviors for conversational systems (context carried, corrections honored: see conversation-design): single-turn QA evals miss the failures users actually hit.
Close the loop continuously. New production failures enter the set weekly (the suite grows like a regression suite: every bug becomes a test: see regression-tests); sample live traffic for scheduled eval-in-production runs so drift (model provider updates, user-population shifts) surfaces on your dashboard, not in support tickets (see drift-monitoring's architecture aimed at quality).

Boundaries