Evaluation frameworks

Evaluation Frameworks

Purpose

LLM systems fail silently and improve anecdotally: a prompt tweak that fixes the case in front of you can break ten cases you did not look at. This skill is the measurement discipline — build the golden set before tuning, baseline before improving, change one variable at a time, and never trust a judge you have not calibrated.

Metadata

• Prerequisites: llm-system-design (the output contract being evaluated should already be designed); validation-and-testing (the risk-first testing mindset this skill extends to LLM behavior).
• Related Skills: llm-integration-reliability (judge ground-truth injection mechanics), prompt-and-context-engineering (the thing most often being compared), research-methodology (evidence discipline and epistemic labels for reporting results).
• Owns: evaluation methodology for LLM systems; golden sets; LLM-judge calibration; regression evals; sample-size honesty.

When to Use / When NOT to Use

Use when:

• Building or changing any prompt, model choice, temperature, or pipeline step in an LLM system.
• Someone (including you) says an AI change "seems better" — that sentence is this skill's trigger.
• Designing an LLM-as-judge or grader and deciding whether to trust its verdicts.
• Choosing between models or providers for a task.

Do NOT use (load the sibling instead):

• Testing deterministic code → validation-and-testing.
• Outputs are malformed/unparseable (a reliability defect, not a quality question) → llm-integration-reliability.
• Deciding what the LLM's job should be in the first place → llm-system-design.
• Exploratory prototyping where you have not committed to shipping → defer evals, but write down that you deferred them (see "when NOT to build eval infrastructure" below).

Definitions & Mental Model

• Golden set: a fixed collection of inputs with known-correct (or human-labeled) expected outputs, used to score the system.
• Baseline: the measured score of the current system before any change. No baseline → no claim of improvement.
• LLM-as-judge: using a model to score free-text outputs against a rubric. Powerful, but itself an LLM system that must be calibrated before trusted.
• Regression eval: the golden-set run wired into the change workflow so a prompt/model change cannot ship without a score.
• Calibration: measuring the judge's agreement with human labels on a sample before using its verdicts.

Mental model: an eval is to an LLM system what a test suite is to code — with one difference: code tests are binary and deterministic; evals are statistical. That means sample size matters (10 examples distinguish nothing), variance matters (run twice, compare the spread before crediting a 2% gain), and the honest unit of reporting is "X/N with these labels, on this dated set" — never "it's better now." Everything else in this skill is the transfer of validation-and-testing's risk-first discipline into statistical territory.

Core Methodology

1. Define what "works" means, mechanically, before tuning anything. One sentence per behavior: "Given a support ticket, the system outputs the correct queue label." If you cannot state it, you cannot measure it — go back to llm-system-design step 2.
2. Build the golden set BEFORE improving the system. Composition rule of thumb:
   • Real cases (the bulk): sampled from production/history, with correct outputs labeled by a human.
   • Adversarial cases: inputs designed to fool the system (ambiguous, misleading, prompt-injection-shaped).
   • Edge cases: empty, maximal-length, wrong-language, malformed — the same awkward-input discipline validation-and-testing demands of parsers.
   • Size: enough that a meaningful difference is visible above noise. Tens distinguish gross failures; low hundreds distinguish real improvements. State the N in every report; never imply significance a small N cannot carry.
   • Freeze and date-stamp the set. Changing the set and the system simultaneously destroys comparability.
3. Select the metric by output type (decision tree):

What shape is the system's output?
├─ Enum / classification (queue names, scores from a fixed set)
│    → exact-match accuracy per class; report the confusion pattern,
│      not just the total (one bad class hides in a good average).
├─ Structured output (JSON with fields)
│    → per-field assertions: pass/fail checks on required fields,
│      value ranges, cross-field consistency. Deterministic — no judge.
├─ Extraction / short factual answer
│    → normalized exact match or contains-match against labeled truth.
└─ Free text (summaries, explanations, reviews)
     → rubric-based LLM-as-judge — BUT:
       ├─ inject the ground truth into the judge's prompt (the evidence,
       │   the source, the reference answer) so the judge scores against
       │   supplied facts, never memory (mechanics:
       │   llm-integration-reliability step 6)
       ├─ enum-constrain the verdict ({"verdict":"pass|partial|fail"})
       └─ CALIBRATE before trusting (step 5)

4. Baseline first, then change ONE variable. Measure the current system on the golden set and record the score. Then change exactly one thing — the prompt, OR the model, OR the temperature, OR a pipeline step — and re-measure. Two changes at once means the result attributes to neither (the same one-variable law as debugging-playbook). Keep a plain log: date, variable changed, score, N.
5. Calibrate any LLM-as-judge before trusting it. Sample 20–50 judge verdicts, label the same items yourself (or with a human), and compute agreement. High agreement → trust with periodic spot-checks. Low agreement → fix the rubric/evidence injection, or fall back to human labels. An uncalibrated judge is a random-number generator with confidence. Re-calibrate whenever the judge's prompt or model changes.
6. Wire the eval into the change workflow. A prompt or model change ships only with a before/after score on the frozen set — the LLM analog of "no fix without a test" (change-control step 8). Prompts are code; their regression suite is the eval.
7. Report honestly. Every result carries: the N, the set version/date, the single variable changed, both scores, and an epistemic label if the margin is within noise ("no detectable difference" is a valid, publishable result — see research-methodology for label discipline).
8. Know when NOT to build eval infrastructure. One-off scripts, throwaway prototypes, and exploration sprints do not need golden sets — but write the deferral down ("no evals; do not ship without adding them"), so the prototype cannot silently become production. Proportionality is engineering-minimalism's law; it applies to evals too.

Eval readiness checklist

• "Works" defined mechanically per behavior
• Golden set exists: real + adversarial + edge cases, human-labeled, frozen, date-stamped, N recorded
• Metric matches output type (deterministic checks wherever possible; judge only for free text)
• Judge (if any): ground truth injected, verdict enum-constrained, calibrated against human labels
• Baseline score recorded before the first change
• Change log: one variable per entry, before/after scores
• Eval runs as a gate on prompt/model changes
• Reports state N and set version; noise-level differences reported as "no detectable difference"

Discovery & Audit Commands

Audit an existing codebase's evaluation posture:

# Is there any eval asset at all? (absence is the finding)
find . -type d -iname "*eval*" -o -type d -iname "*golden*" -o -type d -iname "*bench*" 2>/dev/null | grep -v node_modules
grep -rln -iE "golden|eval_set|test_cases|rubric" --include="*.py" --include="*.ts" --include="*.json" . | grep -v node_modules | head

# Are prompts versioned/diffable? (prompts-as-code prerequisite)
git log --oneline -- "*prompt*" "*PROMPT*" 2>/dev/null | head

# Is there a judge, and is its verdict enum-constrained?
grep -rn -iE "judge|grade|score|verdict|rubric" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20

# Any record of before/after comparisons?
grep -rn -iE "baseline|accuracy|pass_rate" --include="*.md" --include="*.py" . | grep -v node_modules | head

Failure Modes & Anti-patterns

Symptom	Mistake	Correction
"The new prompt seems better" ships	No golden set, no baseline	Freeze a set, measure before and after (steps 2, 4)
Improvement on the demo case, regressions everywhere else	Tuned against the case in front of you	The golden set is the target, never a single example
Prompt AND model changed, gain credited to the prompt	Two variables at once	One variable per comparison (step 4)
Judge passes everything	Uncalibrated LLM-as-judge, vague rubric	Calibrate against human labels; enum verdicts; inject ground truth (steps 3, 5)
Judge scores against its own memory of the topic	Ground truth not injected	Evidence in the judge's prompt (llm-integration-reliability step 6)
7/10 vs 8/10 declared a win	Sample-size dishonesty	State the N; tens detect gross failure only; report "no detectable difference"
Scores not comparable across weeks	Golden set silently edited	Freeze + version + date-stamp the set; changing it resets baselines
One awful class hidden by a good average	Aggregate-only reporting	Per-class/confusion reporting for classification
Free-text output "checked" by eyeballing	No metric for the output type	Decision tree in step 3; rubric judge with calibration
Prototype became production with zero evals	Deferral never written down	"No evals; do not ship without adding them" recorded at deferral time (step 8)

Worked Example

Task: improve the summary quality of a "summarize this incident report" feature.

1. Definition: a good summary names the affected system, the impact, and the current status, in ≤3 sentences, with no invented facts.
2. Golden set: 60 historical incident reports; a human writes reference summaries and labels the three required elements for each; plus 8 adversarial reports (buried impact statements, contradictory updates) and 6 edge cases (one-line reports, non-English, 10-page reports). Frozen as eval/incidents-v1, dated.
3. Metric: structured checks where possible — length ≤3 sentences (deterministic), three required elements present (judge with the source report injected, verdict per element {"present":"yes|no"}), no invented facts (judge with source injected, {"faithful":"yes|no"}).
4. Calibration: 30 judge verdicts spot-labeled by the engineer; agreement 27/30 → trusted with monthly spot-checks.
5. Baseline: current prompt scores 41/60 all-elements-present, 55/60 faithful.
6. One variable: restructure the prompt to demand the three elements explicitly (see prompt-and-context-engineering). Re-run: 52/60 present, 54/60 faithful. The faithfulness dip (55→54) is within single-item noise on N=60 — recorded as "no detectable change."
7. Gate: the eval command is now part of the PR checklist for any change under prompts/.

Repository Examples

Limited repository evidence available (as of 2026-07-04): none of the investigated repositories carries a golden set, judge calibration, or regression evals — evaluation is the least-evidenced discipline in the source corpus, which is precisely why this skill leans on the generic worked example above.

The closest real artifact: NYTW (~/NYTW) implements an LLM-as-judge with the right per-verdict mechanics — grading prompts inject the actual git diff + codebase evidence + user answer and demand strict enum JSON {"score":"correct|partial|incorrect","explanation"} validated on parse (as of 2026-07-04). What it lacks is the framework around the judge: no golden set of answer/score pairs, and no recorded calibration of the grader against human labels — the natural first application of this skill.

Validation Exercise (runnable in any repository with an LLM feature): pick one LLM behavior; write 20 input/expected-output pairs from real usage; measure the current system and record X/20 with today's date; change one variable; re-measure. If you cannot complete this exercise, the missing piece (undefined behavior, no labels, unparseable outputs) is itself the finding — route to llm-system-design or llm-integration-reliability accordingly.

Validation Criteria

You applied this skill correctly when:

1. A frozen, dated, human-labeled golden set exists with real + adversarial + edge cases and a stated N.
2. The metric matches the output type, and everything checkable deterministically is checked deterministically.
3. Any judge is calibrated (agreement number on record) and receives injected ground truth with enum verdicts.
4. Every claimed improvement cites: baseline score, new score, the single variable changed, and the set version.
5. A prompt/model change cannot ship without the eval score attached.
6. At least one report in the log says "no detectable difference" — evidence the noise floor is respected.

Provenance & Maintenance

• Sources: ~/NYTW (judge mechanics) — investigated 2026-07-04; the framework methodology is general practice, not derived from the example repos (their absence of evals is the documented evidence gap). Owner context: LLM output reliability is the owner's hardest problem; evals are its measurement arm. Skill authored 2026-07-06.
• Assumptions: golden-set size guidance (tens vs low hundreds) is a practical heuristic, not a statistical guarantee — for high-stakes decisions compute a real confidence interval. NYTW facts are point-in-time.
• Re-verification commands:

  grep -rn "correct|partial|incorrect" ~/NYTW/quiz --include="*.js" | head
  find ~/NYTW -iname "*golden*" -o -iname "*eval*" | head   # expect: nothing (the gap)
  

• Likely to drift: NYTW may gain a golden set (upgrade the example when it does); judge-calibration best practice evolves with model capability.
• Maintenance checklist:
  • Re-run re-verification; re-stamp the Repository Examples section.
  • If any owner repo gains an eval set, replace the Validation Exercise's prominence with the real case study.
  • Confirm cross-referenced skills still exist under their directory names.