Evaluating llm systems

Evaluating LLM Systems

Overview

LLM systems fail the same way training runs do: silently, with plausible output. The difference is that most teams never build the instrument that would show it.

Core principle: Look at your data, then build evals specific to the failure modes you actually find. Generic metrics measure nothing about your product.

Source: Hamel Husain and Shreya Shankar on applied LLM evals.

"Don't rely on generic evaluation frameworks to measure the quality of your AI. Instead, create an evaluation system specific to your problem."

Error Analysis Comes First

Before writing a single assertion:

1. Collect ~100 real traces (input, retrieved context, output, tools called).
2. Read them and write a free-form note on each failure — no fixed categories yet.
3. Group the notes into recurring failure modes.
4. Count each mode. The counts set your priorities.

Categories invented before reading data describe your imagination, not your product.

"You must remove all friction from the process of looking at data."

Build the trace viewer early. Anything that makes inspection annoying means it stops happening.

Three Levels

Run cheap ones constantly, expensive ones rarely.

Level	What	Cadence
1 — Assertions	Deterministic checks: valid JSON, required field, no placeholder, tool called, length bound	Every change, in CI
2 — LLM judge / human	Subjective quality on a curated set	Set cadence
3 — A/B in production	Real users, real outcomes	After significant changes

Level 1 catches most regressions for almost no cost. Write assertions for the specific failure modes error analysis found — not generic ones.

A 100% pass rate is not the goal. Your target pass rate is a product decision.

Validating The Judge

An unvalidated LLM judge is a random number generator with good manners. Before trusting it:

1. Hand-label 25–50 examples binary — good/bad. Not 1–5; graded scales produce inconsistent labels from humans and judges alike.
2. Run the judge on the same set.
3. Compare precision and recall separately, not raw agreement.
4. Iterate the judge prompt until it aligns, then re-measure on fresh examples.

"Using agreement as a metric... can be misleading when classes are imbalanced. Instead, you should typically measure precision and recall separately."

Re-validate whenever you change the judge prompt or the model behind it. A judge is a model, and it drifts.

Mapping The Training Recipe

The ml-superpowers:building-the-training-skeleton checks all have eval analogues:

Check	LLM eval version
Verify loss @ init	Score a known-perfect answer (must be max) and an empty one (must be floor)
Input-independent baseline	Answer with no retrieved context. If it matches the full system, retrieval contributes nothing
Overfit one batch	Hand-grade 10 items; the harness must agree exactly
Human baseline	Two humans label the same set — their disagreement is your noise floor and your ceiling
Full test set	Grade every item, not a sample

The human-disagreement number matters most: you cannot detect an improvement smaller than the gap between two human labelers.

Avoiding Eval Overfitting

Tuning prompts against a 20-item set until the number rises is overfitting with extra steps. Hold out a set you look at rarely and never tune against, exactly as with a test split.

Common Mistakes

Mistake	Fix
Generic metrics (helpfulness, coherence)	Assertions specific to your observed failures
Categories chosen before reading traces	Read first, categorize after
Trusting an unvalidated judge	Measure precision/recall against human labels
1–5 quality scales	Binary. Graded scales are inconsistent.
Only LLM-judge evals	Most regressions are caught by cheap assertions
Tuning against the whole eval set	Hold one out

Red Flags

• An eval suite written before anyone read production traces
• A judge whose agreement with humans has never been measured
• Prompt changes shipped because the output "looks better"
• No assertion-level tests in CI