Agent eval workflow

Agent Evaluation Workflow

These patterns address the challenges that separate production agents from abandoned prototypes. Agent failures are nondeterministic, nuanced, and often invisible. These patterns give you a systematic approach to finding and fixing them.

Core Principle

Raw accuracy metrics tell you something changed, but not why it changed or what to do about it. You need a complete workflow from failure classification through targeted improvement.

The Eval Workflow (Overview)

List Failure Modes --> Define Business Metrics --> Cross-Reference
    --> Build Eval Suite --> Have SMEs Label Data
    --> Create Datasets from Production --> Evaluate with LLM-as-Judge
    --> Iterate Against Evals --> Repeat

Pattern 10: List Failure Modes

Problem: AI outputs are nondeterministic

An LLM returns different results for the exact same input. Measuring quality is much harder than traditional pass/fail testing. You need to understand not just that it failed, but why.

Solution: Classify failure reasons

Create a classification process that categorizes both which failures occur and why:

• Data quality -- bad input, missing information, extraction errors
• Reasoning failure -- incorrect logic, wrong conclusions from correct data
• Domain-specific rules -- misapplying guidelines, missing edge cases

This is called "interpretability" in ML.

Example: Medical agent review

A medical AI reviewing surgery recommendations watches for three failure modes:

1. Medical Record Extraction -- fails to extract correct clinical info from patient records
2. Clinical Reasoning -- inappropriate medical logic or failure to weigh clinical factors
3. Rules Interpretation -- misapplying insurance guidelines or coverage criteria

Each failure mode leads to different fixes: extraction failures need better RAG, reasoning failures need prompt improvements, rules interpretation needs more domain context.

Pattern 11: List Critical Business Metrics

Problem: Testing the right thing

Engineering evals don't necessarily capture whether business objectives are met. How do you know you're testing what matters?

Solution: Measure what matters with three metric types

1. Accuracy metrics (baseline)

   • False positive rate, false negative rate, overall accuracy
   • These are your starting point

2. Domain-specific outcome metrics

   • Legal: missed critical terms
   • Finance: dollar loss prevention
   • Education: test score improvement
   • Healthcare: false approval rate

3. Human team metrics

   • How would you measure a human doing this job?
   • Time saved, tickets resolved, customer satisfaction

Example

A medical claim review agent's north star metric is "false approvals." Over the course of a project they improved overall accuracy from 95% to 99% by focusing specifically on this metric.

Pattern 12: Cross-Reference Failure Modes and Success Metrics

Problem: Turning metrics into insights

Evals give you numbers, but numbers alone aren't actionable. You need to connect failure modes to business metrics to prioritize improvement work.

Solution: Cross-reference and prioritize

1. Expert review (SME): Subject matter experts review live production outputs and classify failure modes
2. Failure mode prioritization (PM): Product managers cross-reference metrics and failure modes, then set performance targets (e.g., reduce failure rate from 10% to 8% on a specific dataset)
3. Agent performance improvement (Eng): Engineers experiment with different approaches (prompting, models, etc.) using prebuilt failure-mode-specific datasets
4. Validation and deployment (PM): PMs review changes against past production data and make go-live decisions

Key Insight

This creates a tight feedback loop between the people who understand the domain (SMEs), the people who set priorities (PMs), and the people who make technical changes (Engineers).

Pattern 13: Iterate Against Your Evals

Problem: Detecting change impacts

You can't always tell if a change improved your agent or made it worse. What feels like an improvement might introduce new problems.

Solution: Measure against a test dataset

• Benchmarks are the difference between engineering and experimentation
• Measure in CI -- guard against code regressions to agent accuracy
• Establish standards -- don't merge changes that reduce overall accuracy without offsetting improvements
• Give engineers a clear starting point, specific target, and a way to test progress

Example

"Right now we're getting 95% accuracy on this clinical reasoning dataset. Work on this until you get to 99%." The engineer now has:

• A clear starting point (95%)
• A specific target (99%)
• A way to test progress (the benchmark dataset)
• Confidence they're moving in the right direction

Pattern 14: Create an Eval Test Suite

Problem: Tracking performance

Evaluating agents is as important as unit testing traditional software. A fix here can break something over there.

Solution: A test suite of evaluation criteria

Build your eval test suite from multiple sources:

1. Synthetic datasets -- use an LLM to generate realistic scenarios
2. Internal user data -- real interactions from trusted users
3. SME golden answers -- expert-labeled input/output pairs in CSV format
4. Choose metrics -- like a TA grading a paper: completeness, fairness, accuracy

A test suite is often just one test running in a loop with LLM-as-judge comparing agent answers to your baseline/benchmark.

Progression

Start with synthetic data, then replace with real production data over time. This ensures your datasets represent the actual input distribution you're seeing.

Pattern 15: Have SMEs Label Data

Problem: Data that reflects reality

Software engineers are NOT your domain experts. You don't want them judging whether a medical coverage decision is correct.

Solution: Domain experts label output data

• Have SMEs curate an initial "ground truth" dataset during prototyping
• Continue periodic SME reviews after going to production
• How to label: Include an overall grade, category tags, and optional subjective feedback. Use multiple annotators + inter-rater reliability metrics.
• What to label: Initial eval datasets, sampled live data (flagged by guardrails, CI failures, or automated evaluators)
• Give SMEs good tools: An intuitive review UI with complete data. They should see the full trace (user input, tool calls, LLM reasoning), not just the final output.

Example

A medical claim review company built an internal dashboard where clinicians review cases, mark outcomes as correct/incorrect, select specific failure modes, and have a "domain knowledge addition" button for new failure modes.

Pattern 16: Create Datasets from Production Data

Problem: Real data is messy

You need to transform messy production logs into clean, labeled datasets for evaluation.

Solution: Versioned evaluation datasets

Use LLM observability tools to:

• Log production generations for quality assessment (manually or with LLM-as-judge)
• Store evaluation test cases (instead of managing JSONL/CSV files)
• Store human SME reviews (thumbs up/down) to surface new test cases
• Version datasets and store them at cloud scale

Dataset structure

Three top-level fields: inputs, expected outputs, and metadata (timestamp, knowledge source, user type, etc.)

Pattern 17: Evaluate Production Data

Problem: Synthetic data doesn't match production reality

Users come up with new and different types of queries over time. Your CI benchmark must stay representative.

Solution: LLM-as-judge on live production data

1. Define an evaluation prompt -- specific criteria for the LLM judge to score against
2. Choose a scoring method:
   • Binary grading (pass/fail) -- simplest, recommended
   • Categorical grading (good/fair/poor)
   • Avoid numerical scoring (1-10) -- LLMs are better at literacy than numeracy
3. Define sampling frequency -- you don't need to evaluate every single response
4. Combine with human evaluation -- automated evals + SME reviews catch the most issues

Example: Legal agent accuracy

A law firm's contract analysis agent was trained for NDAs and service agreements. Employees started using it for international contracts and merger docs -- results were suboptimal. They:

1. Evaluated production data with binary grading (compliant/noncompliant)
2. Added categorical risk assessment (high/medium/low)
3. Had partners review and rate agent recommendations
4. Cross-referenced to identify gaps in cross-jurisdictional legal reasoning

Gotchas

• Don't outsource data labeling to developers when you need domain expertise
• Start with 2-3 failure modes, not 20. Expand as you learn.
• Binary/categorical grading outperforms numerical scoring for LLM-as-judge
• Replace synthetic eval data with real production data as soon as possible
• A single north star metric (like "false approvals") focuses the team better than a dashboard of 15 metrics
• If accuracy plateaus, the issue might be data quality or retrieval, not prompting
• Evals in CI prevent regressions -- don't skip this step

For detailed eval implementation templates, see references/eval-templates.md.