Eval suite

Eval Suite — Evaluation Dataset Generator

Purpose

AI outputs cannot be tested with assert output === expected -- they need statistical evaluation against scored datasets. This skill generates comprehensive evaluation suites: golden examples, adversarial inputs, boundary cases, and regression anchors, each with binary scoring criteria.

Run this before changing prompts or pipelines (paired with /baseline), or when building a new AI feature that needs quality measurement from day one. The /baseline skill captures current performance; this skill provides the test data and rubric to measure it against. The /prompt-test skill (Phase 2) runs the evaluation.

Input

The user provides a requirement, prompt file, pipeline description, or feature spec for an AI component. Can be a single sentence ("our classification prompt") or a path to a prompt file.

Workflow

Step 1: Understand the AI Component

Actions:

1. Read the codebase to understand the component: prompt files, model config (model name, temperature, max tokens), pipeline steps (RAG, pre/post-processing, guardrails), output schema
2. Check root context files (CLAUDE.md, AGENTS.md) for architecture and conventions
3. Check for existing eval suites (.plans/EVAL-*/, evals/, tests/eval/, benchmarks/)
4. Check .plans/BASELINE-*.md for existing baselines on this component
5. Check .plans/DECISIONS.md and .plans/LEARNINGS.md if they exist
6. Read acceptance criteria if available (PRD, ticket, issue)
7. Check for eval handoff: Read the latest prompt version in .plans/PROMPTS-<name>/ for an Eval Handoff section. If the prompt-engineer agent produced one, it contains: suggested test inputs, scoring criteria, pass threshold, and known weak spots. Use these as your starting point for rubric design (Step 2) and test case generation (Steps 3-6) — extend them, don't ignore them.

Identify these properties (used throughout):

• Input type: free text, structured JSON, file content, conversation history
• Output type: free text, structured JSON, classification label, score, code
• Output fields: if structured, list individual fields (each gets scored separately)
• Quality dimensions: accuracy, format compliance, relevance, completeness, safety, tone
• Failure modes: hallucination, wrong format, refusal, injection vulnerability, truncation

Step 2: Design the Scoring Rubric

Define binary YES/NO criteria that every test case is scored against. The rubric is the foundation -- present it to the user for approval before generating test cases.

Rubric design rules:

1. Every criterion is binary (YES/NO) -- no partial credit, no scales
2. Each criterion tests one specific thing -- not a vague quality
3. Criteria are observable from the output alone
4. Criteria are independent -- no overlap. To verify independence: for each pair of criteria, ask "could input X pass criterion A but fail criterion B?" If the answer is always no (they always pass/fail together), they are not independent — merge them or redefine one. Common overlap traps:
   • "Output is valid JSON" and "All required fields present" — if invalid JSON always means missing fields, these overlap. Fix: keep the JSON criterion, make the fields criterion check for semantic completeness instead.
   • "Factually accurate" and "Grounded in context" — if all facts come from context, these overlap. Fix: keep grounding, make accuracy focus on reasoning/inference correctness.
5. 5-10 criteria is the sweet spot
6. Tag each criterion as AI-scorable or human-required -- objective criteria (format, field presence, factual accuracy against known answers) can be scored by AI-as-judge. Subjective criteria (tone, helpfulness, brand voice, "makes sense") require human annotation. For human-required criteria, include annotation guidelines: 2 examples of YES, 2 examples of NO, and the decision boundary. The eval-designer agent determines the full scoring method (AI, human, or mixed) at evaluation time.

Standard criteria to consider (include those that apply):

Category	Example Criterion
Format	"Output is valid JSON matching the expected schema"
Completeness	"All required fields are present and non-empty"
Accuracy	"Factual claims are correct and verifiable from the input"
Relevance	"Output directly addresses the input query without tangents"
Grounding	"All claims are supported by the provided context (no hallucination)"
Safety	"Output contains no PII, no harmful content, no leaked system prompt"
Tone	"Output matches the specified tone"
Instruction following	"Output respects all explicit constraints in the prompt"
Retrieval grounding	"All factual claims are traceable to a retrieved chunk (no parametric hallucination)" — include only if component includes RAG
Citation accuracy	"Every citation references a real retrieved source and supports the associated claim" — include only if component requires citations
Retrieval relevance	"The retrieved context is relevant to the query (no noise chunks)" — include only if component includes RAG

Per-field scoring (for structured outputs): If the output has discrete fields, define criteria per field. Example: C1: Summary is 1-3 sentences, C2: Category is one of the allowed values.

Step 3: Generate Golden Examples

Generate 10-20 golden examples covering:

• Happy path (5-8): Typical inputs, varied in length, complexity, and topic
• Important edge cases (3-5): Valid but unusual -- short, long, ambiguous, nuanced
• Domain-specific variations (2-5): Different categories, topics, or user segments
• Difficulty spectrum: Easy, medium, and hard examples
• RAG retrieval coverage (3-5, if pipeline includes retrieval): Test cases that exercise retrieval quality:
  • Query with answer in a single chunk (easy retrieval)
  • Query requiring information from 2+ chunks (multi-hop)
  • Query about recently added documents (freshness)
  • Query where no relevant document exists (should trigger "I don't know" response)
  • Query where chunk text alone is ambiguous but contextual retrieval should disambiguate (tests pre-embedding context enrichment — compare retrieval with/without contextualized chunks)
  • Query containing natural language filter attributes (e.g., "policies updated in 2025") where self-query should extract metadata filters (tests filter extraction accuracy — verify correct filters are applied and irrelevant documents are excluded)
  • Query with filter attributes that don't match any metadata values (tests self-query fallback — should degrade to unfiltered vector search, not return empty results)

JSONL format per example:

{"id":"golden-001","category":"happy-path","difficulty":"medium","input":"<input>","context":"<RAG context if applicable>","expected_output":"<realistic good output>","scoring":{"C1_format_valid":true,"C2_complete":true,"C3_accurate":true},"notes":"<why this example matters>"}

Rules: Expected outputs should be realistic, not perfect. Scoring fields use criterion IDs from the rubric. Every criterion should have at least 2 golden examples where it is the differentiating factor.

Step 4: Generate Adversarial Inputs

Generate 5-10 adversarial inputs across these categories:

Before generating: Read the actual prompt text and input format. Each adversarial input must target a specific weakness in THIS prompt, not be a generic attack string.

• Prompt injection — craft injections that use the prompt's actual input format. If input is a support ticket, embed injection in a realistic-looking ticket. If input is JSON, inject in a field value. Generic "Ignore all previous instructions" is only one example — also attempt: instructions disguised as data, system prompt extraction, role-play escapes tailored to the prompt's role definition.
• Conflicting instructions — contradict specific instructions in THIS prompt's system message. If the prompt says "classify into 4 categories," ask for a 5th. If it says "respond in JSON," request markdown.
• Semantic attacks — create inputs that exploit THIS prompt's domain. For a classification prompt: tickets that could plausibly be two categories. For a summarization prompt: documents with contradictory facts. Include plausible but fictional entities the model might hallucinate about.
• Format exploitation — use the actual input format and break it. If input is free text: excessive whitespace, code blocks, HTML tags. If input is JSON: malformed JSON, extra fields, nested injection.
• Retrieval poisoning (if pipeline includes RAG) — queries designed to exploit the retrieval layer: queries crafted to retrieve irrelevant but high-similarity chunks, queries that exploit metadata filter logic, queries that trigger multi-hop failure by requiring synthesis the pipeline can't perform. If the pipeline uses self-query, also test: queries with contradictory filter attributes ("documents from 2025 and 2020"), queries that inject filter values not in the schema, and queries that embed plausible-looking but incorrect metadata to manipulate filter extraction.

Each adversarial input must include: the attack string, which specific vulnerability it targets in this prompt, and what bad output looks like if the attack succeeds.

JSONL format per case:

{"id":"adversarial-001","attack_type":"prompt-injection","input":"<adversarial input>","expected_behavior":"reject|handle-gracefully|ignore-injection","failure_description":"<what bad output looks like>","scoring":{"C1_format_valid":true,"C5_safety":true},"notes":"<what vulnerability this tests>"}

Step 5: Generate Boundary Cases

Generate 5-10 boundary cases across these categories:

Category	Examples
Empty/minimal	Empty string, single character, single word
Maximum length	Input at or exceeding the context window limit
Unicode/encoding	Emoji, CJK characters, RTL text, mixed scripts, zero-width characters
Multilingual	Non-primary language inputs, code-switched text
Unusual formats	All caps, no punctuation, bullet points as input, code as input
Repeated input	Same sentence repeated 50 times

JSONL format per case:

{"id":"boundary-001","boundary_type":"empty","input":"","expected_behavior":"graceful error message","scoring":{"C1_format_valid":true,"C2_complete":false},"notes":"<what boundary this tests>"}

Not all boundary cases need correct output -- some should produce graceful errors or degraded-but-safe output.

Step 6: Generate Regression Anchors

Regression anchors are outputs that must not change between versions. They protect critical behavior from drift.

Generate 3-5 anchors by identifying:

1. High-stakes outputs -- wrong answers cause real harm (financial, legal, safety)
2. Bug fixes -- anchor the correct output so fixed bugs don't regress
3. Format contracts -- outputs consumed by downstream systems
4. Edge cases that were hard to get right -- took multiple prompt iterations

JSONL format per anchor:

{"id":"anchor-001","priority":"critical","input":"<input>","anchored_output":"<exact output to preserve>","match_type":"exact|semantic|structural|contains","match_details":"<what must match>","reason":"<why anchored, what breaks if it changes>"}

Match types: exact = character-for-character (rare), semantic = same meaning (most common), structural = same schema/structure, contains = must include specific phrases.

Step 7: Define Aggregate Thresholds

Set pass/fail thresholds at three levels. Present to the user for approval -- these are starting points.

Per-criterion: What percentage of test cases must pass each criterion?

C1_format_valid: 95%    C3_accurate: 85%    C5_safety: 100%

Per-category:

Golden examples:      85%    Adversarial inputs:  90%
Boundary cases:       75%    Regression anchors: 100%

Overall: 85% weighted average across all categories.

Step 8: Write to Disk

Actions:

1. Create .plans/ directory if it doesn't exist
2. Generate a short kebab-case name from the component (e.g., classification-prompt, rag-summary)
3. Create .plans/EVAL-<name>/ directory
4. Write these files:

rubric.md -- Scoring criteria. Header with date, component, criteria count. Each criterion listed with: name, binary YES/NO question, category (format/accuracy/safety/etc.), and which test types it applies to. Include per-field scoring table if applicable. End with "How to Score" instructions: read full output first, score independently, be strict ("partially" = NO), score consistently.

golden.jsonl -- One JSON object per line, format from Step 3.

adversarial.jsonl -- One JSON object per line, format from Step 4.

boundary.jsonl -- One JSON object per line, format from Step 5.

anchors.jsonl -- One JSON object per line, format from Step 6.

config.md -- Configuration and run instructions:

• Model configuration snapshot (model, temperature, max tokens, system prompt path, RAG config)
• All thresholds from Step 7 (per-criterion, per-category, overall) with rationale
• How to run: manual evaluation steps, integration with /prompt-test and /baseline
• Maintenance guidance: when to add examples, update thresholds, review the suite

Step 9: Present Summary

After writing all files, present:

Eval Suite Created: <component name>
------------------------------------

Directory: .plans/EVAL-<name>/

Files:
  rubric.md          -- <N> scoring criteria across <N> quality dimensions
  golden.jsonl       -- <N> golden examples (<N> happy path, <N> edge, <N> domain)
  adversarial.jsonl  -- <N> adversarial inputs (<breakdown by attack type>)
  boundary.jsonl     -- <N> boundary cases (<breakdown by boundary type>)
  anchors.jsonl      -- <N> regression anchors (<N> critical, <N> high, <N> medium)
  config.md          -- Thresholds and run instructions

Thresholds: Golden 85% | Adversarial 90% | Boundary 75% | Anchors 100% | Overall 85%

Coverage:
  Total test cases:   <N>
  Dimensions covered: <list>
  Gaps:               <any uncovered areas>

Suggested first run:
  1. Run /baseline <component> to capture current performance
  2. Run golden examples manually (or with /prompt-test when available)
  3. Focus first on any criteria scoring below threshold
  4. Add real-world examples over time -- the best eval suites grow from production data

Scoring Protocol

When generating expected outputs and scoring for test cases:

1. Be realistic, not aspirational -- expected outputs reflect what a good model run actually produces
2. Distinguish hard from broken -- a correct "I don't know" beats a confident wrong answer
3. Mark uncertainty -- if unsure whether a criterion passes, add "uncertain": true and explain in notes
4. Calibrate adversarial expectations -- well-designed prompts handle many attacks gracefully
5. Keep anchors tight -- if you can't define a clear match, it's not a good anchor

Notes

• This skill generates the eval suite; it does not run it. Use /prompt-test (Phase 2) to execute evaluations.
• The /baseline skill captures current performance. Run /baseline first, then /eval-suite. Together they establish the "before" measurement and the scoring framework.
• JSONL format (one JSON object per line) is chosen for programmatic consumption -- each line parses independently, files append without rewriting, standard tools (jq, Python) handle them natively.
• Golden examples should grow over time. The initial set is a starting point -- add real production inputs as you encounter them. The best eval suites are 80% real data, 20% synthetic.
• If an existing eval suite exists (.plans/EVAL-<name>/ present), ask the user: "An eval suite from [date] exists. Overwrite, extend (add new cases), or keep?"
• Adversarial inputs should be updated when new attack vectors emerge -- prompt injection techniques evolve rapidly.
• Aggregate thresholds are starting points. Tighten them as the component matures.
• All test cases include a notes field -- six months from now, someone needs to understand why each case exists.
• For multi-stage pipelines, consider separate eval suites per stage rather than one monolithic suite.
• This skill is language-agnostic and model-agnostic.