Eval set

Eval Set

"It worked when I tried it" is not a test. The model that passed your one manual check yesterday gets upgraded next week and quietly starts doing the wrong thing, and nothing tells you.

But you almost certainly don't have, and don't need, a pytest suite over a Claude pipeline. The verification that actually holds in production is simpler and runs inline before anything ships: the output's shape is a contract, a short list of invariants must be true, and a judge scores whether the work is complete. This skill writes those three things for your case.

Build the checks, then stop.

Inputs (ask for whatever is missing)

• What's being checked (required): the skill, agent, or automation, and what it produces.
• Its contract: how you'd know it worked. If a build-spec exists, pull the acceptance criteria. If not, ask "what must be true about the output every time?" and write that down first.
• Optional, makes it sharper: the output's data shape, the worst output you've seen, and any bug it shipped before.

The method

1. Make the shape the first check. Write (or point at) the schema the output must satisfy — the fields, the types, the allowed enum values. An output that fails the schema fails, full stop, and you catch it at build/run time instead of in front of a client. Most silent breakage is a shape violation: an invented enum label, a missing field, a null where a value was promised.

2. Write the pre-ship invariant checklist. The short list of things that must be true before this output is allowed to ship or send, each a plain yes/no gate. These are the ones that have actually burned people: enum values are real and not invented, no fabricated URLs or IDs, an email is verified before it's used, the qualifying gate was passed, no required field is empty. Write the invariants for your output. This list is the heart of the eval — it's cheap, it's specific, and it catches the failures that matter.

3. Add a completeness judge. One model pass that scores the output against a short rubric (1-10) and returns the specific things still missing, so "good enough to ship" has a number, not a vibe. Set the gate (e.g. ">= 8 or it loops once and tries again"). Make a parse failure of the judge default to pass, not an infinite loop — a broken judge should never wedge the pipeline.

4. Capture the regressions. Every bug already found becomes one fixed case with its input and the expected correct output. These never get deleted. They're the only part that grows over time, and they're what stops you shipping the same break twice.

5. State the ship bar. Combine the three: schema valid, every must-pass invariant green, completeness >= the gate, all regression cases pass. That sentence is now your definition of "done."

6. Make it re-runnable. Note how to run the whole thing in one go and where to record the result, so checking it after a model upgrade is one command. The whole point is to re-run it the day the model changes under you.

Output

Produce a single checks.md: the schema (or a reference to it), the pre-ship invariant checklist as yes/no gates, the judge rubric and its numeric gate, the regression cases, and the ship-bar sentence beneath. If the invariants are clean enough to assert in code, note which are a schema check vs a runtime assertion vs a human read.

Then stop. You now have something to run before each ship — and the honest version of an eval set for a system whose work is done by a model.