Eval driven development
How this repo does eval-driven development (EDD) for Smithers workflows — write the failing suite first, build until green, validate on a holdout, then optimize. Use when adding evals to a workflow, changing a prompt/model/graph that has a suite, setting up a dev/holdout split, or wiring eval gates into CI. For suite-authoring mechanics see skills/eval-writer/SKILL.md; this skill is the loop and the discipline.From its SKILL.md
npx -y skills add smithersai/smithers --skill eval-driven-developmentAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
SKILL.md
7.0 KB, as published. Nobody here has run it
Eval-Driven Development
One passing run proves nothing. EDD is the loop that makes a workflow's quality repeatable: encode acceptance criteria as a failing eval suite before building, go green, and never tune against the holdout. A change to a prompt, model, schema, or graph edge is "done" when the suite says so, not when a run looks good.
The loop
- Red first. Before building or changing the workflow, write cases from the acceptance criteria and run the suite. It must fail (or lack coverage) — that failure is the spec. If it passes already, your cases are too weak: add the adversarial one that trips a weak run.
- Build until green. Change the prompt, schema, agent, or graph. Re-run the dev suite. Iterate against the report, not against vibes.
- Validate on the holdout. Once dev is green, run the
*-holdoutsuite. Holdout cases are never used to tune prompts, pick models, or accept an optimization — they exist to catch overfitting to the dev split. - Optimize (optional).
smithers optimizeruns the dev suite twice (baseline + GEPA-patched) and writes an artifact. Accept it only if the holdout suite, run with--optimization <artifact.json>, does not regress. - Gate in CI. The
evalcommand exits0when all cases pass,1on any failure,4for an invalid case file. Wire the non-zero exit into CI; a suite nobody runs is documentation.
Repo conventions
Suites live in .smithers/evals/ as JSONL, one case per line:
<suite>.jsonl— the dev split; tune against this.<suite>-holdout.jsonl— the holdout split; validate only.- Every case carries
metadata:split(dev/holdout),category(the behavior class, e.g.durable-routing), andcriterion(the plain-English acceptance criterion the case encodes). - Reports land at
.smithers/evals/<suite>.jsonby default; sweep runs go to.smithers/evals/reports/<label>/. .smithers/evals/run-sweep.sh <label> [concurrency]runs every seeded suite (dev + holdout) and prints a per-suite pass summary — the pattern for a full regression check before shipping a harness change.
Case shape:
{"id":"etl-silent-row-drop","input":{"prompt":"..."},"expected":{"status":"finished","outputContains":{"recommend":[{"recommendedWorkflow":"debug"}]}},"metadata":{"split":"dev","category":"durable-routing","criterion":"A root-cause defect lands on debug."}}
expected.status:finished,failed,waiting-approval, etc.expected.outputContains: recursive partial match; arrays match by containment. The usual choice — key it to the output schema's load-bearing typed fields, never to prose.expected.output: exact match (brittle; rare).expected.errorContains: for adversarial cases that should fail.judge:{"instructions": "...", "threshold": 0.7}for semantic checks (tone, completeness) where JSON match can't work. Deterministic assertions AND the judge must both pass. Select the judge with--judge-provider/--judge-model.
Adding evals to a workflow
-
List the acceptance criteria. Turn each into ≥1 case: a happy path, the quality-gate criterion itself, and an adversarial/edge case.
-
Write
.smithers/evals/<suite>.jsonlwithmetadata.split: "dev", and a<suite>-holdout.jsonlof fresh cases drawn from the same criteria but not shared with dev. -
Dry-run the shape before spending:
bun apps/cli/src/index.js eval .smithers/workflows/<wf>.tsx \ --cases .smithers/evals/<suite>.jsonl --suite <suite> --dry-run -
Run it (in-repo use
bun apps/cli/src/index.js eval; consumers usebunx smthrs eval):bun apps/cli/src/index.js eval .smithers/workflows/<wf>.tsx \ --cases .smithers/evals/<suite>.jsonl --suite <suite> \ --concurrency 4 --force -
Read the report at
.smithers/evals/<suite>.json; fix the workflow, not the test — unless the case encodes the criterion wrong, or the failure is environmental (see the classify rule below). -
For graded, non-binary quality, attach scorers (
faithfulness,relevancy,schemaAdherence,llmJudge) to the load-bearing<Task>and read them withsmithers scores <run-id>. Assertions are the hard gate; scorers are the trend. -
Add the suite to the sweep script / CI so it runs on every relevant change.
Discipline rules
- Classify red before acting. Every failed case is one of three things:
a product bug (fix the workflow), a wrong case (fix the case, with spec-
change scrutiny), or a harness/environment fault (fix the harness). The
report marks known environment faults INCONCLUSIVE and
smithers evalexits5(not1) when they are the only reds: on that signal repair the harness and never touch the workflow. A red that could not have observed the workflow (connection refused, TLS failure, network denied, missing binary, OOM, rate limit) is not evidence against it. - Green ratchet. If a case that passed in round N fails in round N+1 and the only intervening change was to the harness or environment, that is a harness regression: revert the harness change. Never widen a suite's acceptance criteria while it is red; get back to the last green slice first.
- Circuit breaker. After 3 consecutive rounds with zero net new green
cases, stop iterating. Change strategy (gather evidence, widen scope) or
escalate to a human via
smithers ask-humanwith what you know. More rounds against the same failure signature is an incident, not progress. - Never tune against holdout. No prompt edits, model picks, or optimization acceptance based on holdout results. Green holdout = ship; red holdout = your dev split stopped representing the criteria — fix the split, not just the workflow.
- Failed case → first-class bug. Reproduce with the run ID in the report
(
smithers inspect <run-id>,smithers why <run-id>), fix, re-run. - Suite changes are spec changes. Editing a case to make a run pass rewrites the acceptance criteria; treat it with the same scrutiny as the workflow change itself.
- Keep suites cheap to run. Use
--max-casesfor smoke checks,--concurrencyfor throughput, and--rootto sandbox tool execution.
Pointers
- Authoring mechanics (scorer wiring,
eval-authorbootstrap):skills/eval-writer/SKILL.md - CLI surface:
smithers-evalandsmithers-optimizeskills; full option list indocs/guides/evals-quickstart.mdx - Scorer API reference:
docs/reference/scorers(source:packages/scorers) - Eval engine:
packages/engine+apps/clieval/optimizecommands