agentsclimarketplace

Eval driven development

Skill smithersai/smithers/skills/eval-driven-development

How this repo does eval-driven development (EDD) for Smithers workflows — write the failing suite first, build until green, validate on a holdout, then optimize. Use when adding evals to a workflow, changing a prompt/model/graph that has a suite, setting up a dev/holdout split, or wiring eval gates into CI. For suite-authoring mechanics see skills/eval-writer/SKILL.md; this skill is the loop and the discipline.From its SKILL.md

Install
npx -y skills add smithersai/smithers --skill eval-driven-development

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

7.0 KB, as published. Nobody here has run it

Eval-Driven Development

One passing run proves nothing. EDD is the loop that makes a workflow's quality repeatable: encode acceptance criteria as a failing eval suite before building, go green, and never tune against the holdout. A change to a prompt, model, schema, or graph edge is "done" when the suite says so, not when a run looks good.

The loop

  1. Red first. Before building or changing the workflow, write cases from the acceptance criteria and run the suite. It must fail (or lack coverage) — that failure is the spec. If it passes already, your cases are too weak: add the adversarial one that trips a weak run.
  2. Build until green. Change the prompt, schema, agent, or graph. Re-run the dev suite. Iterate against the report, not against vibes.
  3. Validate on the holdout. Once dev is green, run the *-holdout suite. Holdout cases are never used to tune prompts, pick models, or accept an optimization — they exist to catch overfitting to the dev split.
  4. Optimize (optional). smithers optimize runs the dev suite twice (baseline + GEPA-patched) and writes an artifact. Accept it only if the holdout suite, run with --optimization <artifact.json>, does not regress.
  5. Gate in CI. The eval command exits 0 when all cases pass, 1 on any failure, 4 for an invalid case file. Wire the non-zero exit into CI; a suite nobody runs is documentation.

Repo conventions

Suites live in .smithers/evals/ as JSONL, one case per line:

  • <suite>.jsonl — the dev split; tune against this.
  • <suite>-holdout.jsonl — the holdout split; validate only.
  • Every case carries metadata: split (dev/holdout), category (the behavior class, e.g. durable-routing), and criterion (the plain-English acceptance criterion the case encodes).
  • Reports land at .smithers/evals/<suite>.json by default; sweep runs go to .smithers/evals/reports/<label>/.
  • .smithers/evals/run-sweep.sh <label> [concurrency] runs every seeded suite (dev + holdout) and prints a per-suite pass summary — the pattern for a full regression check before shipping a harness change.

Case shape:

{"id":"etl-silent-row-drop","input":{"prompt":"..."},"expected":{"status":"finished","outputContains":{"recommend":[{"recommendedWorkflow":"debug"}]}},"metadata":{"split":"dev","category":"durable-routing","criterion":"A root-cause defect lands on debug."}}
  • expected.status: finished, failed, waiting-approval, etc.
  • expected.outputContains: recursive partial match; arrays match by containment. The usual choice — key it to the output schema's load-bearing typed fields, never to prose.
  • expected.output: exact match (brittle; rare).
  • expected.errorContains: for adversarial cases that should fail.
  • judge: {"instructions": "...", "threshold": 0.7} for semantic checks (tone, completeness) where JSON match can't work. Deterministic assertions AND the judge must both pass. Select the judge with --judge-provider / --judge-model.

Adding evals to a workflow

  1. List the acceptance criteria. Turn each into ≥1 case: a happy path, the quality-gate criterion itself, and an adversarial/edge case.

  2. Write .smithers/evals/<suite>.jsonl with metadata.split: "dev", and a <suite>-holdout.jsonl of fresh cases drawn from the same criteria but not shared with dev.

  3. Dry-run the shape before spending:

    bun apps/cli/src/index.js eval .smithers/workflows/<wf>.tsx \
      --cases .smithers/evals/<suite>.jsonl --suite <suite> --dry-run
    
  4. Run it (in-repo use bun apps/cli/src/index.js eval; consumers use bunx smthrs eval):

    bun apps/cli/src/index.js eval .smithers/workflows/<wf>.tsx \
      --cases .smithers/evals/<suite>.jsonl --suite <suite> \
      --concurrency 4 --force
    
  5. Read the report at .smithers/evals/<suite>.json; fix the workflow, not the test — unless the case encodes the criterion wrong, or the failure is environmental (see the classify rule below).

  6. For graded, non-binary quality, attach scorers (faithfulness, relevancy, schemaAdherence, llmJudge) to the load-bearing <Task> and read them with smithers scores <run-id>. Assertions are the hard gate; scorers are the trend.

  7. Add the suite to the sweep script / CI so it runs on every relevant change.

Discipline rules

  • Classify red before acting. Every failed case is one of three things: a product bug (fix the workflow), a wrong case (fix the case, with spec- change scrutiny), or a harness/environment fault (fix the harness). The report marks known environment faults INCONCLUSIVE and smithers eval exits 5 (not 1) when they are the only reds: on that signal repair the harness and never touch the workflow. A red that could not have observed the workflow (connection refused, TLS failure, network denied, missing binary, OOM, rate limit) is not evidence against it.
  • Green ratchet. If a case that passed in round N fails in round N+1 and the only intervening change was to the harness or environment, that is a harness regression: revert the harness change. Never widen a suite's acceptance criteria while it is red; get back to the last green slice first.
  • Circuit breaker. After 3 consecutive rounds with zero net new green cases, stop iterating. Change strategy (gather evidence, widen scope) or escalate to a human via smithers ask-human with what you know. More rounds against the same failure signature is an incident, not progress.
  • Never tune against holdout. No prompt edits, model picks, or optimization acceptance based on holdout results. Green holdout = ship; red holdout = your dev split stopped representing the criteria — fix the split, not just the workflow.
  • Failed case → first-class bug. Reproduce with the run ID in the report (smithers inspect <run-id>, smithers why <run-id>), fix, re-run.
  • Suite changes are spec changes. Editing a case to make a run pass rewrites the acceptance criteria; treat it with the same scrutiny as the workflow change itself.
  • Keep suites cheap to run. Use --max-cases for smoke checks, --concurrency for throughput, and --root to sandbox tool execution.

Pointers

  • Authoring mechanics (scorer wiring, eval-author bootstrap): skills/eval-writer/SKILL.md
  • CLI surface: smithers-eval and smithers-optimize skills; full option list in docs/guides/evals-quickstart.mdx
  • Scorer API reference: docs/reference/scorers (source: packages/scorers)
  • Eval engine: packages/engine + apps/cli eval/optimize commands

Keep looking

Skills are one crate of 613,373. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.