Workflow skill evals

Generate an evolve evaluation suite for a skill

Turn a skill's SKILL.md into the two graded artifacts evolve runs against it:

• triggers.json (Tier 1) — does the skill activate on the prompts that should reach for it, and stay quiet on the ones that shouldn't?
• evals.json (Tier 2) — when it does fire, does it do the job? Real tasks in throwaway workspaces, graded by deterministic checks and, where only prose can judge, an LLM judge.

Both live beside the skill, never inside it: evals/<skill>/, a sibling of skills/<skill>/.

0. Read the upstream guide first — this skill is meant to evolve

evolve gains assertion types, eval fields, and CLI features over time. The authoritative, current source is upstream, not this file. Before authoring, fetch and skim:

• Guide: https://oss.bitwisemedia.uk/evolve/evaluations/ (sub-pages: /triggers/, /evals/, /assertions/, /execution/, /results/).
• Triggers schema: https://raw.githubusercontent.com/bitwise-media-group/evolve/main/schemas/triggers.schema.json
• Evals schema: https://raw.githubusercontent.com/bitwise-media-group/evolve/main/schemas/evals.schema.json

If the guide or a schema lists a field, assertion type, or tier this file doesn't mention, the upstream source wins — use it. To discover new CLI surface in a repo that vendors evolve, run evolve --help, evolve run --help, and evolve docs (here: go tool evolve …). See reference.md for the current assertion table, the staging rule, and worked examples, but treat upstream as the tiebreaker.

1. Locate the skill and its eval directory

Read the target SKILL.md end to end — the frontmatter name and description drive the triggers; the body drives the behavioral evals. Then create evals/<name>/ where <name> equals the skill's name (and its directory). In this marketplace evals are plugins/<plugin>/evals/<skill>/; elsewhere mirror whatever layout the repo's evolve config (.evolve.json) declares.

2. Author triggers.json (Tier 1) — start here, it's the cheapest signal

Envelope: a triggers array of { "query", "should_trigger" }; skill_name echoes the directory. Write 10–20 entries, roughly balanced:

• Positives — the real phrasings a user types when they want this skill. Vary the shape: an imperative, a question, a review-style ask. Each should name the task and domain concretely.
• Negatives are where the signal lives. A positives-only suite scores 100% and tells you nothing. Weight negatives toward near-misses: (a) sibling skills' headline positives — the adjacent skill in the same plugin is where false activations actually happen; (b) same task, wrong domain — a real positive with the language or framework swapped, which catches a skill keying off the verb instead of the context.

{
  "$schema": "https://raw.githubusercontent.com/bitwise-media-group/evolve/main/schemas/triggers.schema.json",
  "skill_name": "go-style",
  "triggers": [
    { "query": "Refactor this Go code to wrap errors properly", "should_trigger": true },
    { "query": "Convert these log.Printf calls to slog", "should_trigger": true },
    { "query": "Write table-driven tests for this Go function", "should_trigger": false },
    { "query": "Refactor this Rust code to use idiomatic error handling", "should_trigger": false }
  ]
}

Start from templates/triggers.json. If you changed the skill's description, the trigger surface moved — re-run Tier 1.

3. Author evals.json (Tier 2) — prove it does the job

Envelope: an evals array. Each case needs an id (lowercase-kebab; it is the results key), a prompt (the real task), and at least one assertions entry or one expectations entry. Write 2–5 cases covering the skill's headline behaviors and its important refusals/guards.

Prefer deterministic assertions over the LLM judge — they are cheap, fast, and reproducible. Reach for the judge only for holistic claims a rule can't express. The current types:

• file_exists / file_absent — the agent created (or correctly did not create) path.
• regex / not_regex — pattern (Go RE2, multiline) matches a workspace file (path) or, with no path, the agent's final response. A missing path file fails either check.
• command — run run via /bin/sh -c; passes when the exit code equals expect_exit (default 0). Set requires to a binary so the check skips (not fails) where it's absent; cwd runs in a subdir. This is the strongest check — it runs the real toolchain over the output.
• tool_call — the agent actually invoked a tool matching tool (regex), optionally with args matching pattern. Inspects behavior, not just artifacts.
• llm (or a bare string in assertions) — a pinned judge verifies text. expectations is a top-level array of such statements, graded first.

Scope allowed_tools per case (e.g. Read Write Edit Glob Grep Skill Bash(go *)) so a pass reflects the skill, not the model improvising. Stage inputs with files (see §4). Start from templates/evals.json; the full table and examples are in reference.md.

{
  "id": "project-scaffold",
  "prompt": "Scaffold a new Go service called orderd with our canonical cmd / tools-module / Makefile layout.",
  "allowed_tools": "Read Write Edit Glob Grep Skill Bash(go *) Bash(gofmt *) Bash(mkdir *)",
  "assertions": [
    { "type": "file_exists", "path": "cmd/orderd/main.go" },
    { "type": "regex", "path": "Makefile", "pattern": "^pr:" },
    { "type": "command", "run": "go vet ./...", "requires": "go" }
  ]
}

4. Stage fixtures with files

files lists input paths relative to the eval directory, staged into the workspace before the run. One rule decides where each lands:

• A path under files/ stages at its path relative to files/, preserving the tree (files/internal/cli/root.go → internal/cli/root.go). Use this whenever the path matters — subdirectories, multi-file layouts, or basenames that would otherwise collide (two go.mods).
• Any other path stages by basename at the workspace root (fixtures/clidemo/go.mod → go.mod). Use fixtures/<name>/ for a shared scaffold many cases reference. A fixture dir may hold more files than a case names; only listed paths are staged.

Author fixtures as the smallest compiling/valid context the assertions need — a real go.mod that requires the right deps lets a command assertion lean on the toolchain.

5. Validate and run

Add the $schema key to both files (above) for editor validation. Then:

evolve run triggers --runs 5        # Tier 1; odd runs avoid 50/50 ties
evolve run evals --jobs 4           # Tier 2; baselines run automatically

In this repo those are make triggers and make evals (make all for both tiers plus reports), and make fmt && make lint gates the eval JSON, schema, and markdown before committing. The committed results.json is written by the sweeps — never hand-edit it. A brand-new suite simply has none until its first run.