Regen eval baseline

Skill sumithr/sumo-qa/.claude/skills/regen-eval-baseline

Captures a promptfoo skill-eval baseline JSON for one sumo-qa skill and snapshots it to docs/qa/runs/eval-baselines/, with an automatic delta against the prior snapshot. Use this whenever the user mentions baselining a skill, capturing a before/after eval, running a single-skill eval, or measuring the effect of a SKILL.md edit — common during token-optimisation rounds. The actual work runs through a bundled script that handles path conventions, API-key checks, and diffing in one go.From its SKILL.md

Install

npx -y skills add sumithr/sumo-qa --skill regen-eval-baseline

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

6 stars6 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

7.4 KB, as published. Nobody here has run it

regen-eval-baseline

Captures a promptfoo run for one sumo-qa skill and stores its JSON output in docs/qa/runs/eval-baselines/ (gitignored). The deterministic work lives in scripts/run_baseline.py; this document is the guide for picking inputs and reading the output.

When to use

Trigger this skill when the user wants a per-skill eval snapshot. Common phrasings: "baseline this skill", "snapshot the eval", "run the eval for skill X", "capture before/after for the rewrite I just made". The user invokes it explicitly with /regen-eval-baseline; it doesn't auto-trigger.

This is single-skill on purpose. Full-sweep regeneration belongs on npm run eval:all, which also reads the same tests/evals/promptfoo/skill-*.yaml files but runs them sequentially without snapshotting.

Inputs

Pass exactly one config selector — either --skill (the base config) or --config (a suffixed scenario / .ab.yaml control / explicit path) — plus an optional --label.

--skill <name> — the base config tests/evals/promptfoo/skill-<name>.yaml. Resolves that file exactly; it never cross-matches a longer suffixed sibling (e.g. --skill reviewing-before-merge drives skill-reviewing-before-merge.yaml, NOT skill-reviewing-before-merge-adversarial.yaml). If the named config is absent the script lists the available configs.
--config <selector> — for a suffixed scenario config or an .ab.yaml A/B control. Accepts a bare stem (skill-reviewing-before-merge-adversarial), a filename including the double suffix (skill-reviewing-before-merge-adversarial.ab.yaml), or a full/relative path. Resolved exactly — a stem composes one filename and that file must exist; the resolver never falls back to a near-named neighbour, so a base name can't accidentally snapshot a suffixed sibling and vice-versa.
--label — short kebab-case tag for the snapshot, defaults to baseline. Past conventions in the dir: baseline, postcut, greenfix, or describe-the-change like removability-gate. The label vocabulary is open-ended and may itself be multi-hyphen, so the snapshot filename separates the slug from the label with a literal __ (double underscore) — <date>-skill-<slug>__<label>.json. Because a validated kebab slug/label can never contain __, the slug/label boundary stays unambiguous even for a multi-hyphen label. The label is what makes two snapshots taken on the same day distinguishable.

The snapshot filename derives from the resolved config, not just --skill: a .ab infix becomes a hyphen in the slug (skill-x-adversarial.ab.yaml → …-skill-x-adversarial-ab__<label>.json), so the base config, its suffixed scenario, and its .ab control each snapshot to a distinct, non-colliding path and diff only against their own prior snapshots.

Wrapper vs raw `promptfoo eval -c`

Use this wrapper (--skill / --config) whenever you want the repeatable before/after snapshot it exists to provide: it writes the dated JSON to docs/qa/runs/eval-baselines/, prints pass/fail, and diffs against the prior snapshot for the same config. That covers base configs, suffixed scenario configs, and .ab.yaml controls — drive all three through the wrapper so the baseline/postcut delta is preserved (don't hand-roll npx promptfoo eval -c … and lose the snapshot). Reach for raw promptfoo eval -c <path> only for flags the wrapper does not expose (e.g. --repeat N variance runs, -j 1 legible logs, generate dataset) — see tests/evals/promptfoo/README.md. For a normal baseline/postcut capture, the wrapper is the supported path.

Prerequisites the script will check

OPENAI_API_KEY must be set in the environment. The harness reads it from ~/.config/promptfoo-keys.env per tests/evals/promptfoo/README.md; tell the user to source that file if the script reports it missing. Never accept the key pasted in chat — both repo policy and the feedback_never_handle_pasted_secrets memory rule say so.
The selected config must exist: tests/evals/promptfoo/skill-<name>.yaml for --skill, or the exact suffixed / .ab.yaml config for --config. If it doesn't, the script lists every available config so the user can pick again — it does not guess a near-named sibling.
A snapshot at the target path already existing will block the run unless --force is passed. Don't pass --force reflexively — snapshots are evidence of past runs and silently clobbering them loses history.

Run the script

# Base config:
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
  --skill <skill-name> \
  --label <label>

# Suffixed scenario config (no rename to the base skill needed):
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
  --config skill-reviewing-before-merge-adversarial \
  --label baseline

# .ab.yaml A/B control:
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
  --config skill-reviewing-before-merge-adversarial.ab.yaml \
  --label postcut

The script:

Computes the snapshot path: docs/qa/runs/eval-baselines/<today>-skill-<slug>__<label>.json (the slug comes from the resolved config — see Inputs; __ separates slug from label so a multi-hyphen label stays unambiguous).
Runs npx promptfoo eval with --no-cache (so the snapshot reflects fresh judge calls, not stale cache hits) and writes the JSON output to that path.
Prints pass/fail counts.
If a prior snapshot for the same config exists, prints a delta — passed and failed counts vs the previous run.

Reading the output

The pass/fail summary tells you the state of the snapshot. The delta tells you whether the most recent SKILL.md edit moved the needle. Three patterns to watch for:

Passes increased, failures decreased — the edit helped. Keep it.
Passes decreased, failures increased — the edit hurt. Investigate before reverting; the failure may be informative.
No change — either the edit was outside the assertions' coverage, or the judge gave the same verdict on different reasoning. Read the JSON's per-test reasons before drawing conclusions.

When FAILs appear

Don't propose loosening the rubric. The standing repo policy (feedback_eval_fixes_target_skill_not_rubric) is to strengthen the SKILL.md so the candidate naturally satisfies the existing rubric. Hand the failing snapshot to the eval-failure-diagnoser subagent — it reads the JSON, locates the failing assertion, and recommends a concrete SKILL.md edit. Re-run this skill after the edit to confirm the failure flipped.

Why the snapshots are gitignored

docs/qa/ is excluded from the repo (.gitignore line 48). Eval snapshots are local evidence — useful for the contributor doing the optimisation work, but not artefact that ships with the package. Per feedback_no_process_artifacts_in_public_repo, results live alongside the QA work, not in the public source tree. Don't propose committing snapshots or removing the gitignore entry.