Eval regression
Skill reidemeister94/development-skills/skills/eval-regression
Use when the user asks to check a plugin or skill with behavioral evals, compare it with HEAD, or investigate a behavioral regression.From its SKILL.md
npx -y skills add reidemeister94/development-skills --skill eval-regressionAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 12 stars12 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
2.4 KB, as published. Nobody here has run it
Eval regression
Use deterministic repository tests first. Stop when the target has no behavioral diff.
Resolve the plugin from the argument or cwd. Its catalog is evals/evals.json; the shared runner is ../ai-agent-bench/scripts/run_evals.py. Require the user to choose agent, model, and effort. Never select a costly model or high effort silently.
Normal check
- Select cases by changed paths with
--changed-from <base>, or name them with--case. Do not select the full catalog implicitly. - Run the command without
--run. The runner prints the cases, modes, repeat count, session count, per-session timeout, and maximum duration without starting an agent. - Present that plan and stop for explicit cost approval.
- After approval, repeat the same command with
--run. The default is candidate-only, one run per case, 180 seconds per session, and at most four sessions. - Report every failed assertion, timeout, non-zero exit, duration, and token count. Missing evidence is inconclusive.
Escalation
- Compare base and candidate only when the user asks, or when a failed candidate check needs to distinguish a regression from an existing failure. Extract the base with
git archive; run the same selected cases, agent, model, effort, repeat, and timeout on both; compare reports with--compare. - Repeat only a failed or observably unstable case. Three repeats are a stability benchmark, not a default.
--all,--repeat > 1, a larger--max-sessions, or a longer timeout needs a new run plan and explicit approval.- Routing cases stop at the first Skill selection. A tool assertion is appropriate there because routing is the contract; they must not execute the selected skill.
Remove temporary base copies after the comparison. Do not edit or commit the target.
Deterministic assertions are tool, tool_not, clean_worktree, changed_files_exact, file_contains, file_not_contains, transcript_contains, and tool_sequence. Use a semantic judge only when no filesystem, command, tool, ordering, or assistant-output observation can express the contract.