Recipe eval skill
Creates or updates Claude Code skills through interactive dialog, then evaluates effectiveness with sequential paired comparisons. Use when creating new skills, updating existing skills, or evaluating skill quality.From its SKILL.md
npx -y skills add shinpr/rashomon --skill recipe-eval-skillAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 18 stars18 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
5.1 KB, as published. Nobody here has run it
Context: Skill authoring (Phase A) followed by blind A/B evaluation (Phase B)
Mode: $ARGUMENTS
Orchestrator Definition
Core Identity: "I am not a worker. I am an orchestrator."
Execution Method:
- Skill generation/modification → performed by rashomon:skill-creator
- Skill quality grading → performed by rashomon:skill-reviewer
- Test task execution → performed by eval-executor.py script (via
claude -p) - Blind result comparison → performed by rashomon:skill-eval-reporter
Orchestrator invokes sub-agents via Agent tool and scripts via Bash, passes structured data between them.
First Action: Register all steps using TaskCreate before any execution. Phase A steps are defined in the mode-specific reference (create.md or update.md). Phase B steps are defined in eval.md. Update status using TaskUpdate upon each step completion.
Mode Detection
Determine mode from $ARGUMENTS:
| Mode | Criteria |
|---|---|
| Creation | "create", new skill request, no existing skill referenced |
| Update | "improve", "update", existing skill name or path mentioned |
| Unspecified | $ARGUMENTS is empty or ambiguous |
Scope Boundaries
Phase A (Skill Authoring): Create or modify skill content through dialog. Ends with user-approved skill file. Phase B (Evaluation): Measure skill effectiveness through blind execution comparison. Phase B is read-only for the source skill. A finding that requires authoring changes transitions back to Phase A for user review and approval before evaluation restarts.
Responsibility Boundary: This skill completes with the combined evaluation report and ship/revise/reject recommendation.
Workflow
Phase A: Skill Authoring
Read the mode-specific reference and execute:
- Creation mode: Read references/create.md and follow its steps
- Update mode: Read references/update.md and follow its steps
Phase A ends with: user-approved skill content (new or modified).
Phase A → Phase B Handoff
Before starting Phase B, confirm these data are available in context. Phase B cannot proceed without them:
| Data | Source | Required |
|---|---|---|
| Skill name | Phase A dialog | Always |
| Source skill directory | Phase A file write | Always |
| Held-out test requests | Phase A Round 3 (create) / Round 2 (update) | Always |
| Trigger scenarios | Phase A Round 3 (create) / Round 1-2 (update) | Always |
| Old skill directory snapshot and fingerprint | Phase A Step 6 (update mode only) | Update mode |
| Approved source directory fingerprint | Phase A final write | Always |
If fewer than two held-out requests are available, ask before proceeding: "What complete requests does your team actually send for work that requires this skill's rules? Please provide at least two verbatim."
Phase B: Evaluation
Read references/eval.md and execute the evaluation protocol. Pass the handoff data above as context.
Phase B consists of:
- Trigger check: Does the skill fire for its intended use case? (Step 1)
- Trigger fail handling: Diagnose and request an authoring revision when needed (Step 2, conditional)
- Execution effectiveness: Blind A/B comparison of output quality (Steps 3-7)
Final Output
Present combined results to user:
- Phase A result: Skill quality grade (A/B/C from rashomon:skill-reviewer)
- Phase B trigger: Discovered (yes/no), Used (yes/no), usage evidence
- Phase B execution: Blind comparison result (from rashomon:skill-eval-reporter)
- Recommendation: ship / revise / reject
Error Handling
| Scenario | Behavior |
|---|---|
| User cancels during Phase A | Stop. No eval needed. |
| Grade C after 2 repair/re-review iterations | Present content with remaining findings. User decides: accept/revise/abort. |
| One executor fails in Phase B | Preserve diagnostics, mark comparison inconclusive, and make no winner or effectiveness recommendation. |
| Both executors fail in Phase B | Report failure. Phase A result still valid. |
| Worktree creation fails | Report git error. Phase A result still valid. |
Prerequisites
- Git repository with
git worktree locksupport claudeCLI available in PATH- Sufficient disk space for worktree copies
Completion Criteria
Phase A
- Skill knowledge collected through dialog
- rashomon:skill-creator returned valid output
- rashomon:skill-reviewer returned grade A or B
- User approved final content
- File written to target location
Phase B
- Trigger check executed and result presented
- Sequential paired execution completed in worktrees
- Blind comparison completed by rashomon:skill-eval-reporter
- Worktrees cleaned up
- Combined report presented with recommendation
What ships with it: 4 files
44.9 KB alongside SKILL.md, 1 of them executable
scripts/
- eval-executor.pyruns20.2 KB
Gives 0 of the 12 instructions most evals benchmarks skills give
Counted across 499 of the 513 authors here whose files we hold, read 2026-09-06
- Spawn with-skill and baseline runs in the same turnin 31 of 499, across 24 files
- Keep SKILL.md under 500 linesin 31 of 499, across 24 files
- Draft assertions while test runs are in progressin 31 of 499, across 24 files
- Compare against the baseline after changesin 31 of 499, across 13 files
- Define evals before codingin 26 of 499, across 17 files
- Run evals frequently during developmentin 25 of 499, across 16 files
- Keep evals fastin 24 of 499, across 15 files
- Version evals with codein 24 of 499, across 15 files
- Generate the eval viewer before evaluating outputs yourselfin 24 of 499, across 17 files
- Generate an eval report after runsin 24 of 499, across 15 files
- Track pass@k metrics over timein 22 of 499, across 14 files
- Save a baseline before making changesin 21 of 499, across 9 files
Said here and by no other author read
- Register all steps via TaskCreate before starting
- Update step status via TaskUpdate on completion
- Clean up worktrees after collecting both results
- Ask the user when mode is unspecified
- Read the mode-specific reference before authoring
- Confirm handoff data before starting evaluation
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once.