Skill eval

Skill Eval

Benchmark a skill's compliance with its own instructions by extracting verifiable requirements from its SKILL.md and scoring the skill's output from the current session.

Important

• This is a measurement tool, not a feelings tool. Report what happened, not what you wish happened.
• Every requirement gets a verdict. No skipping, no "N/A unless relevant." If the requirement existed, it was either met or it wasn't.
• Deterministic checks are authoritative. If a deterministic check says FAIL, it's FAIL — no judgment override.
• Judgment checks must cite specific evidence from the output. "Seemed fine" is not a verdict.
• If a skill was not used in this session, say so and stop. Don't fabricate an evaluation.
• User-invalidation override. If the user explicitly stated the session output was bad, generic, off-topic, wrong, or otherwise failed to deliver value (e.g., "this is meh", "you over-framed it", "the recs are getting worse", "this is generic", "particularly bad", or any equivalent rejection), the eval CANNOT return a compliance score above 50%, regardless of structural pass count. Structural compliance measures whether headings exist; user invalidation is direct evidence that what those headings contained did not work. The user's verbatim words are the primary finding and must appear at the top of the report. A high structural PASS on a session the user called "meh and generic" is the failure mode this rule prevents.

Instructions

Step 1: Identify Target Skill

Read $ARGUMENTS for the skill name.

• If provided: use it directly (e.g., scientific-writing, career, meta-session-audit)
• If empty: scan the current conversation for the most recently invoked skill. If none found, ask the user which skill to evaluate.

Step 2: Load Skill Instructions

Read the skill's SKILL.md:

~/.claude/skills/{skill-name}/SKILL.md

If the file doesn't exist, check:

• ~/.claude/plugins/**/skills/{skill-name}/SKILL.md
• .claude/skills/{skill-name}/SKILL.md (project-level)

If still not found, report the error and stop.

Step 3: Extract Requirements

Parse the SKILL.md and extract every discrete, testable requirement. A requirement is any instruction that can be verified as followed or not followed. Sources:

1. Explicit imperatives: "Always X", "Never Y", "Must Z", "Do not W"
2. Format specifications: output structure, section headings, required fields
3. Process steps: numbered steps that define a workflow
4. Conditional rules: "If X, then Y" — check if condition was met and rule was followed
5. Negative constraints: "Do NOT", "Avoid", "Never" — things that must be absent

For each requirement, record:

• ID: R1, R2, R3...
• Text: The requirement as stated in SKILL.md (quote it)
• Type: deterministic or judgment
• Check method: How to verify it

Classification rules:

• deterministic: Can be verified by searching the output for presence/absence of specific patterns, structures, sections, or formats. Examples: "Never use bullet points" → grep for - or * . "Must include ## Recommendations section" → search for heading.
• judgment: Requires qualitative assessment against a rubric. Examples: "Write in flowing prose" → assess paragraph quality. "Be brutally honest" → assess tone.

Step 4: Locate Skill Output

Find the skill's output in the current conversation context. This is the content produced after the skill was invoked and before the next user message or skill invocation.

If the skill produced files (wrote to disk), read those files too — they're part of the output.

Step 5: Run Deterministic Checks

For each deterministic requirement:

1. Define the check (what to search for, what pattern to match or not match)
2. Execute against the skill output
3. Verdict: PASS or FAIL
4. Evidence: the specific line, section, or pattern that proves the verdict

No partial credit on deterministic checks. It either passes or it doesn't.

Step 6: Run Judgment Checks

For each judgment requirement:

1. State the rubric: what does "meeting this requirement" look like concretely?
2. Cite 1-3 specific passages from the output as evidence
3. Verdict: PASS, PARTIAL, or FAIL
4. Reasoning: 1-2 sentences explaining the verdict with reference to the evidence

Scoring judgment checks conservatively:

• PASS: Clear, unambiguous compliance with cited evidence
• PARTIAL: Partially met — some evidence of compliance, some of non-compliance
• FAIL: No meaningful evidence of compliance, or clear violation

Step 7: Calculate Score

Compliance = (PASS_count + 0.5 * PARTIAL_count) / total_requirements * 100

Round to nearest integer. No decimal points — false precision.

Step 8: Generate Report

# Skill Eval: {skill-name}
**Compliance: {score}%** ({pass_count}/{total} passed, {partial_count} partial, {fail_count} failed)

## Deterministic Checks ({n_det_pass}/{n_det_total})

| ID | Requirement | Verdict | Evidence |
|----|------------|---------|----------|
| R1 | "Never use bullet points in final output" | PASS | No bullet-point lists found in output |
| R2 | "Must include ## Recommendations section" | FAIL | Section absent from output |

## Judgment Checks ({n_jdg_pass}/{n_jdg_total})

| ID | Requirement | Verdict | Evidence |
|----|------------|---------|----------|
| R5 | "Write in flowing prose" | PARTIAL | Introduction flows well, but Discussion §2 is choppy — 4 consecutive short sentences without transitions |
| R6 | "Be brutally honest" | PASS | Report directly stated "session was bad" without hedging (para 3) |

## Missed Requirements — Detail

### R2: "Must include ## Recommendations section"
**Expected**: A dedicated section with actionable recommendations
**Found**: Recommendations scattered in closing paragraph, no dedicated section
**Impact**: Reader cannot quickly find actionable takeaways

## Summary

**Strongest compliance**: [area where most requirements were met]
**Weakest compliance**: [area where most requirements were missed]
**Top fix**: [single highest-impact change that would improve compliance most]

Step 9: Persist (REQUIRED — enforced by Stop hook)

The Stop hook at ~/.claude/skills/skill-eval/hooks/stop-guard.sh blocks turn-end until a yaml file exists at ~/.claude/skill-evals/{skill-name}/YYYY-MM-DD.yaml AND that file contains the required keys. Persistence is not optional. If you skip this step the hook will fire and you will have to come back and write the file anyway.

Required yaml keys (all non-skippable, even when the user invalidated the eval):

skill: skill-name              # REQUIRED
date: YYYY-MM-DD               # REQUIRED
compliance: N                  # REQUIRED — capped at 50 if user_invalidated: true
total_requirements: N          # REQUIRED
passed: N                      # REQUIRED
partial: N                     # REQUIRED
failed: N                      # REQUIRED
user_invalidated: true|false   # REQUIRED if applicable, omit if false
requirements:                  # REQUIRED — never empty, never omitted
  - id: R1
    text: "requirement text"
    type: deterministic|judgment
    verdict: PASS|PARTIAL|FAIL
    evidence: "brief evidence"

The requirements array is the most-commonly-omitted field. The Stop hook will reject any yaml that lacks it. If you ran the eval, you have requirements. Write them. Even if the user invalidated the framing of the eval (e.g., "you scored the wrong thing"), you still have the structural requirements you graded against — write them with their verdicts, then add user_invalidated: true and a user_invalidation_reason field above the requirements block to surface why the score is not the real signal.

Pre-write checklist (run before calling Write):

1. Does the yaml have skill, date, compliance, total_requirements, passed, partial, failed?
2. Does the yaml have a requirements: key with at least one entry?
3. If user invalidated: is compliance ≤ 50 and is user_invalidated: true present?

If any answer is no, fix before writing. Do not rely on the Stop hook to catch you.

If previous evals exist for this skill, note the trend: improving, stable, or regressing — but the trend line is only meaningful when scores are comparable. A 77% structural pass next to a 77% structural pass on a user-invalidated session is meaningless because the second 77% should have been ≤50 per the override.

Error Handling

1. Skill not found: Report error, list available skills from ~/.claude/skills/, stop.
2. Skill not used in session: Report that no output was found for this skill in the current session. Suggest running the skill first, then re-running /skill-eval.
3. Skill output compressed/truncated: Note that only partial output was available. Score what's visible, flag requirements that couldn't be checked as UNKNOWN (don't count toward score).
4. Ambiguous requirement: If a requirement in SKILL.md is too vague to evaluate (e.g., "be helpful"), classify as judgment, note it's underspecified, and grade conservatively.
5. Skill has no testable requirements: Report that the SKILL.md lacks verifiable instructions. Suggest improvements to make it evaluable.

Examples

Example 1: Evaluating scientific-writing after a manuscript

Input: /skill-eval research-search

Output: Reads scientific-writing/SKILL.md, extracts 18 requirements (e.g., "Never bullet points in final manuscript", "IMRAD structure", "Two-stage process", "Graphical abstract required"). Scores the manuscript output. Finds 3 bullet lists in Discussion (R1 FAIL), no graphical abstract generated (R7 FAIL), IMRAD structure present (R3 PASS). Reports 78% compliance.

Example 2: Evaluating meta-session-audit

Input: /skill-eval meta-session-audit

Output: Extracts requirements like "Every error gets a root cause", "Classify session quality", "Save to ~/.claude/session-audits/". Checks audit output — finds 2 errors without root causes (R4 PARTIAL), quality classification present (R6 PASS), file persisted correctly (R8 PASS). Reports 85% compliance.

Example 3: No skill used in session

Input: /skill-eval business-plan

Output: "No output found for the business-plan skill in this session. Run /business-plan first, then /skill-eval business-plan to benchmark it."