Three layer eval

three-layer-eval

Three-layer evaluation framework for skill quality assessment.

Layer 1: Static Analysis (<2s)

Deterministic checks — fast, no LLM required.

Checks

1. Frontmatter completeness

   • name present and matches directory name
   • description present, >20 chars, <200 chars
   • triggers array present, at least 1 trigger, each trigger >3 chars

2. Trigger-phrase presence

   • At least one trigger appears in the skill body (natural usage)
   • Triggers are not generic words ("help", "run", "do")

3. Body size

   • SKILL.md body <8KB (excluding frontmatter)
   • SKILL.md <150 lines
   • Total directory <50KB

4. Reference link validity

   • All relative links resolve to existing files
   • No broken anchors (#section links)

5. Structural patterns

   • Has clear "When to use" or trigger section
   • Has output/return format documented
   • No TODO or placeholder content

Output

Layer 1: STATIC — PASS/FAIL
  frontmatter: OK
  triggers: OK (3 triggers, all appear in body)
  size: OK (4.2KB body, 87 lines)
  links: FAIL (1 broken: references/old-api.md)
  structure: OK

Layer 2: LLM Judge (Haiku + Sonnet)

Semantic evaluation across 4 dimensions. Uses Haiku for fast scoring, Sonnet for nuanced cases.

Dimensions

1. Completeness (0-10)

   • Does the skill cover its stated use cases?
   • Are edge cases documented?
   • Is the output format clear?

2. Trigger-accuracy (0-10)

   • Do triggers accurately predict when to invoke?
   • Are there false positives (triggers that match unrelated work)?
   • Are there false negatives (missing triggers for valid use)?

3. Safety (0-10)

   • Does the skill avoid destructive operations by default?
   • Are there guardrails for irreversible actions?
   • Does it respect the harness boundary (no project-specific imports)?

4. Portability (0-10)

   • Can the skill run in any repo?
   • Does it depend on specific tools or environments?
   • Are assumptions documented?

Execution

Haiku pass: score each dimension 0-10 with brief justification
Sonnet pass: review Haiku scores, adjust if needed, provide detailed feedback

Output

Layer 2: LLM JUDGE — avg 7.8/10
  completeness: 8/10 — covers main use cases, missing edge case X
  trigger-accuracy: 9/10 — triggers are precise, no false positives
  safety: 7/10 — has guardrails but missing checkpoint for operation Y
  portability: 6/10 — assumes tool Z is installed, should document
  
  Sonnet notes: The skill is solid but could benefit from explicit
  failure modes section. Trigger "fix bug" is too generic — consider
  narrowing to "fix test failure" or "fix build error".

Layer 3: Monte Carlo Simulation (50-100 runs)

Statistical reliability testing via simulated invocations.

Method

1. Generate 50-100 synthetic prompts that should/shouldn't trigger the skill
2. Run the skill's trigger-matching logic against each prompt
3. Measure:
   • Precision: % of triggered runs that were correct invocations
   • Recall: % of valid invocations that triggered the skill
   • False positive rate: % of triggered runs that were incorrect
   • False negative rate: % of valid invocations that didn't trigger

Execution

# Generate test prompts
./skills/three-layer-eval/generate-prompts.sh <skill-name>

# Run simulation
./skills/three-layer-eval/simulate.sh <skill-name> --runs 100

# Analyze results
./skills/three-layer-eval/analyze.sh <skill-name>

Output

Layer 3: MONTE CARLO — 100 runs
  precision: 0.92 (92/100 triggered runs were correct)
  recall: 0.87 (87/100 valid invocations triggered)
  false positive: 0.08 (8/100 triggered incorrectly)
  false negative: 0.13 (13/100 valid invocations missed)
  
  Confidence interval (95%): ±0.06
  
  Failure modes:
    - Prompt "debug this" triggers but is too generic (3 cases)
    - Prompt "fix test" doesn't trigger but should (7 cases)

Combined report

SKILL EVALUATION: <skill-name>
================================

Layer 1: STATIC — PASS (1 issue)
Layer 2: LLM JUDGE — 7.8/10 avg
Layer 3: MONTE CARLO — precision 0.92, recall 0.87

OVERALL: GOOD (minor improvements needed)

Top 3 improvements:
1. Fix broken link: references/old-api.md
2. Narrow trigger "fix bug" → "fix test failure"
3. Add checkpoint for operation Y

Detailed findings: <attached>

When to use

• Before merging a new skill
• Quarterly skill quality audit
• When a skill feels unreliable (false positives/negatives)
• After major skill refactors

References

• Inspired by lm-eval-harness EvalResults schema
• Inspired by CodeWhale typed receipts
• See standards/typed-result-schema.md for output contract