agentsclimarketplace
Sign in

Eval harness

Skill a5c-ai/babysitter/library/methodologies/everything-claude-code/skills/eval-harness

Evaluation harness for testing agent and skill quality through structured benchmarks, regression tests, and quality scoring.From its SKILL.md

Install
npx -y skills add a5c-ai/babysitter --skill eval-harness

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

2.1 KB, 358 tokens by cl100k_base, as published. Nobody here has run it

  • Define test cases with known-correct outputs
  • Run agent against each test case
  • Score: accuracy, completeness, relevance
  • Compare against baseline performance
  • Track performance over time

2. Skill Quality Testing

  • Verify skill instructions produce expected outcomes
  • Test edge cases and boundary conditions
  • Measure consistency across multiple runs
  • Check for harmful or incorrect outputs
  • Validate against ground truth

3. Regression Suite

  • Collection of previously-passing test cases
  • Run after any agent/skill modification
  • Flag regressions with before/after comparison
  • Maintain pass rate threshold (>= 95%)

4. Process Verification

  • End-to-end process execution with known inputs
  • Verify each phase produces expected outputs
  • Check task ordering and dependency satisfaction
  • Measure total execution time

Quality Scoring

Accuracy Score (0-100)

  • Correctness of output vs expected
  • Partial credit for partially correct outputs
  • Penalty for hallucinated or fabricated content

Completeness Score (0-100)

  • Coverage of required output elements
  • Missing sections flagged and scored
  • Bonus for useful additional context

Consistency Score (0-100)

  • Run same input 3 times
  • Compare outputs for semantic similarity
  • Flag inconsistencies

Composite Score

  • (accuracy * 0.4 + completeness * 0.3 + consistency * 0.3)
  • Threshold: 80 to pass

When to Use

  • After creating new agents or skills
  • After modifying existing agents or skills
  • Periodic quality audits
  • Before promoting skills to production

Agents Used

  • Used by process-level evaluation orchestrators
  • No specific agent dependency (evaluates other agents)

What ships with it: 1 file

230 B alongside SKILL.md

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.