Ai eval harness

AI Engineering Operating System

Install

npx -y skills add willianbs/skills --skill ai-eval-harness

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

19 days oldThe repository was created 19 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Designs and runs evaluation harnesses for AI features and for agent skills themselves. Use when shipping LLM features or measuring skill quality. Emits EVAL_PLAN / EVAL_REPORT. Never claims production quality from a handful of vibes-only prompts.

SKILL.md

3.0 KB, as published. Nobody here has run it

Purpose

Define datasets, metrics, graders, and regression gates for AI systems (product features or this skill portfolio).

When to Use / When NOT to Use

Use when: LLM/agent features; prompt/skill changes; “are we better?”; offline evals; skill portfolio regression.

Do not use when: non-AI CRUD with no model calls; pure infra without generative behavior.

Preconditions

Clear target behavior and some way to run the system/skill under test (or design-only mode if runtime unavailable).

Inputs / Outputs

Inputs: feature/skill under test, CONTEXT_PACK, success criteria, existing evals if any.

Outputs: EVAL_PLAN then EVAL_REPORT when executed.

Upstream / Downstream

Upstream: spec-validator, test-strategy-designer, engineering-os, skill authors.

Downstream: feature-implementer, quality-gate, pr-generator, delivery-planner.

Core Principles

Evals need fixed cases + graded outcomes — not vibes alone.
Split: unit-ish (deterministic checks), scenario, adversarial/safety, human review sample.
Measure regressions; version datasets.
Cost/latency are metrics too.
For skills: trigger accuracy, artifact validity, stop-condition obedience.
Report confidence intervals qualitatively when N is small.
Never fabricate scores.

Process

Modes

product — AI feature in the app
skill — evaluate an Agent Skill / portfolio path

Define behaviors and failure modes to catch.
Build/extend dataset cases (inputs, expected traits, forbidden traits).
Choose graders: exact, rubric LLM-as-judge (with caution), programmatic assertions.
Set pass thresholds and CI gating vs offline-only.
Run if possible → EVAL_REPORT with per-case results.
Recommend next dataset expansions.

Evidence Requirements

Case IDs, commands, raw graded outputs summarized (redact PII). N stated.

Stop Conditions / Failure Modes

Condition	Action
No cases defined	Block “quality” claims
Runtime unavailable	Ship EVAL_PLAN only; Decision ProceedWithConditions
Safety cases failing	Block ship for product mode

Severity + Confidence

Safety/eval failures on HighRisk AI → Critical/High for release.

Output Contract

## EVAL_PLAN
Mode: product | skill
Behaviors: ...
Dataset: ...
Graders: ...
Thresholds: ...
## EVAL_REPORT (if run)
N: ...
Results summary: ...
Failures: ...
Decision: Proceed | ProceedWithConditions | Revise | Block

Handoffs

quality-gate, feature-implementer, test-strategy-designer, security-auditor (prompt injection / data leak).

Never

Never invent pass rates.
Never rely solely on a single cherry-picked demo prompt.
Never log secrets from model outputs.