Skillhone evaluation

SkillHone Evaluation

Run evaluations and diagnose results to inform improvement decisions.

Evaluator Harness Model

The evaluator is not just a scorer. It is a harness that runs the skill against private tasks and leaves a structured evidence trail:

• Input: a skill checkout plus an eval repo containing datasets, verifier logic, task contract, and optional compiler/audit helpers.
• Execution: each eval item runs in an isolated solver workdir. The solver may create files, call scripts, and write the required artifact.
• Verification: the eval repo verifier scores the produced answer and may call task-local validators, compilers, parsers, renderers, or audit helpers.
• Output JSON: aggregate score plus redacted per-item trace summaries.
• Workdir evidence: trajectory.jsonl, produced artifacts, and stderr-like signals that explain failures the score cannot explain.

This separation matters. A low score may come from weak skill instructions, but it may also come from missing files, tool crashes, invalid compiled artifacts, over-strict verifier rules, or infrastructure errors. Evaluation work is about mapping the failure to the correct harness layer.

On Forgejo-backed repos, status.py is a read-only context check before PR validation or merge decisions:

python3 ~/.skillhone/skills/skillhone/scripts/status.py

Core capability: eval.py

# Probe — fast iteration signal
python3 ~/.skillhone/skills/skillhone/scripts/eval.py \
  --skill-dir /path/to/skill --eval-dir /path/to/eval-repo \
  --split probe --output _data/probe_result.json

# Test — final benchmark (NEVER during iteration)
python3 ~/.skillhone/skills/skillhone/scripts/eval.py \
  --skill-dir /path/to/skill --eval-dir /path/to/eval-repo \
  --split test --output test_result.json

The output JSON includes a "workdir" field pointing to solver working directories (e.g. /data/tmp/eval_agent_xyz/) containing trajectory.jsonl files and produced artifacts for deeper diagnosis.

Subagents

Subagent	What it does
trajectory-analyzer	Reads workdir/work_<uid>/trajectory.jsonl files to diagnose tool errors (rate limits, wrong tool calls, script crashes). Outputs redacted _data/trajectory_diagnosis.json safe to share with improver.
pr-quality-reviewer	Merge gate for skill PRs — runs static check + rubric scoring, posts PR comment, returns APPROVE/REQUEST_CHANGES.

Trajectory Diagnosis

probe_result.json captures scores and some error categories, but it is only the top of the evidence trail. It cannot fully explain runtime behavior such as:

• Wikipedia API rate limiting (HTTP 429/403)
• Agent calling web_search directly instead of Bash("python3 scripts/web_search.py ...")
• Script crashes (exit code 1 with traceback)
• The solver never writing the required artifact
• The solver writing a file in the wrong location

The trajectory-analyzer subagent fills this gap by reading raw solver logs. Its output distinguishes infrastructure failures (fix scripts/config) from skill failures (fix SKILL.md). This distinction is critical for avoiding wasted iterations.

Compiler Feedback Diagnosis

Some artifact tasks are compiler-like: Mermaid, LaTeX, TypeScript, Python tests, SQL parsers, JSON/YAML schema validators, browser renderers, and similar tools produce actionable stderr or diagnostics. Do not reduce these failures to wrong_answer.

When a failed trace produced an artifact, inspect the solver workdir from the workdir field and run the task-local compiler, validator, renderer, or audit helper on that artifact. Prefer commands and helpers shipped by the eval repo or described in its README/contract; if none exist, use the standard local compiler for that artifact type. Capture only concise, non-gold diagnostic summaries:

• compiler/parser command used
• first error line and location, if available
• failed rule name or failed score key, if available
• artifact-level pattern, e.g. "invalid Mermaid subgraph syntax" or "missing answer file"

Write this as _data/compiler_diagnosis.json or include it in the existing diagnosis file. It is safe to share with the improver when it contains only error messages, failed score names, and artifact snippets needed to identify the syntax class; do not include gold answers or full eval questions.

If the task-local verifier already exposes detailed failed score keys, preserve those names. They are usually better improvement signals than a rewritten natural-language summary.

Interpreting scores

Field	Meaning
score	pass rate (0.0–1.0)
avg_duration_s	efficiency; rising duration with flat score = looping/waste
traces[].error	"hard timeout", "agent_process_error", or empty
workdir	path to solver trajectories for deeper analysis

Decision thresholds:

• ≥ +0.02 → real improvement
• ±0.02 → noise, don't claim improvement
• ≤ −0.04 → regression, revert

Score Provenance

Always label which harness run produced a score. In a full SkillHone run there may be several valid scores: baseline probe, iteration probe, PR validation, and a final driver re-score after the master agent exits. These can differ because they may use different skill checkouts, regenerated eval data, or output paths.

When writing issues, PR comments, wiki observations, or user summaries, cite the score source in words: split, output JSON path if available, workdir if useful, and whether it is an internal iteration score or final harness score. Do not collapse multiple scores into one number without naming the source.

Splits and data visibility

Split	Purpose	Who sees
probe	Iteration signal	orchestrator (redacted traces)
pr_val	PR merge gate	orchestrator (aggregate only)
test	Final benchmark	orchestrator only, NEVER during iteration

Constraints

• Never run test during iteration — it contaminates the final benchmark.
• Never forward gold answers or full questions to the improver.
• Never pass the eval repo path to the iterating agent.
• trajectory_diagnosis.json is fully redacted (uid + counts only) — safe to share.

Reference

• references/scoring.md — error category catalog, match rules, close-but-not-credited cases