Tool eval

Tool Eval

A tool is the contract between a deterministic system and a non-deterministic agent. When you score that contract with exact string match against a stored ground truth, the score itself becomes untrustworthy in two directions at once: it marks a correct answer wrong when the formatting differs, and it marks a wrong answer right when the answer happens to match a ground truth that is itself wrong. The headline number hides both.

This skill does not grade the tool with another model's opinion. It re-derives the tool's real accuracy from the raw inputs, separates the formatting misses from the genuine failures, and hands back a receipt anyone can rerun. The qualitative judge layer, whether the tool is well named and well documented, is kept, but it stays labeled as an opinion and never enters the number.

How to run it

Run tool-eval on this tool evaluation. Take the committed run transcript (tool calls, answers, stored ground truth) and score it three ways: naive exact match, normalized value match (strip currency, commas, whitespace, then compare the numbers), and a recompute from the raw inputs that ignores the stored ground truth. Report the three accuracies, the format-only mismatches (correct value, scored wrong), and the silent-wrong cases (scored right, value wrong). Keep any tool-quality feedback as a separate judge layer. Verify every number with runnable code and write a receipt that fails closed.

Core commitments

• Verified numbers only. Every figure traces to a measurement produced by runnable code. No remembered or approximate values.
• Score value, not string. Exact match is the scorer that created the false negative. The object of study is whether the tool returned the right value, measured against the answer recomputed from the raw inputs, not against a stored string.
• Recompute the ground truth. A stored ground truth can be wrong. Where the task allows the answer to be recomputed from its inputs, do that, and treat any disagreement with the stored truth as a finding, not noise.
• Judge and receipt are different layers. A model's opinion on tool quality is useful commentary. It is never mixed into the reproducible accuracy.
• Reproducibility is part of the deliverable. The fixture regenerates from a fixed, model-free transcript. Report the compute envelope (here, none: standard library).
• Never regenerate committed fixture data. If the target already ships a committed transcript, treat it as immutable and re-derive against it as-is.

The procedure

Step 0 - Name the contract and the scorer

Write down what the tool is supposed to return (the value and its type), and how the existing eval scores it. If the scorer is exact string match against a stored ground truth, you already have your two suspect failure directions: format false-negatives and silent false-positives.

Step 1 - Score naive, normalized, and by recompute

On the committed transcript, compute three accuracies side by side: naive exact match (the existing scorer), normalized value match (parse the number, strip currency and separators, then compare), and recompute-from-inputs (compute the expected answer from the raw operands and compare the tool's value to that). The three numbers, and where they disagree, are the finding.

Step 2 - Separate the slices

Partition the failures: format-only mismatches (naive marks wrong, but the value is correct against the recompute), silent-wrong (naive marks right, but the value is wrong against the recompute), and true errors (wrong by both). The dangerous slice is silent-wrong: the eval passed it and it is incorrect.

Step 3 - Keep the judge layer separate

If a model offers feedback on tool naming, documentation, or error messages, record it under a clearly labeled judge section. It does not change the accuracy and is never folded into it.

Step 4 - Write the receipt

Produce a verify.py that re-derives all three accuracies and the slice membership from the raw transcript, imports nothing it cannot rerun, and exits non-zero if any committed number fails to reproduce. Bundle the fixture so the whole thing regenerates with no model and no GPU.

Report structure

# [Tool] Tool-Eval Receipt

## Summary
One paragraph: the naive score, the real (recompute) score, and the headline finding (for
example, "the naive score clears a wrong answer and fails a correct one").

## Three scorers
Naive exact-match, normalized value, and recompute-from-inputs accuracy, side by side.

## The slices
Format-only mismatches, silent-wrong, and true errors, each by task id.

## The receipt
What verify.py re-derives, and confirmation that it fails closed on a tampered transcript.

## Judge layer (labeled, not scored)
Any qualitative tool-quality feedback, kept separate from the number.

## Open questions
What is not yet established, and the measurement that would resolve it.

Guardrails

• Exact match is a format test, not a correctness test. Never report it as accuracy without the value-level number beside it.
• A stored ground truth can be the thing that is wrong. Recompute where you can; a tool that matches a wrong truth is still wrong.
• Silent-wrong is the headline, not the footnote. A wrong value that passes the eval is more dangerous than a correct value the eval failed.
• Do not let the judge layer leak into the number. Opinion about tool quality is not a measurement of tool correctness.

Worked illustration

A calculator tool eval scores 0.750 by exact string match. Re-derived from the raw operands it also scores 0.750, but on a different set of failing tasks: the naive scorer marks a correct answer wrong because it carries a dollar sign and a comma, and marks a wrong answer right because it matches a transposed, incorrect stored ground truth. Normalizing format alone lifts the number to 0.875 by recovering the false negative, but it still cannot see the silent error, because that failure is about truth, not format. The whole thing reproduces from the standard library with no model and no GPU, and the verification script fails closed if a single number is tampered. The full worked example is in agent-reliability-receipts.