Token eval harness

Skill yashpatil582/agent-skills/token-eval-harness

Anthropic Agent Skills packaging shipped eval methods: gaming-resistant clinical-note grounding eval + token-F1 scorer. Stdlib-only, runs offline.

Install

npx -y skills add yashpatil582/agent-skills --skill token-eval-harness

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Scores free-text model output against reference text using token-level multiset F1 (precision, recall, F1) — the honest surface-overlap metric used to evaluate generated clinical notes, RAG answers, summaries, and extractions against gold references. Use this whenever asked to compute token F1 or token-overlap F1; to batch-evaluate predictions against references from a JSONL file; to reproduce or extend the Open-Scribe PriMock57 / ACI-Bench note-generation eval; to macro-average per-example F1; or to set a fast, deterministic regression baseline that does NOT claim semantic equivalence (dense clinical shorthand scores ~0.28 even when clinically correct). Triggers on token F1, F1 score, precision recall, eval harness, reference-based scoring, PriMock57, ACI-Bench, whole-note F1, section F1, overlap metric, gold reference. Fully offline — no LLM or API key required.

The file declares its own license as Apache-2.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

5.2 KB, as published. Nobody here has run it

Token Eval Harness

Score free-text model output against a reference (gold) text with token-level multiset F1 — precision, recall, and F1 over the tokens the two texts share — per example and macro-averaged across a batch. This is the scoring core of the Open-Scribe PriMock57 / ACI-Bench eval (eval/metrics.py), wrapped as a fully offline batch runner.

When to use this

Use this skill to:

Compute token F1 (precision / recall / F1) of a prediction against a reference.
Batch-score predictions vs references from a JSONL file and get a macro-average.
Reproduce or extend the Open-Scribe PriMock57 / ACI-Bench note-generation eval.
Get a fast, deterministic regression signal for RAG answers, extractions, or summaries.

Do not read a single absolute score as a quality grade — token F1 is surface overlap, not semantic similarity (see Limitations). For paraphrase-aware grounding, use clinical-note-eval or a semantic judge instead.

The metric

Tokenize both texts (lowercase; keep 138/86, well-controlled, 7.1 whole; drop punctuation), take the multiset overlap (repeats preserved), then:

precision = overlap / |prediction tokens|
recall    = overlap / |reference tokens|
f1        = 2·P·R / (P+R)         (0 when P+R == 0)

Edge cases: both empty → 1.0; exactly one empty → 0.0. whole_note_f1 scores the whole text as one bag (free-form references); section_f1 scores a section -> text map per section (structured/SOAP references). Full detail: references/methodology.md.

Workflow

- [ ] Build a JSONL file: one {"id", "prediction", "reference"} object per line.
- [ ] Run the scorer: python scripts/run_eval.py --input <file.jsonl>.
- [ ] Read per-example F1 + the macro whole-note aggregate; --out to save full JSON.

Worked example (offline, no key):

python scripts/run_eval.py --input assets/sample.jsonl

Four hand-verifiable rows → per-example F1 of 1.0, 0.8, 0.0, 0.7 and macro F1 0.625.

Input / output

Input JSONL, whole-note shape (sectioned shape and the full results schema: references/data-format.md):

{"id": "ex1", "prediction": "patient has a fever and cough", "reference": "patient has fever cough"}

Output aggregate (keys match upstream Open-Scribe run_eval.py):

{
  "aggregate": {
    "n": 4, "n_failed": 0, "elapsed_sec": 0.001,
    "whole_note_f1_mean": 0.625,
    "whole_note_precision_mean": 0.5917,
    "whole_note_recall_mean": 0.675
  },
  "per_example": [
    {"id": "extra-tokens", "whole_note": {"precision": 0.6667, "recall": 1.0, "f1": 0.8}}
  ]
}

Whole-note vs section F1

Whole-note — free-form references (PriMock57 prose, RAG answers, summaries). This is what the published PriMock57 number is computed from.
Section — structured references (e.g. SOAP); returns one P/R/F1 per section. Offered here, but not what the upstream PriMock57 run reports.

Limitations (read these)

Token F1 is surface overlap, not semantic equivalence. Clinically equivalent text in different words scores low.
Concretely, Open-Scribe's published PriMock57 baseline is macro F1 0.279 (P 0.266, R 0.315, 57/57) and ACI-Bench 0.447 (35/40) — because dense shorthand gold notes (3/7, LLQ, hx) share few surface tokens with full-English prose, even when the prose is correct. A low number here is a property of the metric, not necessarily a broken model.
Best used as a regression signal and as one input alongside a semantic check.

Provenance

Ported verbatim-equivalent from Open-Scribe eval/metrics.py (Apache-2.0): https://github.com/yashpatil582/open-scribe. This skill ports the scoring core and adds an offline JSONL batch runner; the upstream harness additionally runs ASR + note generation before scoring. Related live demo: the "Score a note" eval at https://yashpatil582.github.io/#eval.