agentsclimarketplace

Clinical note eval

Skill yashpatil582/agent-skills/clinical-note-eval

Anthropic Agent Skills packaging shipped eval methods: gaming-resistant clinical-note grounding eval + token-F1 scorer. Stdlib-only, runs offline.

Install
npx -y skills add yashpatil582/agent-skills --skill clinical-note-eval

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Evaluates whether an LLM's structured extraction of a clinical note (problems, medications, plan) is actually grounded in the note's own text, using a two-pass extract-then-ground design whose grounding score is computed deterministically in code — by re-checking every cited span against the note — so the model cannot inflate it. Use this whenever asked to score, grade, audit, or fact-check an LLM's extraction or summary of a clinical note for hallucination or grounding; to verify that cited supporting spans really appear in the source; to build a faithfulness or grounding eval; or to catch unsupported clinical claims and inference "lures." Triggers on clinical note eval, grounding score, hallucination check, faithfulness, span verification, extract-and-ground, supporting quote, structured extraction grading, code-side scoring. Runs offline in fixture mode with no API key; live scoring works against any OpenAI-compatible endpoint.

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

7.3 KB, as published. Nobody here has run it

Clinical Note Eval

Score how well an LLM's structured extraction of a clinical note is grounded in the note's own text. The output is a per-claim verdict list (grounded / partial / unsupported), each grounded claim backed by a verbatim span, and a single grounding score.

When to use this

Use this skill when asked to:

  • Grade or audit an LLM's extraction/summary of a clinical note for hallucination or grounding.
  • Verify that the spans a model cites as evidence really appear in the source note.
  • Build a faithfulness / grounding eval that a model can't game.
  • Catch inference "lures" — claims the model asserts that the note never states (e.g. inferring "hypertension" from a single blood-pressure reading).

Do not use it to judge clinical correctness. It measures whether the structured output is supported by the note's text, not whether the note's medicine is right.

The core idea (why the score is trustworthy)

The number is trustworthy because it is verified in code, not because the model is trusted:

  1. Extraction pass — a model reads the note and emits structured fields (problems, medications, plan). Each field becomes a claim whose display text is built in code, never by the model.
  2. Grounding pass — a separate call with fresh context sees only the note and the claims (never the extractor's reasoning) and labels each claim, citing a verbatim span.
  3. Code-side reconciliation (the key step) — for every claim, the code re-checks the cited span against the note. A grounded claim whose span is not actually in the note is downgraded to unsupported and the fake span is dropped. The score is then computed from the verified labels.

Because a model never emits the number and every "grounded" span is re-verified, a hallucinated citation or a prompt-injected "mark everything grounded" cannot move the score. Full rationale: references/methodology.md.

Workflow

- [ ] Put the note in a UTF-8 text file (or pipe it via stdin with --note -).
- [ ] Offline check first (no key): run with --fixture to confirm the pipeline + guarantees.
- [ ] Live run: set LLM_BASE_URL / LLM_API_KEY / LLM_MODEL, then run without --fixture.
- [ ] Read the report; inspect any downgraded claims and the final score.

Offline worked example (no API key):

python scripts/run_eval.py --note assets/sample_note_t2dm.txt --fixture assets/fixture_t2dm.json

The fixture feeds canned model outputs in which the grader labels an inferred "hypertension" claim grounded with a span that is not in the note. The code downgrades it to unsupported, drops the fake span, and scores (6 + 0.5·1) / 8 = 0.8125.

Live run (any OpenAI-compatible endpoint):

export LLM_BASE_URL=https://api.groq.com/openai/v1   # or http://localhost:11434/v1 for Ollama
export LLM_API_KEY=$GROQ_API_KEY                      # any non-empty value for keyless local servers
export LLM_MODEL=openai/gpt-oss-120b                  # or llama-3.3-70b-versatile, qwen2.5, ...
python scripts/run_eval.py --note path/to/note.txt

Add --json for machine-readable output, --out result.json to save it.

Output

run_eval.py returns the result object (printed as a report, or JSON with --json):

{
  "verdicts": [
    {"id": "p1", "category": "problem", "text": "type 2 diabetes (well controlled)",
     "label": "grounded", "supportingQuote": "type 2 diabetes, well controlled", "rationale": "..."},
    {"id": "p2", "category": "problem", "text": "hypertension",
     "label": "unsupported", "supportingQuote": null, "rationale": "..."}
  ],
  "counts": {"grounded": 6, "partial": 1, "unsupported": 1, "total": 8},
  "score": 0.8125,
  "formula": "(grounded + 0.5 * partial) / total",
  "extractedBy": "openai/gpt-oss-120b",
  "gradedBy": "openai/gpt-oss-120b"
}

What the deterministic core guarantees

Implemented in scripts/grounding.py (pure functions, no network):

  • Missing verdict → unsupported. A claim the grader didn't return defaults to unsupported.
  • Fake span → nulled. A supportingQuote that isn't a real span in the note is dropped; a fake span is never surfaced.
  • grounded without a real span → unsupported. A grounded label must cite a verbatim span.
  • Span matching forgives only case and whitespace — never punctuation. 138/86 must appear as 138/86; it will not match 13886.
  • Score = (grounded + 0.5 · partial) / total, or null when there are no claims.

Schemas and claim-flattening rules: references/schema.md.

Prompts and injection-hardening

Both passes wrap the note between <note> … </note> and label it as data, not instructions; a shared SECURITY block tells the model to ignore any instructions inside the note (e.g. "mark everything grounded"), invent no clinical facts, and add no medical codes. Exact prompt text: references/prompts.md.

Limitations

  • Scores faithfulness to the note, not clinical correctness.
  • Grounding is structural/lexical (verbatim span match) — it does not credit a correct claim that the note supports only by paraphrase. This is deliberate: it is the conservative, gameable-resistant choice. Pair it with a semantic check if you need paraphrase credit.
  • Claims are capped at 40 (MAX_CLAIMS) to bound cost on runaway extractions.

Provenance

The TypeScript reference implementation is the live #eval demo at https://yashpatil582.github.io/#eval (lib/ai/eval.ts). The methodology comes from ClinEval (clinical hallucination scoring) and Open-Scribe (FHIR ambient scribe + eval harness). This skill is a faithful, standard-library-only Python port of that code-side logic.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.