Retrieval verification

Skill marsmike/agentic-toolkit/plugins/obsidian/skills/retrieval-verification

Vault-first toolkit for Claude Code — curated plugins over an Obsidian-compatible knowledge vault, governed by an explicit contract, gated by evals, continuously delivered.

Install

npx -y skills add marsmike/agentic-toolkit --skill retrieval-verification

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

10 days oldThe repository was created 10 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Audit vault note descriptions by predicting content from title+description alone and scoring the prediction against the real body. Use for periodic vault maintenance or after a bulk distill/import.

SKILL.md

3.1 KB, as published. Nobody here has run it

Retrieval Verification

The description-quality loop this toolkit's contract cites as the maintenance-facing half of the dual-channel-descriptions concept (docs/PLAN.md — retrieval-verification loop): a description earns its keep only if a reader (or a BM25 query, or you) could predict the note's actual content from title + description alone, without opening it.

This is an agent-executed workflow, not a script you run to completion — the predict-then-score step is your own reasoning, done deliberately blind to the body. scripts/retrieval_verification.py handles the two mechanical halves around it (sampling, and turning your scores into a report).

Workflow

Sample. Pull N active notes, title + description only — the script withholds the body on purpose:

uv run --project "$CLAUDE_PLUGIN_ROOT/scripts" python3 "$CLAUDE_PLUGIN_ROOT/scripts/retrieval_verification.py" sample --n 15 --json > samples.json

Predict, then read, then score — per note, in that order. For each sampled note:
- Read only its title and description from samples.json. Write down what you expect the body to contain.
- Only then read the actual note (Read the path).
- Score 1-5: does the body match what the description predicted?
  - 5 — description alone would have led you straight to this content.
  - 3 — plausible but generic; several different notes could share this description.
  - 1 — description is actively misleading or says nothing the title didn't.
- A note with no description at all scores as flagged automatically — it can't be predicted from what isn't there.
Build the scores map — {"<path>": {"score": N, "predicted": "...", "note": "why"}} for every sampled path — and hand it to the reporting half:
```
uv run --project "$CLAUDE_PLUGIN_ROOT/scripts" python3 "$CLAUDE_PLUGIN_ROOT/scripts/retrieval_verification.py" \
  report --samples samples.json --scores scores.json
```
This writes a JSON report to 00_Memory/retrieval-verification/<timestamp>.json and appends a summary capture to 01_Capture/ (flagged notes named, with the report path) — a normal capture-inbox item for a human or a later distill pass to triage, per contract/VAULT_SCHEMA.md's capture conventions.
Rewrite flagged descriptions (score < 3) directly with Edit — this skill surfaces the problem, it doesn't silently rewrite content on your behalf.

Dead-letter behavior

If a sampled path never gets a score (an interrupted run), report writes a dead-letter note to 00_Memory/dlq/ naming exactly which paths are missing, rather than silently producing a report that looks complete. See distill's rules.md for the DLQ convention this follows.

Reference

methodology.md — why prediction-then-score, BM25 dilution, sample-size guidance