Detecting doc drift

Detecting Doc Drift

Overview

A doc is a set of claims about the repo; drift is a claim the repo no longer backs. This skill declares the shape of drift detection so it runs the same way every time and can be invoked programmatically to trigger updates — not a one-off prose review.

Two non-negotiables make the output usable by automation:

1. A verdict requires evidence. Never mark a claim VERIFIED because doc and code "seem consistent." Verified means you ran the command, opened the line, or matched the grep.
2. The result is structured, not prose. Emit the declared record shape below. A human summary on top is fine; the structured block is the contract downstream tooling parses.

REQUIRED SUB-SKILL: Use writing-docs for any fix you propose — every fix must meet its bar (real output, no aspirational claims, marked+anchored rationale). This skill finds and classifies drift; writing-docs governs how the correction reads. fixing-doc-drift consumes this skill's records and applies the fixes (an optional auto-trigger layer, designed and shipped as the scheduling-doc-sync skill, wires detect→fix to cron/PR).

The engine (run these four steps, in order)

1. Extract each checkable claim from the doc, tagged by kind — one of exactly: command, path, symbol, behavior, structure, value (use these strings verbatim; automation switches on them). Pure prose is not a claim — except lines that sound factual but name no checkable thing ("robust", "production-ready", "reasonably fast", "handles most workloads"). Extract those too, kind value: they become UNVERIFIABLE. Do not skip them — an unbacked quality claim is the most common drift a human eye waves through. And they stay UNVERIFIABLE even when you can build a code case that the boast overreaches ("handles arbitrarily large inputs" vs. a whole-buffer read): put that argument in evidence, not in the verdict. STALE is reserved for claims with a checkable true value to restore — puffery has none, so any replacement line you'd draft is new authorship, and cutting or rewording it is a human decision, not a sync. In a scheduled install the human's third option is a durable waiver (.github/doc-sync/drift-waivers.json, owned by scheduling-doc-sync): an accepted claim stops resurfacing on run surfaces, while this skill keeps emitting it — detection stays pure; disposition is the pipeline's job. Lines already marked > UNVERIFIED: <claim> (the marker llm-doc-writer writes) are extracted like any claim and default to UNVERIFIABLE unless the repo now makes them checkable.
2. Verify each claim against the repo at the appropriate tier (below).
3. Classify each: VERIFIED / STALE / UNVERIFIABLE.
4. Emit the drift report (the contract below), then validate it mechanically before handing it off: pipe it through ${CLAUDE_PLUGIN_ROOT}/skills/detecting-doc-drift/scripts/validate-drift-output.py (reads the JSON on stdin or as a file arg). It enforces the enum/fix/evidence/summary rules and exits nonzero on any violation — don't emit a result it rejects. It checks shape, not whether a verdict is right; that judgment is still yours.

Verification tiers + escalation rule

Tier	Cost	Does	Catches
1 STATIC	seconds	grep/glob: path/symbol exists, command exists in Makefile/package.json, link resolves	renames, moves, deletions
2 SHALLOW	moderate	read the cited line; run safe --help/--version/dry-run	changed flags, values, signatures
3 DEEP	expensive	read implementing code; run the documented workflow where safe	behavior drift

Every claim starts at Tier 1. Escalate a claim only when (a) Tier 1 flags suspicion, (b) the claim's subject is in the diff (diff-scoped mode), or (c) a deep audit was requested. This concentrates cost where drift is likely.

Anchors: open the line, but judge the claim, not the line number. A file:line anchor is not evidence — open it. But the anchor is metadata on a claim, never its own claim: do not extract "the exit is at line 14" as a separate record and grade its precision. Classify the underlying claim on whether the referenced construct is there and the stated value is right. An anchor that lands a few lines off the exact statement (points at the guard instead of the exit() it guards) but still locates the right code is VERIFIED. Never emit a STALE record whose fix only changes a line number — a line-number-only correction is not drift. Mark STALE only when the value/behavior/symbol is wrong or the anchor points to a construct that moved or no longer exists.

The output contract (this is the "shape")

The drift report holds one record per extracted claim — STALE records drive fixes; UNVERIFIABLE records are surfaced for human review, not edit targets; VERIFIED records prove coverage. Each record uses exactly these fields (no extras): claim, location (a single file:line, no ranges), kind, tier, verdict, evidence, fix.

Rules: kind is one of command / path / symbol / behavior / structure / value; verdict is one of VERIFIED / STALE / UNVERIFIABLE — literal enum strings, no invented values. fix is non-null only for STALE, and it is the complete replacement text for the line at location — never an instruction like "change X to Y" — and must meet the writing-docs bar. evidence is mandatory for every verdict, including VERIFIED (the grep/command/line that proves it) — and it is one line: pointer + fact. The file:line or command, and the fact it shows. No history (prior PRs, how the drift arose), no restated command output, no reasoning narrative — the verdict carries the conclusion; evidence carries only what proves it. Emit the canonical wrapped object {"records": [...], "summary": {"verified": N, "stale": N, "unverifiable": N}}; on success the validator prints a summary: line as JSON, recomputed from the records, that automation can gate on.

See output-contract.md for a worked three-record example (a STALE command, a STALE behavior, an UNVERIFIABLE quality claim) with every field populated.

The validator (step 4) also cross-checks a wrapped object's summary counts against its records, and accepts a bare array, recomputing the authoritative summary for you.

Modes

• Full audit (manual / nightly sweep): extract every claim across the target docs, Tier 1 by default, escalate per the rule. Emit the full drift report, severity-ordered (wrong command/behavior before stale prose).
• Diff-scoped (PR check / what automation calls): input is a diff or commit range. For each changed file/symbol/value, grep every doc for passages referencing it — including command output blocks and examples, not just the obvious gotcha line — then verify each referencing claim, escalating past Tier 1 to the tier that settles it. A changed subject in the diff is exactly the (b) escalation trigger, so do not stop at a Tier-1 grep that only confirms the name still appears: read the cited line (Tier 2), and read implementing code (Tier 3) for any behavior/value claim, where the rename-grep gives false comfort. A rename/move/deletion is settled cheaper — don't pay Tier 3 for a claim Tier 1–2 already resolves. Output only records for touched claims. Completeness is the metric: a changed symbol referenced in five docs must produce five records.

Red flags — STOP

• Writing "looks consistent" / "should be fine" without opening the file → not a verdict.
• Trusting a file:line anchor instead of reading that line → anchors drift a few lines; read the line and judge the claim (the anchor rule above).
• Emitting a prose report with no structured records → automation can't trigger on it.
• A VERIFIED record with an empty evidence field → unverified; go get the evidence.
• Diff-scoped run that checked the one obvious doc → grep ALL docs for the changed subject.
• Eyeballing a command table instead of grepping the Makefile/package.json for each target.
• Skipping a "robust"/"fast"/"production-ready" line because it's "just prose" → extract it as UNVERIFIABLE; that is the finding.
• Marking a quality boast STALE because the code argues against it, and drafting a replacement → still UNVERIFIABLE; the contradiction goes in evidence. A fix must restore a checkable true value, not reword puffery.
• A record with an invented kind (e.g. schema_mismatch) → use the six enum values only.
• Marking an anchor STALE for being off by a line, or emitting a fix that only changes a line number → not drift. The anchor is metadata, not its own claim.
• Evidence that tells a story — prior fixes, what re-staled the line, pasted command output → one line, pointer + fact. History lives in git; the record proves, it doesn't narrate.