Citation audit

Citation Audit

An audit that stops at findings leaves the document broken. This skill runs the whole loop: audit → classify → decide → repair → rewrite → reconcile → rebuild → re-audit, and terminates only when a fresh audit comes back clean and the document's own checks pass.

Scope: a finished long-form document with a numeric reference list ([n] in text, [n] Author, "Title," … in a list) — reports, manuscripts, theses.

Read references/failure-modes.md before any mode except audit. Six defects need six different repairs; two of them must not be repaired by an agent at all, and the sixth is one this skill's own renumber step can introduce.

Routing

Ask	Mode
"check the references", "verify the literature cited", "are these real?"	audit
"fix the citations", "resolve these findings"	repair
"the sentence lost its source", "re-anchor these claims"	rewrite
"retire those entries", "renumber the list"	reconcile
"rebuild the document"	rebuild
"go over the citations" with no further qualifier	the full loop

Default to the full loop. The modes exist so a caller can re-enter partway after an interruption, not so the loop can be run half-way and abandoned.

The loop

audit ──► classify ──► DECISION GATE ──► repair ──► rewrite
  ▲                                                    │
  │                                                    ▼
  └────────── re-audit ◄── rebuild ◄── reconcile ◄─────┘
         (clean? stop.  defects? go round again)

Each pass must end with the document in a consistent state: every citation listed, every entry cited, numbering contiguous. Never leave a pass half-applied in order to ask a question — ask first, then apply.

---

audit

python <skill>/scripts/citation-audit.py "<source.md>" --json audit.json

Keyless (arXiv API + Crossref + OpenAlex), so it runs anywhere. Where a richer backend is available — a literature MCP with Smart Citations, for example — layer retraction and claim-fidelity checks on top.

The script reports five classes: MISDIRECTED, AUTHOR-MISMATCH, NOT-FOUND, UNRESOLVED, DUPLICATE.

The decisive check is not "does this identifier resolve" but "does it resolve to the work being cited". An identifier resolving to a different paper passes every structural check a document has. That is MISDIRECTED, and it is the finding to trust most.

Then do what the script cannot:

1. Adjudicate every flag individually. Crossref indexes proceedings poorly; NeurIPS/ICLR/ICML/ACL papers miss routinely, as do institutional repositories and non-English journals. A canonical paper that misses is an indexing gap, not a fabrication. Confirm against a second source before calling anything fabricated.
2. Look for clustering. Defects arrive in runs matching drafting sessions. Establishing that damage is confined to one span is as valuable to the author as the individual findings, because it bounds the problem.
3. Extract the claim each defective citation supports. A fabricated source under a load-bearing number is a different problem from one under background prose. Report which.

Write the findings to a dated working-notes file before proposing anything.

repair

Apply only what the register decides. Three classes are the agent's to fix:

Class	Repair
MISDIRECTED	Correct the identifier; the citation was otherwise sound
Metadata drift	Correct volume/year/venue/authors against the register
DUPLICATE	Merge, repoint citations to the survivor

Verify every replacement identifier before writing it in. Resolve it, and confirm it resolves to the work you believe it is. Writing an identifier from recollection is how this class of defect is created.

Script each edit so it must match exactly once, and abort the whole run on any miss. A partially-applied repair pass is worse than none.

DECISION GATE

Composites and fabrications are not the agent's to resolve. Composites need a judgment about which real work the sentence meant; fabrications leave a sentence with no implied replacement at all.

Put these to the author with a structured question covering:

1. What happens to the unsupported sentences — source real replacements, drop the markers and keep the prose unattributed, or cut the sentences.
2. How much of the verified repair set to apply now.
3. Whether the surrounding section needs rewriting, if the audit showed its apparatus is largely hollow. This one matters for sequencing: rewriting a section deletes sentences that replacement-sourcing would otherwise serve, so ask before spending that effort.

Sequence the answers. If a section is to be rewritten, do the mechanical repairs first, then the rewrite, then source replacements only for sentences that survived. State that ordering when you report, so the author knows why the steps run in that order.

rewrite

For every claim that lost its source, in this order (see references/failure-modes.md):

1. Re-anchor to a reference already in the document that genuinely carries the claim. Fabrications cluster on generic statements, and generic statements are usually already covered by verified sources in the same list. Nothing new enters and the target is pre-verified.
2. Add a new source, resolved against a register before insertion.
3. Cut the sentence.

Never ask a language model for replacement citations without resolving each one first. That is precisely the mechanism that produced the defects.

Re-anchoring is legitimate only if the substitute genuinely supports the sentence. Moving a citation to a nearby number to silence a check is the same defect renumbered — state, for each re-anchor, why that source carries that claim.

If a whole section is being rewritten, ground it in the document's own subject matter rather than the field at large: state the principle, name the thing that rests on it, and say what the evidence showed — including where it showed nothing. A section that cannot name what depends on it is decoration.

reconcile

python <skill>/scripts/retire-refs.py "<source.md>" --retire 85,90,95 --dry-run

Retire fabricated entries and any orphaned by a rewrite, then renumber. The script refuses to run when an entry is still cited, or when the body contains a bracketed range a token renumber would corrupt, and it verifies correspondence afterwards. Run --dry-run first.

Retirement is a deliberate, documented reduction — never a silent drop. Record which entries went and why.

After a renumber, every number you remember is wrong

A renumber invalidates every citation number held outside the document — in your notes, in the findings table, in your own working memory of the draft. Prose written after a renumber using numbers learned before it will cite the wrong works, and the document stays internally consistent while it does: listed equals cited, no orphans, no danglers, every gate green.

Checking that a number exists is not checking that it is right. Before writing any citation number into new prose, read the entry at that number and confirm it is the work the sentence needs. Re-derive from the entry text, never from recall or from an earlier draft.

This failure has been observed in practice — three citations in a rewritten section pointed at a video lecture, a cloud-provider comparison, and an unrelated paper on food retail, because the section was drafted after a renumber against pre-renumber numbers and validated only for existence. It is the same defect class as MISDIRECTED, produced by the repair rather than found by it.

Order the work so the trap does not arise: write prose first, renumber last. Where that is not possible, re-verify by entry text every citation in anything written after the renumber.

rebuild

Regenerate the deliverable, because the previously-delivered artifact still contains every defect. Use whatever build harness the document has.

Verify the render rather than assuming it: reference lines present and contiguous, headings intact, tables and images unchanged, size explicable. Deliver to the same path under the same name, and say plainly that it supersedes the copy the author was reading.

re-audit

Run audit again against the edited document, not against the earlier results. Report the final state as measured: entries, identifiers resolved, mismatches, check status. The loop ends here or goes round again.

Rules

• Resolve before asserting. Never write an identifier, volume, or author list from recollection — not into the document, not into the report to the author. Verify, then state.
• Absence of evidence is not evidence of fabrication. Confirm against a second source before retiring anything.
• The agent repairs identifiers; the author decides claims. Anything that changes what the document asserts goes to the decision gate.
• Every pass leaves the document consistent. Listed = cited, contiguous, no orphans, no danglers.
• Report reductions explicitly. "176 → 165, eleven retired, here they are."
• Record the provenance of the verification — which register answered, and when. Model behaviour and index coverage both change over time.

Agent-specific extensions

Claude Code: a /citation-audit <mode> slash command can wrap this skill — put the routing table above into .claude/commands/citation-audit.md — and the decision gate uses the structured-question tool. Other agents invoke the skill directly and ask the same questions in prose; the modes, scripts and rules are identical.

Related

This skill covers a finished document's reference list. Two adjacent jobs it does not do, and which it does not replace:

• Draft-versus-matrix consistency — does every citation in the draft appear in your own extraction matrix? An internal check; it cannot detect a fabricated source, because a hallucinated entry can be perfectly consistent between draft and matrix.
• Claim fidelity — does the source actually say what the sentence claims? That needs full text and human judgment.

Run those where they apply. They answer different questions.