Citation verification

Skill AllanWessels/Bratan/skills/citation-verification

Bratan is a self-improving Retrieval-Augmented Generation framework built on an adversarial three-agent loop

Install

npx -y skills add AllanWessels/Bratan --skill citation-verification

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use this skill to add a post-generation pass that verifies every citation in the LLM's answer actually appears in (and is supported by) the cited chunk. Reach for it when the faithfulness score is low even though retrieval is healthy, or when invented citations are surfacing as a failure category.

SKILL.md

3.0 KB, as published. Nobody here has run it

Citation Verification

When to use

Add a verification pass when /reports/latest.json shows:

High retrieval recall (>= 0.7 avg) but low faithfulness (< 0.6 avg)
Failures in the "invented citation" category, surfaced by the judge

Don't add it if faithfulness is already high — it's an expensive pass that you should only enable when the evidence demands it.

How it works

After the LLM generates an answer with chunk-numbered citations, run a separate verification pass:

Parse the answer into (claim, cited-chunk-id) pairs. Most citation formats like ... [3] ... are easy to regex.

For each (claim, chunk) pair, call a fast LLM with a fixed prompt:

Claim: {claim}
Chunk: {chunk_text}

Does the chunk directly support this claim? Reply with one word:
YES, NO, or PARTIAL.

If all pairs return YES, the answer is verified. If any return NO, the answer has at least one fabricated citation. PARTIAL is a soft warning.

What to do on failure

Three reasonable strategies depending on stakes:

Retry: regenerate the answer with the same chunks but stronger grounding instructions. Usually fixes single-claim issues.
Downgrade: replace the offending claim with "I'm not sure" or drop it from the answer. Keeps the rest of the answer usable.
Refuse: return an honest "I don't have enough information" response. Highest-trust, lowest-coverage option.

For most consumer-facing systems, retry once then downgrade. For high-stakes (medical, legal, financial) systems, refuse rather than serve unverified content.

Implementation requirements

Your generation prompt needs to use a citation format the verifier can parse:

Use citations in the format [N] where N is the chunk number.
Place the citation immediately after the claim it supports.
Every factual claim must have a citation.

Then your verifier parses on \[(\d+)\] and matches each claim (sentence containing the citation) to the chunk by number.

Cost considerations

A verification pass adds one LLM call per cited claim. For a typical answer with 3-5 citations, that's 3-5 small LLM calls added to each query. The calls are cheap (small context, single word output) but they're real latency. Budget for them.

Optimization: many claims share chunks. Group claims by chunk and verify in batch — one call per (chunk, set-of-claims) pair instead of per claim.

When NOT to use

Low-stakes use cases where occasional fabrication is acceptable
Cases where the user can verify the answer themselves
Early iteration — focus on retrieval and prompt-level grounding before adding a verification pass