Hallucite

hallucite

Detect fabricated references in academic paper PDF files. Three stages: extract and verify use no LLM (verification queries the online databases unless --offline restricts it to the local ones); triage is the only step that uses an LLM (cloud or local), run on the references that no database could confirm.

Stop conditions -- never fabricate a verdict

This skill exists to catch fabricated references; it must never produce one. A hallucination verdict is a serious accusation against named authors, so every claim you make must trace to real tool output, not to your own reading of a .bib/.bbl/PDF.

• No script output ⇒ no verdict. A reference may be called real, partial-match, likely-hallucinated, or unclear only on the basis of (a) a Stage 1+2 db_verification record the audit actually wrote, or (b) Stage 3 web-search evidence you actually gathered. Eyeballing a bibliography is not a verification method and never yields a verdict.
• Treat any non-zero exit or a HALLUCITE_BOOTSTRAP_FAILED: line as a blocking error. Stop, report the failure verbatim to the user, and do not work around it by inspecting the references yourself. "The tool would not run" is the correct, honest outcome -- not a hand-written report.
• Run the preflight (check-env) first. If it does not print HALLUCITE_OK, the environment is not ready; surface its message and stop.
• If commands start erroring or returning empty output, halt and say so. Do not begin assembling findings from memory or from the source files while the pipeline is broken.

Running the pipeline

Always invoke the pipeline through the run.sh wrapper. It resolves (or, on first use, provisions) a Python 3.12 that can import hallucinator, so you never call a bare python/mise/uv that may be missing from the plugin's shell. It is the single supported entry point.

resolve_hallucite_run() {
  # 1. Claude Code plugin install.
  if [ -n "${CLAUDE_PLUGIN_ROOT:-}" ]; then
    candidate="${CLAUDE_PLUGIN_ROOT}/skills/hallucite/scripts/run.sh"
    [ -x "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }
  fi

  # 2. Codex repo-local skill discovery shim.
  candidate=".agents/skills/hallucite/scripts/run.sh"
  [ -x "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }

  # 3. Direct repo clone.
  candidate="skills/hallucite/scripts/run.sh"
  [ -x "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }

  # 4. Claude Code plugin cache (version-scan fallback for when CLAUDE_PLUGIN_ROOT
  # is unset in the tool shell, which skips branch 1 even on a valid install).
  # Choose the highest cached version (sort -V) so a marketplace update always
  # wins here -- and so a stale Codex cache copy can never be preferred over a
  # current Claude install.
  claude_cache_root="${CLAUDE_CONFIG_DIR:-$HOME/.claude}/plugins/cache"
  if [ -d "$claude_cache_root" ]; then
    candidate="$(find "$claude_cache_root" -path '*/hallucite/hallucite/*/skills/hallucite/scripts/run.sh' -type f 2>/dev/null | sort -V | tail -n 1)"
    [ -n "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }
    candidate="$(find "$claude_cache_root" -path '*/hallucite/*/skills/hallucite/scripts/run.sh' -type f 2>/dev/null | sort -V | tail -n 1)"
    [ -n "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }
  fi

  # 5. Codex plugin cache. Prefer the hallucite marketplace's hallucite, then any
  # marketplace's hallucite, choosing the highest cached version (sort -V, so
  # 1.10.0 beats 1.9.0 -- plain lexicographic sort would invert them).
  cache_root="${CODEX_HOME:-$HOME/.codex}/plugins/cache"
  if [ -d "$cache_root" ]; then
    candidate="$(find "$cache_root" -path '*/hallucite/hallucite/*/skills/hallucite/scripts/run.sh' -type f 2>/dev/null | sort -V | tail -n 1)"
    [ -n "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }
    candidate="$(find "$cache_root" -path '*/hallucite/*/skills/hallucite/scripts/run.sh' -type f 2>/dev/null | sort -V | tail -n 1)"
    [ -n "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }
  fi

  printf '%s\n' 'HALLUCITE_BOOTSTRAP_FAILED: cannot locate hallucite scripts/run.sh' >&2
  return 3
}

RUN="$(resolve_hallucite_run)" || exit $?

Subcommands: run.sh check-env (preflight), upgrade, audit ..., triage ..., lint ..., python .... On success it is transparent (runs the script, forwards its exit code); on a setup failure it prints HALLUCITE_BOOTSTRAP_FAILED: <reason> to stderr and exits non-zero -- that sentinel means no audit ran, so there is nothing to interpret.

First run builds a cached venv at ${XDG_CACHE_HOME:-~/.cache}/hallucite/venv (needs uv or a Python 3.12, plus network for pip install hallucinator); set $HALLUCITE_VENV to relocate it. Later runs reuse it and are instant. To reuse an environment that already has hallucinator, set $HALLUCITE_PYTHON to its interpreter and no venv is built. (In a repo clone you can equivalently use the mise run ... tasks.)

Preflight (every session, before any audit)

"$RUN" check-env    # must print `HALLUCITE_OK: <python> (hallucinator <version>)`

If it prints HALLUCITE_BOOTSTRAP_FAILED: instead, relay that line to the user and stop -- see the stop conditions above.

Two different versions show up around this skill; keep them apart when reporting one. check-env prints the hallucinator package version (the extract/verify library inside the venv), which is not the hallucite plugin/skill version in metadata.version above.

Keeping hallucinator current

The managed venv is reused as soon as it can import hallucinator, so it never upgrades on its own and can sit on a months-old release indefinitely. check-env compares it against PyPI and warns when a newer one exists; act on that warning:

"$RUN" upgrade      # -> `HALLUCITE_OK: hallucinator <old> -> <new> at <python>`

It refuses when $HALLUCITE_PYTHON is set, since that interpreter is yours and run.sh does not modify it -- upgrade it yourself there. Set $HALLUCITE_NO_VERSION_CHECK to skip the PyPI lookup when offline.

Setup (once)

1. pip install hallucinator. It ships CPython 3.12 wheels; on 3.13 pip builds from source, so a 3.12 venv is the easy path (uv venv -p 3.12).
2. Build the offline DBLP database with the hallucinator CLI (prebuilt binary from https://github.com/gianlucasb/hallucinator/releases/latest, checksum-verified, or cargo install hallucinator-cli): hallucinator-cli update-dblp ~/hallucite/dblp.db (about 4.6 GB download, 20-30 min, builds a ~2.5 GB SQLite+FTS5 file). Keep it outside protected dirs such as ~/Downloads. To store it elsewhere, set $HALLUCITE_DBLP to the target path and pass that path here instead.
3. Updates: the audit (Stage 1+2 below) checks the database's age at run time and warns when it is over 30 days old. Recent papers cite recent work, so rebuild it when that warning appears.

Resolve the target

A directory of PDF files, or a single <file>.pdf. Given a bare paper number or name, resolve it against the directory the user means (ask if ambiguous).

Stage 1+2: extract and verify (no LLM)

"$RUN" audit <pdf-file-or-dir> --out <outdir> --mailto <your-email>

Writes <outdir>/<paper_id>.json (every reference, parsed fields plus per-database verification) and <outdir>/summary.json. The offline DBLP DB defaults to $HALLUCITE_DBLP (else ~/hallucite/dblp.db); override it with --dblp PATH. Flags: --offline (no network: offline DBLP plus hallucinator's built-in Standards matcher; a missing DBLP file disables DBLP rather than falling back to dblp.org), --disable-dbs LIST (disable named backends, comma-separated), --no-verify (extraction only), --retry-degraded N (re-check references a backend failed to answer for; default 1), --no-candidates (skip the CrossRef lookup that attaches candidate real records to unverified references; implied by --offline). Extraction is lineno- and two-column-aware and handles numeric, bracket-label, and author-year bibliographies; the target is 0 unparsed references. References the backends miss get two automatic local recovery passes before they land in the worklist: an all-candidates title+author check against the offline DBLP file (shown as source DBLP (hallucite)), and a re-verification with line-break hyphens removed.

Read the run's closing warnings. Verification stops at the first backend that matches, so later backends are only ever asked about the residue -- the same references that reach triage -- and that is where rate limiting lands. The audit therefore marks any reference a backend failed to answer for as degraded and prints a per-backend tally at the end. A backend that answered nothing weakens every not_found it should have had a say in; if the tally looks bad, re-run before triaging rather than judging references on an incomplete check. The closing warnings also list papers that yielded 0 references -- an unsupported bibliography layout means that paper was never checked at all -- and the per-paper output flags author-year references that look like two entries merged into one (the second was then never verified on its own); check those in the PDF.

References are named by the handle a reader can actually find. A numbered bibliography keeps its printed [12]; an author-year one is named by its citation key (de Dieu et al. (2025c)), since nothing is numbered in the paper. A trailing [#n] is hallucite's own index, appears nowhere in the paper, and is only what triage record takes as its <number>.

If this exits non-zero -- whether a HALLUCITE_BOOTSTRAP_FAILED: line (no Python/hallucinator) or a per-paper error from the audit -- stop and report it; do not infer verdicts by hand (see the stop conditions above).

Stage 3: triage the residue (LLM)

"$RUN" triage worklist --out <outdir>             # -> <outdir>/triage_worklist.json
"$RUN" triage worklist --pending --out <outdir>   # only refs not yet recorded
"$RUN" triage worklist --paper <paper_id> --out <outdir>  # one paper's slice
"$RUN" triage status --out <outdir>               # per-paper done / pending counts

These read the per-paper JSON the audit has already written, so Stage 3 can run on finished papers while Stage 1+2 is still processing the rest. Verdicts accumulate per <paper_id>:<number>, and --pending surfaces only the references that have not been recorded yet.

To fan triage out across papers, give each worker its own slice with worklist --paper <paper_id> (exact id match, written to triage_worklist-<paper_id>.json) rather than the shared worklist -- a worker that self-filters the corpus-wide file by id can grab the wrong paper (e.g. paper6 vs paper66); the per-paper slice removes that step and errors out on an unknown id. record takes an exclusive lock on the verdicts file, so concurrent workers can record in parallel without losing each other's verdicts.

For each entry (references whose db_verification.status is anything other than verified -- not_found, mismatch, or unparsed), investigate with parallel web queries and classify it:

• Start from the evidence the audit already gathered. A worklist entry carries candidates (CrossRef's closest real records, with their DOIs and a title similarity) and matched (what a backend matched, and the authors it holds). Check those before searching: a high-similarity candidate usually is the cited work, and confirming or rejecting it is faster and more reliable than a blank-page search. For a mismatch, open the matched record first and diff it field by field -- and confirm it is even the same work, since a backend can match a thesis, preprint, or extended abstract that shares its paper's title.
• degraded: true means the check was incomplete -- a backend errored or rate-limited, so the databases that would have matched this reference may never have been asked. Its not_found is not evidence of anything. A fabrication verdict on a degraded entry must rest entirely on your own searches; if those are inconclusive, the answer is unclear, not likely-hallucinated.
• Search in escalating breadth; only conclude "not found" after the broad pass. Start with the DOI (resolve https://api.crossref.org/works/<doi>; a 404, or resolution to an unrelated paper, is a strong fabrication signal), then the exact title in quotes plus the first-author surname (and site:arxiv.org "<title>"), then the exact title in quotes alone and on Google Scholar.
• If those find nothing, drop the quotes and search the title as plain keywords, then read the results for a genuine match. Obscure or predatory venues are poorly indexed, so a narrow query returning nothing is not evidence of fabrication; broaden first, and fetch the venue page or any embedded DOI/URL directly.
• When a narrow search returns the same authors under different titles, that is the signal to escalate, not a conclusion. It is simultaneously the fabrication signature and exactly what a real but poorly-indexed, very recent, or abbreviated-title paper looks like at narrow breadth. Before any title_match=no, run a fuzzy metadata query -- https://api.crossref.org/works?query.bibliographic=<title+first+authors>&rows=5&select=title,author,container-title,issued,DOI -- and a plain-keyword pass. A cited "LLMs as assistants in software architecture design" that a quoted search could not find resolves this way to "Large Language Models as Assistants in Software Architecture Design" in IEEE Software, with a DOI. Classify title-first. Two independent questions, asked in order -- do not collapse them:

1. Does a publication bearing the cited title exist? Match on the title (allowing only trivial OCR/formatting/word-order differences, never a different topic). Finding a real paper by the same authors, or in the same venue, with a different title does not answer yes -- that is a different work, and the cited one still does not exist.
2. Only if step 1 is yes: are the metadata fields (authors, venue, year, volume/pages, DOI) consistent with that real publication?

Match the title on meaning, not exact string, in two steps:

• Formatting-only differences are the same title (title_match=yes, normally real-*): a present or absent subtitle or volume designation, spacing, hyphenation (including line-break hyphens such as distribu-tional), capitalization, punctuation, & vs and, diacritics, ligatures, other OCR/extraction artifacts, and British/American spelling. An abbreviation and its expansion of the same term are the same title too -- LLMs for Large Language Models, ML for machine learning, an expanded or contracted venue acronym. These name the same thing, so they do not change what the work is about; do not treat them as the wrong-content-word case below.
• A wrong, missing, or added content word (one that changes what the work is about -- PLS where the real paper says cluster analysis) still counts as title_match=yes only if an independent identifier pins the citation to one specific real publication: a DOI that resolves to it, or an exact author+venue+year(+pages) match. Then record that publication as matched_title and use partial-match -- the wrong title is the citation error. If only a loose resemblance links it to a real work -- a shared title template, the same topic, or a single shared author, with no resolving identifier -- the cited title names no real publication: title_match=no (likely-hallucinated, or unclear if you cannot tell). The rescue covers one or two slipped words, never a wholly different title: when the cited and the real title share little beyond a template, the identifier is not pinning a garbled title to its work -- it is a real record stapled to an invented title, and that is title_match=no.

Then assign the category:

• real-published / real-grey-literature / real-preprint-or-unpublished (low): a publication with the cited title exists and the metadata matches.
• partial-match (citation error, medium): a publication with the cited title exists, but one or two metadata fields are wrong -- wrong year, an off-by-one DOI digit, an abbreviated or swapped venue, a misspelled/missing co-author, a wrong first name. State the matched publication's actual title in the finding to prove the title matched. Regular human mistakes look like this: a real, locatable work with a slipped field. They do not invent titles.
• likely-hallucinated (high): no publication bears the cited title after a thorough escalating search. A real author (or real author group) or a real venue attached to a non-existent title is the canonical case -- do not rescue it to partial-match because the authors/venue happen to match some other real paper. State explicitly in the finding that no work bears the cited title.
• unclear (leave for a human): you cannot establish whether a work with the title exists (e.g. an obscure/predatory venue, an embargoed artifact). A valid verdict -- use it rather than guessing either way.

Independent fabrication signals (note each that applies in the finding): (T) no publication has the cited title -- the decisive one; (A) the author set/order never co-published, or initials-only generic authors; (V) an impossible or non-existent venue/year/volume (e.g. a proceedings entry and page range that do not exist, a defunct journal); (D) a DOI that 404s or resolves to an unrelated paper, or a placeholder arXiv id such as 2310.XXXX. A non-existent title (T) is itself a fabrication signature and grounds to flag the paper for desk-rejection -- even when the authors and venue are otherwise real (a real author group on an invented title is the hardest case to catch); A, V, and D strengthen the case but are not required. Do not push borderline cases to real-* to make the report look clean.

Record each verdict (it persists immediately and is resumable) with the structured signals that back the category:

"$RUN" triage record <paper_id> <number> <category> "<finding>" \
  --signals '{"title_match":"no","authors_match":"yes","venue_match":"no","doi_status":"none"}' \
  --out <outdir>

--signals is a JSON object: title_match (yes|no|unsure|na), matched_title (the real publication's title), authors_match/venue_match (yes|no|partial|unsure|na), and doi_status (resolves|404|mismatch|none|unsure). record enforces the title-first rule: a partial-match requires title_match=yes (plus a matched_title) or na, and likely-hallucinated requires title_match=no -- so a fabricated title cannot be filed as a mere citation error, nor vice versa. Use unclear when title_match is unsure; unclear cannot carry title_match=no, which asserts the likely-hallucinated finding while hedging the category. record rejects a <paper_id> <number> that matches no audited reference (the verdict could never appear in a report). Signals are optional for real-*. report then prints each flag's matched title and signal summary and lists papers whose likely-hallucinated references have a non-existent title under a Desk-reject candidates heading.

Each verdict is fingerprinted against the reference text it was recorded on. If a re-audit changes that text (author-year numbering is extraction-order, so numbers can shift), report marks the verdict stale and the reference pending -- it never reattaches the old category to the new text -- and worklist --pending resurfaces the reference. Re-triage it; do not trust the old category.

Then assemble the reports:

"$RUN" triage report --out <outdir>

• <outdir>/reports/reference-check-<paper_id>.md: per paper, including an Indistinguishable entries section (see below).

• <outdir>/reports/potential-hallucinations.md: corpus rollup for human review, led by a per-paper severity table and a Desk-reject candidates section (references whose cited title matches no real publication, compounded by a fabricated author constellation, venue, or DOI).

• <outdir>/reports/verify-<paper_id>.md: for each paper with flags, a manual-verification checklist (per-reference verdict line plus one-click Scholar/Google/DOI/arXiv links).

Both reports also list repeated bibliography entries: different citation keys carrying the same authors and title. Verification cannot find these -- each entry is a real reference that verifies on its own -- so only a whole-bibliography pass does. They come in two kinds, and the report separates them because the evidence differs:

• Duplicate -- every field matches: authors, title, venue, volume, pages. Two distinct articles cannot share a venue, volume, and article number, so this is one work entered more than once. State it plainly; there is nothing left to weigh.
• Conflicting -- authors and title match, venue/volume/pages do not. Genuinely open: one work whose entries disagree about where it appeared, or separate works (an extended version, a preprint) sharing a title. Only what each key is cited for in the text settles it, so report the disagreement and leave the call to the author.

Neither is a fabrication. Recommend only what the evidence carries: for a duplicate that is consolidation to a single key; for a conflict it is nothing yet, so report the disagreement and say what would settle it. Wrongly merging destroys distinct citations, while leaving them separate stays fixable -- so an unresolved conflict defaults to no action, not to a guess.

report auto-lints every file it writes with the bundled Markdown linter (lint_markdown.py), so the reports are valid GFM without a manual pass.

Notes

• Triage is the slow step that calls an LLM. Default to one paper at a time; confirm before the whole corpus.
• Most database-unverified references are real but uncovered (books, standards, tech reports, vendor docs, preprints). Verify them; do not assume fabrication. Genuine hallucinations cluster and show the signatures above. A "hallucinated" call against named authors is serious: flag it for review, do not accuse.