agentsclimarketplace

Reference checking

Skill snap-stanford/snap-skills/skills/reference-checking

Use this skill when the user asks to check, verify, or audit references and citations in a scientific manuscript, grant, thesis, or supplementary note. Triggers on "check references", "verify citations", "audit my bib", "are these citations real", "do the citations support the claims", "reference check this paper", etc.From its SKILL.md

Install
npx -y skills add snap-stanford/snap-skills --skill reference-checking

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

17.9 KB, as published. Nobody here has run it

Reference Checking

Verify references and supported claims in scientific manuscripts. Default operation only audits citations already inserted by the author. Optional second mode (full-claim audit) additionally extracts unsupported claims and searches for sources.

Design principle: deterministic, three-pass. Extract first, verify against source code second, verify against rendered output third. Never interleave.

Setup (one-time, user-side)

This skill drives several other skills for the web-based database lookups. Before it can run end-to-end, the user should have available:

  • citation-management — DOI / PMID / arXiv → CrossRef metadata.
  • openalex-database — title-based lookup and full author list for chimeric detection.
  • pubmed-database — biomedical references via PMID / PMC.
  • research-lookup — Mode B only, for finding sources of unsourced claims (Perplexity Sonar via OpenRouter).

If any of these are missing, fall back to direct HTTP queries against the respective public APIs (CrossRef, OpenAlex, NCBI E-utilities) and note the degradation in the report.

DBLP (https://dblp.org/search/publ/api) and arXiv (http://export.arxiv.org/api/query) are always queried by direct HTTP — there are no wrapper skills for them. Throttle DBLP to ≤1 req/3s; it resets the connection under heavier load.

When to use

  • Reviewing a manuscript, grant, thesis, or supplementary note before submission.
  • Auditing an AI-assisted draft for fabricated or chimeric references (real title, wrong authors).
  • Checking that each in-text citation actually supports the sentence it is attached to.
  • Cross-checking statistics, comparative claims, and quotes against their cited source.

Two modes

Mode A — Indicated-citation audit (DEFAULT)

Scope: only sentences that carry an in-text citation.

For each citation:

  1. Verify the cited paper exists.
  2. Verify author/year/title match across databases (catch chimeric and modified references).
  3. Verify that the cited paper actually supports the sentence it backs.

External knowledge or unsourced claims are not extracted in this mode.

Mode B — Full-claim audit (opt-in)

Trigger phrases: "full check", "extract all claims", "find sources for everything", "verify uncited claims too".

Adds Mode A's pipeline plus:

  1. Extract every factual claim using the rules below.
  2. For unsourced claims, search for an authoritative source via research-lookup.
TypePatternExample
StatisticNumber with unit/context"92.3% accuracy", "$4.7B market"
ComparativeX is [comparative] than Y"3× faster than baseline"
TemporalTime-bound assertion"In 2024, adoption reached..."
AttributionClaim tied to source"According to WHO...", "Smith et al. found..."
CausalX causes/leads to/results in Y"This reduces latency by..."
ExistenceAsserts something exists/is true"There are 500M users", "supports zero-shot..."
RankingPosition claims"largest", "first", "top 3"
QuoteDirect quotationAny quoted text attributed to a source

Do NOT extract: definitions, opinions, hypotheticals, questions, methodology descriptions, acknowledgments, unsourced future predictions.

Source-authority hierarchy when picking a source: primary > government/institutional > peer-reviewed > named industry reports > reputable news citing primary > secondary compilations. Use unverified when nothing is found — absence of evidence ≠ evidence of absence.

Inputs: source AND exported

Always check both the source files (.md, .tex, .org, .docx, ...) AND the exported version (usually the rendered PDF). They drive two separate verification passes:

  • Source drives Pass 2 — citation keys / BibTeX bindings give the (citation, claim) tuples and the metadata to verify.
  • Exported PDF drives Pass 3 — only the rendered output reveals what the reader actually sees.

Pass 1 — Extraction (no verification)

Collect every (citation, supported-sentence) tuple from the manuscript. Mode B additionally extracts unsourced claims (see Mode B table).

[E01] | "Vaswani et al. 2017"        | "The transformer architecture introduced multi-head self-attention." | p.3, §2.1
[E02] | "Smith & Lee, 2023, Table 2" | "Survival improved 38% over standard of care."                       | p.7, §3.2
[E03] | "Chen et al. 2024"           | "Performance reached 96.555% on ImageNet."                           | p.4, Fig.2 caption

Present Pass 1 output to the user before proceeding to Pass 2.

Pass 2 — Source code verification (no re-extraction)

Tooling

Web-only. Do not trust the local .bib or local PDFs as the source of truth — verify everything against the live databases. (The point of the audit is to catch errors the local library may already encode.)

SourceUse for
citation-managementDOI / PMID / arXiv → CrossRef metadata. scripts/extract_metadata.py is the workhorse for the existence check.
openalex-databaseTitle-based lookup when no DOI; full author list (chimeric detection); open-access PDF/abstract URL via open_access.oa_url.
pubmed-databaseBiomedical refs: PMID → abstract; resolve to PMC ID for open-access full text.
DBLP (direct HTTP)ML / CS conference papers (ICML, ICLR, NeurIPS, AAAI, CVPR, …). PMLR proceedings and OpenReview-hosted ICLR papers are not consistently indexed in CrossRef or OpenAlex; DBLP is the canonical source. Query: https://dblp.org/search/publ/api?q=<title>&format=json. Throttle to ≤1 req/3s — DBLP resets the connection under heavier load.
arXiv API (direct)Verify arXiv DOIs (10.48550/arXiv.*) and fetch preprint abstracts. CrossRef does not always carry arXiv records. Query: http://export.arxiv.org/api/query?id_list=<id>.
Zenodo API (direct)Resolve 10.5281/zenodo.* DOIs (CrossRef does not handle Zenodo). Query: https://zenodo.org/api/records/<id>.
research-lookupMode B only — find a source for an unsourced claim (Perplexity Sonar via OpenRouter).

Citation existence check (Mode A & B, citation claims)

For each citation:

  1. Read the entry as it appears in the manuscript / .bib and extract the claimed DOI / title / authors. Treat this as untrusted input.

  2. If DOI claimed: resolve via citation-management (CrossRef). Confirm the resolved title and author list match what the manuscript claims.

  3. If no DOI: openalex-database title search → confirm authors match. Cross-check with pubmed-database for biomedical refs.

  4. Verified iff DOI resolves with matching title and ≥1 overlapping author, OR title found in OpenAlex with ≥1 overlapping author and year match (±1).

  5. Preprint supersession check. If the resolved record is an arXiv / bioRxiv / medRxiv preprint, query three sources by title + first-author surname (year window: preprint year + [0, 3]) for a peer-reviewed counterpart, in this order:

    1. DBLP (https://dblp.org/search/publ/api) — required for any arXiv preprint in ML / CS. ICML and NeurIPS proceedings (PMLR / proceedings.mlr.press) and ICLR papers (OpenReview) do not register CrossRef DOIs and are unreliable in OpenAlex. DBLP is the canonical index for ML/CS conferences. Skipping this step is the most common reason for false-negative supersession checks.
    2. CrossRef (citation-management) — for journal articles.
    3. OpenAlex — secondary cross-check; covers some conference proceedings via Springer LNCS / IEEE.

    For biomedical preprints (bioRxiv / medRxiv), also query PubMed to catch the published Nature / Cell / Science version. If a published version exists in any source, set status = preprint_superseded and recommend citing it. Google Scholar (and many BibTeX exports) default to the preprint even after publication.

Output statuses for the existence check:

StatusMeaning
verifiedCitation exists, metadata consistent.
chimericTitle matches a real paper, authors disjoint. Highest-risk hallucination.
modifiedReal paper, but year / venue / authors subtly altered.
not_foundNo hit across all databases. Likely fabricated.
preprint_supersededCited as arXiv/bioRxiv/medRxiv but a peer-reviewed version exists. Recommend citing the published version.
suspiciousFound in 1 source only, or metadata inconsistent. Manual check.

Red-flag heuristics: invalid DOI prefix (not 10.xxxx/), generic survey title patterns, future publication year, single-token author names, missing year.

Bibliography consistency (Mode A & B)

One pass over the full .bib / reference list, independent of per-citation checks. bibtex export usually handles this, but residual errors are common.

  • Author-name format. The same author must use one form throughout (all Smith, J. or all Smith, Joe, not mixed). Detect by grouping entries on last-name+first-initial; flag groups with mixed first-name forms.
  • Venue format. The same venue must use one form throughout (all ICML or all International Conference on Machine Learning, not mixed). Detect by grouping CrossRef container-title per ISSN / conference series.
  • Pages field. Either every journal/conference entry carries pages = X--Y or none does — no partial coverage.

Report each finding as inconsistent_format with the affected entry keys and a recommended canonical form.

Claim-support check (Mode A & B)

For each (citation, supported-sentence) tuple:

  1. Fetch the cited paper from the web. In priority order: open-access PDF via openalex-database (open_access.oa_url), PMC full text (pubmed-database → PMC ID), arXiv PDF for preprints, publisher abstract from CrossRef (citation-management). If only the abstract is reachable, use the abstract and note "abstract-only" in the finding.
  2. Locate the supporting passage. For numerical claims, locate the table/figure/sentence with the exact number.
  3. Classify:
START
├─ Does the source contain the claimed topic?
│   ├─ NO  → Misquoted (source exists but does not discuss this)
│   └─ YES → Does the source support the exact claim?
│             ├─ Exact / paraphrase → Verified
│             ├─ Partial / missing context → Misleading
│             └─ Contradicts → Hallucination

For numerical claims, direction-of-effect and order-of-magnitude always matter; rounding within the source's stated significant figures is fine.

Mode B only — sourcing unsourced claims

Use research-lookup for an initial web-grounded search, then openalex-database / pubmed-database to anchor on a primary source. Apply the authority hierarchy listed under Mode B above.

Pass 3 — Rendering verification

Inspect the exported/rendered PDF (or whatever the reader actually receives). Pass 3 catches failures invisible in source — things the typesetter introduced or silently dropped. Independent of Pass 2: even a perfectly-sourced manuscript can render broken.

You may need to export the project yourself (pdflatex + bibtex, pandoc, org-export, ...). If the build is non-trivial, ask the user to provide the rendered PDF rather than guessing the toolchain.

Scan for:

  • Unresolved citation keys — [?], ??, \cite{} / \citep{} echoed verbatim, (citation?), [CITATION] placeholders. Each one is a missing .bib entry or a typo in the cite key.
  • Broken cross-references — Figure ??, Table ??, Section ??, [link] with no target.
  • Duplicate or mis-numbered figures/tables — if "Table 2" appears twice, every claim citing "Table 2" is ambiguous. Renumber and re-check Pass 2 claims that point to figures/tables.
  • Silently dropped citations — a citation present in source but absent from the rendered bibliography (e.g. cite key typo, missing .bib entry, \nocite mishap).
  • Bibliography order / style consistency — if the journal requires a specific style (numeric vs. author-year, alphabetical vs. order-of-citation), confirm the output matches.

Report each finding as rendering_error with the page/section in the PDF, the offending fragment, and the suggested fix in source.

Output format

Always lead with the summary block.

## Reference Check Report

**Mode:** Indicated-citation audit | Full-claim audit
**Document:** <filename>
**BibTeX:** <path to .bib if available>

### Summary
| Metric                       | Count |
|------------------------------|-------|
| Citations checked            | N     |
| Verified                     | N     |
| Chimeric                     | N     |
| Modified                     | N     |
| Not found                    | N     |
| Preprint superseded          | N     |
| Suspicious                   | N     |
| Inconsistent format          | N     |
| Misquoted                    | N     |
| Hallucinations               | N     |
| Misleading                   | N     |
| Rendering errors (Pass 3)    | N     |
| Unverified (Mode B)          | N     |

**Verdict:** PASS / FAIL

Then per-status tables. For each finding give: citation key, manuscript location, the claim, the source value, the discrepancy, and a recommended fix.

Critical rules

  1. Three passes, in order: extract (Pass 1) → source-code verify (Pass 2) → rendering verify (Pass 3). Never interleave.
  2. Pass 2 needs source files; Pass 3 needs the exported PDF. Skipping Pass 3 misses unresolved keys, broken cross-refs, and silently-dropped citations.
  3. Default to Mode A. Enter Mode B only on explicit request.
  4. Every citation gets an existence check AND a support check. A real paper can still be misquoted.
  5. When uncertain, prefer the conservative status (suspicious over verified, unverified over verified).
  6. Output a recommended fix for every non-verified item.
  7. Supersession check requires DBLP for ML/CS preprints. CrossRef and OpenAlex miss PMLR (proceedings.mlr.press) and OpenReview (ICLR). Treat an "only on arXiv" finding as unconfirmed until DBLP has been queried.

Indexing gaps to remember

These coverage gaps in the major catalogs have produced false negatives in past audits:

Venue / DOI prefixIndexed byCaveats
PMLR (ICML, AISTATS, UAI, …)DBLPproceedings.mlr.press papers do not have CrossRef DOIs; OpenAlex misses many. Per-paper YAML at github.com/mlresearch/v<vol>/_posts/ has canonical metadata.
OpenReview (ICLR)DBLPICLR oral / spotlight / poster papers are hosted only on OpenReview; no CrossRef DOI. DBLP has the venue / year / authors.
arXiv DOIs (10.48550/arXiv.*)arXiv API directlyCrossRef does not always carry arXiv records. Verify via http://export.arxiv.org/api/query?id_list=<id>.
Zenodo DOIs (10.5281/zenodo.*)Zenodo APICrossRef does not handle Zenodo. Verify via https://zenodo.org/api/records/<id>.
openRxiv DOIs (10.64898/*)CrossRefNew bioRxiv prefix (assigned 2026). CrossRef records exist as type: posted-content.
Springer LNCS (MICCAI, ECCV, …)CrossRef + OpenAlexSome publisher / open-access bibtex exports mangle author lists when their separator collides with bibtex's and separator (e.g. AND between names). Always re-resolve via DOI.
Recently-published bio preprintsPubMedA bioRxiv preprint can still appear as "preprint only" in CrossRef / OpenAlex for months after publication. PubMed picks up the Nature / Cell / Science version sooner.

Reference: source skills

Keep looking

Skills are one crate of 610,045. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.