Case verification

case-verification

Deterministic citation verification for US legal writing. Built against the Sullivan & Cromwell Schedule A of errors from In re Prince Global Holdings Limited (Bankr. S.D.N.Y. Apr. 18, 2026) as the benchmark: fabricated cases, pincite drift, and fabricated quotations — the exact failure modes of current AI legal drafting.

What this skill does

Given a brief, motion, memo, opinion, or any legal document, the skill runs a deterministic pipeline that:

1. Extracts every case citation with eyecite (Free Law Project parser)
2. Resolves each citation against CourtListener (5M+ US opinions, free API)
3. Fetches the full opinion text where the citation resolves
4. Checks whether any quoted language associated with the citation actually appears in that opinion, with normalization for quote style, whitespace, ellipses, and bracketed editorial insertions
5. Where Westlaw-style star-pagination (*5, *10) is present in the opinion, checks that the pincite in the brief matches where the quoted language actually appears
6. Produces a structured evidence ledger + a human-readable report

Verification is fail-closed. A citation is marked verified only when the deterministic checks pass with evidence in hand. When the evidence isn't there, the output says unresolved or not_found, not "probably fine."

What this skill does NOT do

Be direct with the user about the scope:

• It does not decide whether a cited case actually supports the proposition for which it is cited. That requires legal judgment and remains with the human reviewer. The report retrieves and displays the relevant passage so the human can decide — but the machine does not pretend to clear substantive support.
• It does not check Shepard's / KeyCite treatment (overruled, abrogated, distinguished). Those citators are not freely available.
• It does not cover non-US law. BVI, Cayman, UK, Canadian, Australian, and other foreign citations are flagged unresolved with a note. That is the honest answer; say so.
• It does not verify statutes, regulations, legislative history, or secondary sources.

When the user asks about any of the above, explain the scope limit rather than guessing.

When to use this skill

Use this skill, and call the verify.py script, whenever the user:

• Uploads or pastes a brief, motion, memo, opinion, or legal draft and asks for any form of citation review, accuracy check, or hallucination audit
• Asks whether a specific case citation is real
• Asks whether a quote attributed to a case actually appears in the case
• Asks to audit opposing counsel's filing for errors (adversary mode)
• Asks to produce a pre-filing citation check (pre-flight mode)
• Uses language like "check the cites", "find hallucinations", "is this real", "did they make this up", "verify this quote", "review before filing", "is this case on point"

Do NOT answer these questions from memory. Always run the script. The entire point of the skill is that the model's confidence about whether a citation is real is not evidence.

Two modes

Pre-flight mode (default, --mode preflight): the user is verifying their own draft. Output lists each citation with its status, the specific error class if any, and a narrow suggested correction. Tone is that of a careful colleague pointing out what needs fixing.

Adversary mode (--mode adversary): the user has received the adversary's document and wants to identify unverifiable citations to use in a response, a letter to the court, or a 28(j) filing. Output leads with a flagged-items section that contains finding-style framing plus a draftable paragraph for each flagged citation. The draftable paragraph is written to be usable, with citations to the retrieved opinion text as the evidentiary basis.

Infer the mode from context when possible. If the user says "I'm writing this brief", it's pre-flight. If they say "opposing counsel filed this", it's adversary. When it's genuinely unclear, ask.

Workflow

Follow these steps in order.

1. Read the skill scripts

Before doing anything else, view scripts/verify.py and note the CLI flags it exposes. The skill's logic lives in the scripts — do not re-implement any of it in prose.

2. Prepare the input

The input is a brief as plain text or markdown. If the user uploads a PDF or Word file, convert it to plain text first (the document-reading skills handle this; ask the user to confirm the conversion preserved the citations). Save the text to a working file.

3. Check for an API token

CourtListener works unauthenticated but is rate-limited to 100 lookups per day. For any real-world brief the user should set COURTLISTENER_API_TOKEN (free token, sign up at courtlistener.com). If the env var is not set, tell the user once at the start of the session; do not repeat the warning every run.

4. Run the verifier

python3 scripts/verify.py <brief.txt> --mode <preflight|adversary> --format md --output report.md

Useful flags:

• --mode preflight (default) or --mode adversary
• --format md (default) or --format json or --format both
• --output <file> — write to file instead of stdout
• --offline — parse-only, no CourtListener calls. Use this for a dry-run when the network is unavailable or when the user wants to see what the tool will check without hitting the API.
• --token <token> — override the env var

5. Present the report

The script's own output is already the report. Do not rewrite it into prose or summarize it to the point of losing the per-citation detail — the user needs to see which specific paragraphs have which specific issues, with the retrieved evidence snippets and source URLs. That traceability is the whole point.

For long reports, it is fine to give the user a one-paragraph summary at the top of your response followed by the full report (or a link to the output file if it was written to disk).

6. Respond to follow-ups

Common follow-ups and how to handle them:

• "Can you re-check citation X?" — re-run verify.py with a small input that contains just that citation. Do not answer from memory.
• "What does this case actually say about Y?" — fetch the opinion text via the CourtListener URL that's already in the report, read it, and answer from the retrieved text with a direct source citation. This is legal research, not verification — be clear about the switch.
• "Write the response brief paragraph" — in adversary mode the report already contains draftable paragraphs. Use those as the basis, tighten them for the specific context, but do not invent additional case citations.

Output format

The script outputs per-citation findings with layered status — each citation is checked on multiple dimensions and each dimension is reported separately. The top-level badge is a roll-up, and deployers must not treat it as a substitute for the layer detail.

Layered status (shown in every finding)

Layer	What it checks
A. Authority resolution	Does the citation resolve in CourtListener?
B. Source retrieval	Can we retrieve opinion text?
C. Metadata consistency	Do year and case name match the brief?
D. Quote verification	Does quoted language appear in the opinion?
E. Pincite verification	Is the *N page consistent with where the quote appears?
F. Proposition support	ALWAYS not_reviewed — engine does not judge
G. Treatment / citator	ALWAYS not_reviewed — no KeyCite / Shepard's

Layers F and G being permanently not_reviewed is not a bug. The engine refuses to claim it has checked things it has not checked. See GOVERNANCE.md for the full honesty model.

Top-level status vocabulary

Status	Meaning
verified_quote_and_metadata	Resolved; quote verified; metadata consistent. Support NOT reviewed.
resolved_no_quote_support_not_checked	Resolved; no quote asserted. Support NOT reviewed.
pincite_error	Case exists, quote is in it, but at a different *N page
quote_not_found	Case exists, but the quoted language isn't in it
metadata_mismatch	Resolved, but year or case name disagrees
not_found	Did not resolve in CourtListener — potential hallucination
ambiguous	Multiple candidate cases match the citation
unresolved	Out of scope, or CourtListener couldn't be reached

Note that no top-level status says bare verified. Even in the best case, the label is verified_quote_and_metadata — a deliberate reminder that proposition support was not checked.

Each finding includes: the citation as used, canonical record (if found), the quoted language (if any), the quote-check result, the pincite-check result, a retrieved evidence snippet, a source URL, provider class (PUBLIC_FALLBACK for the default engine), and — in pre-flight mode — a narrow suggested correction.

Honesty rules

These are non-negotiable:

1. The engine never emits bare verified. Even the best-case badge is verified_quote_and_metadata, to keep the reader aware that proposition support was not checked.
2. Layers F and G are always not_reviewed. Proposition support and treatment are outside engine scope. The layered status table must show this explicitly on every finding.
3. If CourtListener is not reachable or the opinion text is not retrievable, no quote can be called verified. The layers show text_unavailable and the top-level status is unresolved.
4. Non-US citations or reporters CourtListener doesn't cover are flagged unresolved — out of scope with a note. Do not attempt to verify foreign citations from model knowledge.
5. Every finding discloses its provider class (PUBLIC_FALLBACK for the default CourtListener engine). A firm deployment overriding the provider must update this disclosure.
6. If the user asks whether a case is still good law, or whether a case actually supports a proposition, say directly that the engine does not check these things and point them to GOVERNANCE.md or to a commercial citator for treatment.

Files in this skill

case-verification/
├── SKILL.md                    this file
├── README.md                   GitHub-facing overview
├── GOVERNANCE.md               deployment policy for firms (status
│                               semantics, provider classes, materiality,
│                               filing-gate recommendations)
├── scripts/
│   ├── verify.py               main entry point
│   ├── parse.py                eyecite-based extraction of verification units
│   ├── cl_client.py            CourtListener API client (stdlib only)
│   └── text_utils.py           quote normalization + star-pagination
└── tests/
    ├── sample_brief.txt        S&C-style fixture with known errors
    └── test_pipeline.py        integration tests including adversarial cases

The only external dependency is eyecite (install with pip install eyecite). The CourtListener client uses Python stdlib only, no requests.