Check answer consistency
Skill ContextJet-ai/awesome-llm-observability/skills/check-answer-consistency
Use this to get a cheap, reference-free signal that an LLM answer might be made up, by sampling the same prompt a few times and measuring agreement. Trigger on "is this answer reliable", "flag low-confidence answers", "cheap hallucination check", "confidence score without a ground truth", "self-consistency check". Ships a runnable, tested scorer you can put inline or on sampled traffic.From its SKILL.md
npx -y skills add ContextJet-ai/awesome-llm-observability --skill check-answer-consistencyAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
What its file declares
Copied from the file, not written here
The file declares its own license as CC0-1.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.
SKILL.md
2.6 KB, 534 tokens by cl100k_base, as published. Nobody here has run it
Check answer consistency
A model that knows the answer repeats it across re-samples; a hallucinating model wanders. This skill ships a small consistency scorer that turns that idea into a number, with no reference answer required.
Use the bundled script
scripts/self_consistency.py is pure Python, no install needed:
from self_consistency import consistency_score, is_likely_hallucination
answers = [call_model(prompt, temperature=0.7) for _ in range(5)]
score = consistency_score(answers) # 1.0 = all agree, ~0 = all differ
if is_likely_hallucination(answers, threshold=0.5):
route_to_review()
Run it directly to see confident vs unsure examples: python scripts/self_consistency.py.
How to apply it
- Sample with temperature > 0 (e.g. 5 samples). Consistency methods need diversity; a single deterministic answer tells you nothing.
- Score, then act: flag/block/route-to-review when the score is below your threshold, or attach it to the trace as a reliability signal.
- Reserve it for high-stakes answers if latency matters, sampling N times multiplies cost and latency by ~N.
This is a lightweight SelfCheckGPT-style check (normalized-string agreement). For meaning-level robustness (wording differs but the fact is the same), move to entailment clustering / semantic entropy, see the detect-hallucinations skill.
Validation
Run the tests: pytest skills/check-answer-consistency/tests/. They confirm identical answers score 1.0 (ignoring case/punctuation/whitespace), all-different answers score low and flag, a 3-of-4 majority scores 0.75 and does not flag, and that fewer than two answers returns 0.0.
Grounding
Consistency-based hallucination detection: SelfCheckGPT, Manakul et al. 2023 (arXiv:2303.08896). Meaning-level variant: semantic entropy, Farquhar et al., Nature 2024.
Anti-patterns
- Running it at temperature 0 (no diversity, the score is meaningless).
- Treating a high consistency score as proof of correctness (a model can be confidently and consistently wrong).
- Sampling N times inline on every request when latency/cost matter (sample offline or only on risky answers).
What ships with it: 2 files
3.0 KB alongside SKILL.md, 2 of them executable
scripts/
- self_consistency.pyruns1.9 KB
tests/
- test_self_consistency.pyruns1.1 KB
Gives 0 of the 12 instructions most quality gates skills give in 534 tokens
Counted across 1,524 of the 2,830 authors here whose files we hold, read 2026-09-06
- Read full output and check exit codein 45 of 1524, across 40 files
- Verify output confirms the claimin 44 of 1524, across 39 files
- Identify the command that proves the claimin 43 of 1524, across 39 files
- Execute the full verification commandin 36 of 1524, across 30 files
- Produce a verification reportin 34 of 1524, across 18 files
- Review git diff changesin 30 of 1524, across 16 files
- Fix build failures immediatelyin 29 of 1524, across 9 files
- Group findings by severityin 28 of 1524
- State claim only with evidencein 27 of 1524, across 22 files
- Verify regression tests with red-green cyclein 26 of 1524, across 22 files
- Run the full test suitein 26 of 1524, across 25 files
- Run test suite with coveragein 25 of 1524, across 10 files
Said here and by no other author read
- Sample the prompt multiple times
- Use a temperature greater than zero
- Calculate the consistency score
- Flag answers below the confidence threshold
- Route low confidence answers to review
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.