Check answer consistency
Skill ContextJet-ai/awesome-llm-observability/skills/check-answer-consistency
50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.
npx -y skills add ContextJet-ai/awesome-llm-observability --skill check-answer-consistencyAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
What its author says it does
Copied from the file, not written here
Use this to get a cheap, reference-free signal that an LLM answer might be made up, by sampling the same prompt a few times and measuring agreement. Trigger on "is this answer reliable", "flag low-confidence answers", "cheap hallucination check", "confidence score without a ground truth", "self-consistency check". Ships a runnable, tested scorer you can put inline or on sampled traffic.
The file declares its own license as CC0-1.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.
SKILL.md
2.6 KB, as published. Nobody here has run it
Check answer consistency
A model that knows the answer repeats it across re-samples; a hallucinating model wanders. This skill ships a small consistency scorer that turns that idea into a number, with no reference answer required.
Use the bundled script
scripts/self_consistency.py is pure Python, no install needed:
from self_consistency import consistency_score, is_likely_hallucination
answers = [call_model(prompt, temperature=0.7) for _ in range(5)]
score = consistency_score(answers) # 1.0 = all agree, ~0 = all differ
if is_likely_hallucination(answers, threshold=0.5):
route_to_review()
Run it directly to see confident vs unsure examples: python scripts/self_consistency.py.
How to apply it
- Sample with temperature > 0 (e.g. 5 samples). Consistency methods need diversity; a single deterministic answer tells you nothing.
- Score, then act: flag/block/route-to-review when the score is below your threshold, or attach it to the trace as a reliability signal.
- Reserve it for high-stakes answers if latency matters, sampling N times multiplies cost and latency by ~N.
This is a lightweight SelfCheckGPT-style check (normalized-string agreement). For meaning-level robustness (wording differs but the fact is the same), move to entailment clustering / semantic entropy, see the detect-hallucinations skill.
Validation
Run the tests: pytest skills/check-answer-consistency/tests/. They confirm identical answers score 1.0 (ignoring case/punctuation/whitespace), all-different answers score low and flag, a 3-of-4 majority scores 0.75 and does not flag, and that fewer than two answers returns 0.0.
Grounding
Consistency-based hallucination detection: SelfCheckGPT, Manakul et al. 2023 (arXiv:2303.08896). Meaning-level variant: semantic entropy, Farquhar et al., Nature 2024.
Anti-patterns
- Running it at temperature 0 (no diversity, the score is meaningless).
- Treating a high consistency score as proof of correctness (a model can be confidently and consistently wrong).
- Sampling N times inline on every request when latency/cost matter (sample offline or only on risky answers).