Failure clustering
Bratan is a self-improving Retrieval-Augmented Generation framework built on an adversarial three-agent loop
npx -y skills add AllanWessels/Bratan --skill failure-clusteringAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Use this skill when you need to interpret /reports/latest.json to find the root cause behind multiple failing cases. Reach for it before making changes to the pipeline, so you fix one root cause at a time instead of whacking moles. The output is "here is the cluster, here is the hypothesis, here is the suggested fix layer".
SKILL.md
3.6 KB, as published. Nobody here has run it
Failure Clustering
When to use
Before any blue team change, read this skill to interpret
/reports/latest.json correctly. Most failing cases are not independent
problems; they cluster around a small number of root causes.
The diagnostic matrix
The judge reports both retrieval_recall@5 and faithfulness per case.
Plot the failing cases in your head:
| Low retrieval | High retrieval | |
|---|---|---|
| High faithfulness | Honest "I don't know" — index gap or ingestion gap | Working ✓ (rare in failing cases) |
| Low faithfulness | Both broken — fix retrieval first | Model ignoring context — prompt issue |
Pick which quadrant has the most cases. That tells you which layer to fix.
The four failure modes within those quadrants
-
Answer missing from any chunk. Diagnose by manually searching
/corpus/for the answer. If you can't find it either, the knowledge base has a real gap; nothing pipeline-side will help. If you can find it but the index doesn't have it, that's an ingestion problem (parser, OCR, filter, freshness). -
Right chunk exists but wasn't retrieved. Diagnose by running the failing question with a high
k(50+) and checking whether the known-good chunk appears anywhere in the results. If absent: the embedding model is failing on this query type (try HyDE, multi-query, hybrid). If present but buried: add or upgrade the reranker, or increase initial k. If only one lane found it: check hybrid merge weights. -
Chunk retrieved but model ignored it. Symptoms: the model cites a wrong chunk, or generates from prior knowledge despite having context. Fixes: tighten the prompt (
answer using ONLY the context), reorder chunks so the most relevant is last, require chunk-numbered citations. -
Invented citation. The model cites chunk [N] but chunk [N] doesn't actually contain the claim. Fix: add a post-generation citation verification pass (separate LLM call) that checks each citation. Expensive but unavoidable for high-stakes outputs.
How to write a hypothesis
A good hypothesis for the blue team looks like:
"8 failing cases cluster in
multi_hopcategory. All 8 have low retrieval recall (avg 0.2). Manual inspection shows the question needs information from two corpus passages, but the retriever returns one and ignores the second. Hypothesis: a query decomposition step that splits multi-hop questions into independent sub-queries would lift recall on this cluster without affecting single-hop cases."
A bad hypothesis looks like:
"Lots of cases are failing, let's try a bigger model."
The first is testable, scoped, and predicts a specific outcome. The second is a guess that, even if it works, teaches you nothing.
How to pick the cluster to attack
When multiple clusters exist, prefer (in order):
- The largest cluster — fixing it moves the score most
- The cluster you have the strongest hypothesis about — even if smaller, the iteration loop completes faster
- Clusters at the retrieval layer before clusters at the generation layer — retrieval is upstream and the cheaper layer to fix
If two clusters tie, pick the retrieval one.