Ai rag patterns

Overview

Production RAG in 2026 is no longer "embed, retrieve top-5, generate." Three measurable shifts changed the playbook: hybrid retrieval beating either BM25 or dense alone, reranking moving from optional to default, and Anthropic's Contextual Retrieval (September 2024) collapsing retrieval-failure rates dramatically. This skill encodes the patterns that actually move the metrics, with cited numbers and the failure modes they address.

Use this skill when designing a new RAG pipeline, diagnosing why an existing one underperforms, or evaluating retrieval quality before generation. Skip it for simple in-context document Q&A on a small static corpus where long-context generation (1M Sonnet / 2M Gemini) is the cheaper answer — see §6.

Mental model

A RAG pipeline is a chain of probabilistic filters: chunking decides what units the system can return, indexing decides what's searchable, retrieval ranks candidates, reranking re-orders them, and generation consumes the result. The output is bounded by the weakest link. The seven failure modes (Barnett et al., arXiv:2401.05856) map cleanly: missing content, top-ranked-missed, not-in-context, not-extracted, wrong format, incorrect specificity, incomplete. Diagnose at the layer that owns the failure; tuning generation when retrieval is the bottleneck is wasted effort.

Two compounding levers move 2026 production RAG most: contextual chunking (adding LLM-generated chunk context before indexing) and hybrid retrieval with reranking. The numbers in §1–§3 below are from Anthropic's Contextual Retrieval write-up and the Elastic / TOIS hybrid-retrieval studies.

Instructions

1. Chunking strategies

Strategy	When to use	Source
Fixed-size / recursive (LangChain RecursiveCharacterTextSplitter)	Baseline; works for prose with stable structure	Practitioner default; no canonical paper
Semantic (LlamaIndex SemanticSplitterNodeParser)	Mixed-topic documents where boundaries matter	Practitioner pattern; Greg Kamradt "5 levels of chunking"
Late Chunking (Jina AI, Sep 2024)	Long documents with rich context; uses 8192-token embedding model, mean-pools after the transformer at chunk boundaries	arXiv:2409.04701
Anthropic Contextual Retrieval	Production RAG where retrieval quality matters and cost permits ~$1.02 per million doc tokens to pre-process	Anthropic blog, Sep 2024
Parent-document / small-to-big	Index small for precision, return parent for context	LangChain ParentDocumentRetriever, LlamaIndex AutoMergingRetriever

Anthropic Contextual Retrieval numbers (from a baseline top-20 retrieval-failure rate of 5.7%):

• Contextual Embeddings alone: −35% failure-rate reduction (5.7 → 3.7%)
• Contextual Embeddings + Contextual BM25: −49% (5.7 → 2.9%)
• Contextual Embeddings + Contextual BM25 + Reranking: −67% (5.7 → 1.9%)

The headline 67% requires reranking on top — it is not the chunking technique alone.

2. Retrieval — BM25, dense, hybrid

Method	Strength	Weakness
BM25 (lexical)	Exact term match, no embedding cost, captures rare terms	Misses paraphrase, synonyms, semantic relationships
Dense vector (any modern embedder)	Captures paraphrase and semantic similarity	Weak on rare terms, exact codes, proper nouns
Hybrid (BM25 + dense, reciprocal rank fusion)	Combines both signals; outperforms either alone on most production corpora	More moving parts, two indices to maintain
ColBERT / late interaction	Token-level matching; strong for queries needing exact-phrase grounding	More expensive than dense

Hybrid numbers — Elastic Search Labs reports reciprocal rank fusion at +18% NDCG@10 over BM25 alone and +1.4% over ELSER alone on the BEIR benchmark (Elastic — hybrid retrieval). ACM TOIS (Bruch et al., 2023, doi:10.1145/3596512) shows convex combination at α≈0.5 reaches Recall@5 = 0.726 vs RRF k=60 at 0.695 — tuned weighting beats RRF given ~40 labeled query-result pairs to fit α.

Reciprocal Rank Fusion — Cormack, Clarke, Büttcher, SIGIR 2009 (canonical paper). The formula combines ranks across retrievers: RRF_score(d) = Σ 1 / (k + rank_i(d)), default k=60.

ColBERTv2 — Santhanam et al., NAACL 2022. jina-colbert-v2 reports +6.5% nDCG@10 over ColBERTv2; jina-reranker-v3 reports 61.85 nDCG@10 on BEIR, +4.79% over v2 (arXiv:2509.25085).

3. Query rewriting / expansion

Technique	What it does	Source
HyDE	LLM generates a hypothetical answer document, embeds that for retrieval	arXiv:2212.10496, Gao et al. Dec 2022
Multi-query	LLM rewrites the query into N variations; retrieve for each; merge	LangChain MultiQueryRetriever
Decomposition / least-to-most	Break a complex question into sub-questions; retrieve and answer each	arXiv:2205.10625, Zhou et al. May 2022
Query routing	LLM picks which retriever / index / corpus to use	LlamaIndex RouterQueryEngine; agentic-RAG survey arXiv:2501.09136

Query rewriting is most impactful when user queries are short or ambiguous (chat agents, voice agents). For well-formed search queries, the lift is marginal.

4. Reranking

Reranking is the second filter: take 50–100 candidates from first-stage retrieval, score with a more expensive model, return top 3–10.

Reranker	Source
Cohere Rerank 3.5 / 4	Cohere Rerank docs — pricing $2 per 1K searches (1 query + up to 100 docs)
Cross-encoders (sentence-transformers/ms-marco-MiniLM-L-6-v2)	Reimers & Gurevych, EMNLP 2019
BGE-Reranker (bge-reranker-v2-m3)	Xiao et al., BAAI, arXiv:2309.07597
ColBERT as reranker	Late-interaction over first-stage top-k

Anthropic's Contextual Retrieval write-up reports reranking adds ~18% relative failure-rate reduction on top of hybrid Contextual Retrieval. No single canonical "typical lift" exists — it depends on first-stage noise level. Vendor blogs cite "20–35% RAG accuracy lift" and "NDCG@5 +18 points" — these are vendor-reported, not peer-reviewed, so treat as ballpark.

5. Evaluation

Three frameworks dominate; use one as primary, cross-check with another.

Framework	Metrics	Source
RAGAS	Faithfulness, answer relevancy, context precision, context recall, context entity recall	Es et al., EACL 2024, arXiv:2309.15217; RAGAS metrics docs
TruLens RAG Triad	Context relevance, groundedness, answer relevance	trulens.org RAG Triad
DeepEval	G-Eval, HallucinationMetric, ContextualPrecision/Recall/Relevancy	DeepEval LLM evals; G-Eval based on Liu et al., EMNLP 2023, arXiv:2303.16634

RAGAS scores — original paper reports human-agreement 95% / 78% / 70% on faithfulness / answer relevance / context relevance. Production threshold: >0.8 on faithfulness and context precision is strong; depends on domain.

Recommended flow (practitioner consensus, not a single citation):

1. Offline golden-set: tune retriever with RAGAS context precision/recall.
2. Tune generator: faithfulness + answer relevancy.
3. Online tracing: Phoenix or LangSmith with LLM-as-judge sampling on production traffic.

Cross-check with the eval-design skill in the same domain for statistical-significance methodology and judge selection.

6. Long context vs RAG (the 2025–2026 question)

Anthropic Sonnet/Opus 1M context (Aug 2025); Gemini 1.5/2.0 Pro 2M context. The question: does long-context generation replace RAG?

Databricks "Long Context RAG Performance of LLMs" (blog) finds long-context QA beats chunk-RAG on stable, single-doc tasks but Gemini 1.5 underperforms o1 / GPT-4o / Sonnet 3.5 on DocsQA and FinanceBench. The RULER benchmark (Hsieh et al., arXiv:2404.06654) shows advertised context degrades early — effective context is shorter than the stated window.

Practitioner consensus (no single citation): use hybrid — retrieve a wider top-50 to top-100K-token shortlist, then long-context generate over the shortlist. Pure long-context loses on cost, citation auditability, and corpus freshness; pure short-RAG loses on multi-step reasoning across distant context. Pick by corpus size, refresh rate, citation requirements, and budget.

7. Multi-modal RAG

Technique	Use case	Source
ColPali	Document images, PDFs with charts/tables, slides	Faysse et al., arXiv:2407.01449, Jun 2024; introduces ViDoRe benchmark
ColQwen2, ColSmol	Smaller / multi-language variants	illuin-tech/colpali

ColPali beats text-extraction-then-embed pipelines on document-image benchmarks. Use it when the corpus is visually rich (scans, slides, layouts that don't OCR cleanly).

8. Agentic RAG

Singh et al., "Agentic Retrieval-Augmented Generation: A Survey" (arXiv:2501.09136, Jan 2025). Patterns: reflection (re-query if confidence low), planning (decompose then retrieve per sub-question), tool use (route between corpora and tools), multi-agent (specialist sub-agents per corpus).

Use agentic RAG when queries are heterogeneous (some need code search, some need docs, some need web), or when single-pass retrieval has known failure modes the agent can detect and recover from. Cost: 2–4× a single-pass RAG; justify with evidence the simpler design failed.

9. The seven failure modes (Barnett et al.)

arXiv:2401.05856, Jan 2024 — canonical failure taxonomy:

1. Missing content — answer isn't in the corpus
2. Top-ranked missed — relevant chunk exists but wasn't retrieved
3. Not in context — retrieved but didn't make the generator's prompt window
4. Not extracted — in the prompt but generator missed it (often context pollution)
5. Wrong format — extracted correctly but presented incorrectly
6. Incorrect specificity — too general or too specific for the question
7. Incomplete — partial answer

Diagnose at the failing layer; map fix to the right intervention (corpus curation, retrieval tuning, reranking, chunk-size, generator prompting).

10. Production patterns checklist

• Chunking strategy chosen with rationale tied to corpus structure
• Retrieval is hybrid (BM25 + dense + RRF or convex combination)
• Reranking on top-50 to top-100 first-stage results
• Query rewriting where queries are short / ambiguous
• Eval suite using RAGAS or TruLens RAG Triad with a versioned golden set
• Production tracing via Phoenix / LangSmith / Langfuse
• Drift monitoring on retrieval (embedding centroid distance over time)
• Long-context vs RAG decision documented with rationale
• Multi-modal corpus uses ColPali-family when documents are visually rich
• Agentic patterns only when single-pass has documented failure modes

Composition / References

Within-domain pairings:

• Pairs with the sibling AI skill on agent design (RAG often lives inside an agent loop with retrieval as a tool).
• Pairs with the sibling AI skill on evals (every RAG decision is gated by RAGAS / RAG Triad measurement).
• Pairs with the sibling AI skill on cost optimization (embedding + rerank + vector DB costs are the RAG-specific FinOps lever).

Primary sources:

• Anthropic — Contextual Retrieval
• Late Chunking, arXiv:2409.04701
• RAGAS, arXiv:2309.15217
• TruLens RAG Triad
• Seven Failure Points of RAG, arXiv:2401.05856
• Cormack et al. RRF, SIGIR 2009
• HyDE, arXiv:2212.10496
• Elastic Search Labs — hybrid retrieval
• Hybrid Fusion ACM TOIS
• ColPali, arXiv:2407.01449
• Agentic RAG Survey, arXiv:2501.09136
• RULER, arXiv:2404.06654
• Databricks long-context RAG performance
• Cohere Rerank docs
• DeepEval LLM evals