Agent rag pipeline

Agent RAG Pipeline

RAG lets agents ingest user data and synthesize it with their global knowledge to give high-quality responses. But RAG is often over-engineered. Start simple, check quality, get complex only if needed.

The RAG Decision Tree

Before building a RAG pipeline, try simpler alternatives first:

1. Can you fit all the content in the context window?
   YES --> Full Context Loading (simplest, most reliable)
   NO  --> Go to step 2

2. Can you build tools that access your data directly?
   YES --> Agentic RAG (agent uses tools instead of searching docs)
   NO  --> Go to step 3

3. Build a RAG pipeline

RAG 101: The Pipeline

Documents --> Chunking --> Embedding --> Indexing --> [Vector DB]
                                                         |
User Query --> Embedding --> Querying --> Reranking --> Synthesis --> Response

1. Chunking

Break documents into smaller, manageable pieces for search.

Key decisions:

• Strategy: Recursive (default), character-based, token-aware, or format-specific (Markdown, HTML, JSON, LaTeX)
• Chunk size: 256-1024 tokens is typical. Smaller = more precise retrieval, larger = more context preserved
• Overlap window: 50-100 tokens of overlap between chunks prevents splitting a concept across two chunks

Best practice: Start with recursive chunking, 512 tokens, 50 overlap. Adjust based on retrieval quality.

2. Embedding

Transform text chunks into numerical vectors (arrays of ~1536 values) that capture semantic meaning.

Providers: OpenAI (text-embedding-3-small), Voyage, Cohere Key point: Use the same embedding model for indexing AND querying.

3. Vector Database Selection

Form Factor	Examples	Best For
Extension on existing DB	pgvector (Postgres)	Already using Postgres
Standalone open-source	Chroma	Local dev, prototyping
Hosted cloud service	Pinecone	New projects, nice UI
Cloud provider managed	Cloudflare Vectorize, DataStax Astra	Already on that cloud

Recommendation: Prevent infra sprawl.

• Already using Postgres? Use pgvector
• New project? Pinecone is a good default
• Your cloud provider has one? Use that

4. Indexing

Create an index optimized for similarity search:

• Specify dimension size (must match your embedding model)
• Choose similarity metric: cosine similarity (most common), euclidean, or dot product
• This is a one-time setup per collection

5. Querying

Convert the user's input to an embedding and find the most similar chunks:

• Cosine similarity compares vectors across ~1536 dimensions
• topK: How many results to return (start with 5-10)

Hybrid Queries with Metadata: Combine vector similarity with metadata filtering (dates, categories, tags) for more precise results.

6. Reranking

A post-processing step that reorders results using more sophisticated scoring:

• Considers semantic relevance, vector similarity, AND position bias
• More computationally expensive than initial search
• Run it over your topK results, not the entire database

7. Synthesis

Pass the retrieved chunks plus the user's query to an LLM to generate a response:

"Please answer the following question:
{query}

Based on this context: {relevantContext}
If the context lacks sufficient information, please state that explicitly."

Alternatives to RAG

Agentic RAG

Give the agent tools to access your data directly instead of searching documents:

Agent + Tools (market data API, calculator, portfolio analyzer)
  --> Agent reasons with tools to compute exact answers

Advantage: More precise than searching text chunks. Agent can compute exact answers. Disadvantage: Need to build and maintain the tools. Agent needs to know how to use them effectively.

Reasoning-Augmented Generation (ReAG)

Use LLMs to enrich text chunks before storing them:

• Generate 10x annotations at high temperature and check for consensus
• Send input through an LLM before retrieval to add semantic info
• Extract entities, relationships, and structured data
• Pre-processing is asynchronous, so speed doesn't matter

Full Context Loading

With models supporting 200K-2M token windows, just load everything:

Advantages: Simplest approach, most reliable, model sees all context at once Disadvantages: Cost (full window every call), size limits, potential for distraction

Decision Framework

Situation	Approach
Corpus fits in context window	Full Context Loading
Data is structured/queryable	Agentic RAG (build tools)
Large unstructured document corpus	RAG pipeline
Need highest retrieval quality	RAG + ReAG pre-processing
Quick prototype	Full Context Loading, then iterate

Gotchas

• Engineers over-engineer RAG. Start simple (full context), check quality, add complexity only if needed.
• Same embedding model for indexing and querying -- mixing models produces garbage results.
• Chunk size matters more than you think. Too small = fragmented context. Too large = noise.
• Vector DB choice is mostly commoditized. Pick based on what you're already using, not features.
• Hybrid queries (vector + metadata) dramatically improve precision for structured data.
• Reranking is expensive. Only run it on your topK results, never the full database.
• "RAG is dead" takes are premature. But do try simpler alternatives first.

For implementation examples and code, see references/rag-implementation.md.