Embedding optimization

Embedding Optimization

Optimize embedding generation for cost, performance, and quality in RAG and semantic search systems.

When to Use This Skill

Trigger this skill when:

• Building RAG (Retrieval Augmented Generation) systems
• Implementing semantic search or similarity detection
• Optimizing embedding API costs (reducing by 70-90%)
• Improving document retrieval quality through better chunking
• Processing large document corpora (thousands to millions of documents)
• Selecting between API-based vs. local embedding models

Model Selection Framework

Choose the optimal embedding model based on requirements:

Quick Recommendations:

• Startup/MVP: all-MiniLM-L6-v2 (local, 384 dims, zero API costs)
• Production: text-embedding-3-small (API, 1,536 dims, balanced quality/cost)
• High Quality: text-embedding-3-large (API, 3,072 dims, premium)
• Multilingual: multilingual-e5-base (local, 768 dims) or Cohere embed-multilingual-v3.0

For detailed decision frameworks including cost comparisons, quality benchmarks, and data privacy considerations, see references/model-selection-guide.md.

Model Comparison Summary:

Model	Type	Dimensions	Cost per 1M tokens	Best For
all-MiniLM-L6-v2	Local	384	$0 (compute only)	High volume, tight budgets
BGE-base-en-v1.5	Local	768	$0 (compute only)	Quality + cost balance
text-embedding-3-small	API	1,536	$0.02	General purpose production
text-embedding-3-large	API	3,072	$0.13	Premium quality requirements
embed-multilingual-v3.0	API	1,024	$0.10	100+ language support

Chunking Strategies

Select chunking strategy based on content type and use case:

Content Type → Strategy Mapping:

• Documentation: Recursive (heading-aware), 800 chars, 100 overlap
• Code: Recursive (function-level), 1,000 chars, 100 overlap
• Q&A/FAQ: Fixed-size, 500 chars, 50 overlap (precise retrieval)
• Legal/Technical: Semantic (large), 1,500 chars, 200 overlap (context preservation)
• Blog Posts: Semantic (paragraph), 1,000 chars, 100 overlap
• Academic Papers: Recursive (section-aware), 1,200 chars, 150 overlap

For detailed chunking patterns, decision trees, and implementation guidance, see references/chunking-strategies.md.

Quick Start with CLI:

python scripts/chunk_document.py \
  --input document.txt \
  --content-type markdown \
  --chunk-size 800 \
  --overlap 100 \
  --output chunks.jsonl

Caching Implementation

Achieve 80-90% cost reduction through content-addressable caching.

Caching Architecture by Query Volume:

• <10K queries/month: In-memory cache (Python lru_cache)
• 10K-100K queries/month: Redis (fast, TTL-based expiration)
• 100K-1M queries/month: Redis (hot) + PostgreSQL (warm)
• >1M queries/month: Multi-tier (Redis + PostgreSQL + S3)

Production Caching with Redis:

# Embed documents with caching enabled
python scripts/cached_embedder.py \
  --model text-embedding-3-small \
  --input documents.jsonl \
  --output embeddings.npy \
  --cache-backend redis \
  --cache-ttl 2592000  # 30 days

Caching ROI Example:

• 50,000 document chunks
• 20% duplicate content
• Without caching: $0.50 API cost
• With caching (60% hit rate): $0.20 API cost
• Savings: 60% ($0.30)

Dimensionality Trade-offs

Balance storage, search speed, and quality:

Dimensions	Storage (1M vectors)	Search Speed (p95)	Quality	Use Case
384	1.5 GB	10ms	Good	Large-scale search
768	3 GB	15ms	High	General purpose RAG
1,536	6 GB	25ms	Very High	High-quality retrieval
3,072	12 GB	40ms	Highest	Premium applications

Key Insight: For most RAG applications, 768 dimensions (BGE-base-en-v1.5 local or equivalent) provides the best quality/cost/speed balance.

Batch Processing Optimization

Maximize throughput for large-scale ingestion:

OpenAI API:

• Batch up to 2,048 inputs per request
• Implement rate limiting (tier-dependent: 500-5,000 RPM)
• Use parallel requests with backoff on rate limits

Local Models (sentence-transformers):

• GPU acceleration (CUDA, MPS for Apple Silicon)
• Batch size tuning (32-128 based on GPU memory)
• Multi-GPU support for maximum throughput

Expected Throughput:

• OpenAI API: 1,000-5,000 texts/minute (rate limit dependent)
• Local GPU (RTX 3090): 5,000-10,000 texts/minute
• Local CPU: 100-500 texts/minute

Performance Monitoring

Track key metrics for optimization:

Critical Metrics:

• Latency: Embedding generation time (p50, p95, p99)
• Throughput: Embeddings per second/minute
• Cost: API usage tracking (USD per 1K/1M tokens)
• Cache Efficiency: Hit rate percentage

For detailed monitoring setup, metric collection patterns, and dashboarding, see references/performance-monitoring.md.

Monitor with Wrapper:

from scripts.performance_monitor import MonitoredEmbedder

monitored = MonitoredEmbedder(
    embedder=your_embedder,
    cost_per_1k_tokens=0.00002  # OpenAI pricing
)

embeddings = monitored.embed_batch(texts)
metrics = monitored.get_metrics()
print(f"Cache hit rate: {metrics['cache_hit_rate_pct']}%")
print(f"Total cost: ${metrics['total_cost_usd']}")

Working Examples

See examples/ directory for complete implementations:

Python Examples:

• examples/openai_cached.py - OpenAI embeddings with Redis caching
• examples/local_embedder.py - sentence-transformers local embedding
• examples/smart_chunker.py - Content-aware recursive chunking
• examples/performance_monitor.py - Pipeline performance tracking
• examples/batch_processor.py - Large-scale document processing

All examples include:

• Complete, runnable code
• Dependency installation instructions
• Error handling and retry logic
• Configuration options

Integration Points

Upstream (This skill provides to):

• Vector Databases: Embeddings flow to Pinecone, Weaviate, Qdrant, pgvector
• RAG Systems: Optimized embeddings for retrieval pipelines
• Semantic Search: Query and document embeddings for similarity search

Downstream (This skill uses from):

• Document Processing: Chunk documents before embedding
• Data Ingestion: Process documents from various sources

Related Skills:

• For RAG architecture, see building-ai-chat skill
• For vector database operations, see databases-vector skill
• For data ingestion pipelines, see ingesting-data skill

Common Patterns

Pattern 1: RAG Pipeline

Document → Chunk → Embed → Store (vector DB) → Retrieve

Pattern 2: Semantic Search

Query → Embed → Search (vector DB) → Rank → Display

Pattern 3: Multi-Stage Retrieval (Cost Optimization)

Query → Cheap Embedding (384d) → Initial Search →
Expensive Embedding (1,536d) → Rerank Top-K → Return

Cost Savings: 70% reduction vs. single-stage with expensive embeddings

Quick Reference Checklist

Model Selection:

• Identified data privacy requirements (local vs. API)
• Calculated expected query volume
• Determined quality requirements (good/high/highest)
• Checked multilingual support needs

Chunking:

• Analyzed content type (code, docs, legal, etc.)
• Selected appropriate chunk size (500-1,500 chars)
• Set overlap to prevent context loss (50-200 chars)
• Validated chunks preserve semantic boundaries

Caching:

• Implemented content-addressable hashing
• Selected cache backend (Redis, PostgreSQL)
• Set TTL based on content volatility
• Monitoring cache hit rate (target: >60%)

Performance:

• Tracking latency (embedding generation time)
• Measuring throughput (embeddings/sec)
• Monitoring costs (USD spent on API calls)
• Optimizing batch sizes for maximum efficiency