Rag evaluation matrix
Skill mouadja02/skills/skills/agent-eval/rag-evaluation-matrix
A curated collection of agent skills for your AI agents - engineering craft, prompt engineering, design, growth marketing, ...
npx -y skills add mouadja02/skills --skill rag-evaluation-matrixAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 8 stars8 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Use when comparing basic, enhanced, GraphRAG, or agentic RAG designs, evaluating domain-specific RAG quality, tuning retrieval components, or deciding whether agentic RAG is worth its cost.
SKILL.md
3.0 KB, as published. Nobody here has run it
RAG Evaluation Matrix
Do not choose agentic RAG by fashion. Compare retrieval designs against domain questions, answerability, correctness, cost, latency, and failure reasons.
Use When
- The user asks whether to use basic RAG, enhanced RAG, GraphRAG, or agentic RAG.
- A RAG system works on demos but fails on domain-specific questions.
- You need to compare embedding models, rerankers, chunking, query rewriting, or tool orchestration.
- LLM-as-judge scores need alignment with human review.
Evaluation Flow
- Build a domain question set with answerable, unanswerable, ambiguous, multi-hop, and adversarial cases.
- For each candidate pipeline, log retrieved evidence, final answer, citations, latency, token usage, and cost.
- Score answer correctness and answerability separately.
- Add evidence quality checks: citation support, contradiction handling, source freshness, and missing-source diagnosis.
- Segment results by question type instead of reporting only one aggregate score.
- Inspect low-correctness failures and map them to retrieval, synthesis, tool orchestration, or corpus gaps.
- Pick the simplest design that meets quality, latency, and cost constraints.
Matrix
| Dimension | Basic RAG | Enhanced RAG | Agentic RAG |
|---|---|---|---|
| Best for | Stable FAQ and narrow corpora | Noisy corpora, query mismatch, reranking | Multi-step, ambiguous, tool-rich tasks |
| Main risk | Weak recall and unsupported answers | Pipeline complexity | Cost, latency, loops, tool misuse |
| Eval focus | Retrieval recall and citation support | Component ablations | Trajectory, action choice, stopping behavior |
| Ship gate | Correctness and answerability meet threshold | Ablation proves each module helps | Agentic gains justify extra cost |
Script
Use the helper to combine per-run JSON metrics into a decision table:
python skills/agent-eval/rag-evaluation-matrix/scripts/rag_eval_matrix.py results/*.json
Expected JSON fields: pipeline, question_type, correct, answerable_correct, latency_ms, cost_usd.
Common Mistakes
| Mistake | Fix |
|---|---|
| Optimizing average score only | Break down by question type and domain |
| Judging unanswerable questions as wrong by default | Score answerability separately |
| Skipping human calibration | Sample judge disagreements and tune rubrics |
| Choosing agentic RAG without ablation | Compare against enhanced RAG at equal budget |
| Ignoring failure reasons | Classify each miss before tuning |
References
- arXiv: Is Agentic RAG worth it? - https://arxiv.org/abs/2601.07711
- arXiv: RAGalyst - https://arxiv.org/abs/2511.04502
- Hugging Face Papers: RAGalyst - https://huggingface.co/papers/2511.04502
- Hugging Face dataset: RAGalyst QAC - https://huggingface.co/datasets/hoskerelab/ragalyst-qac