RAG System Evaluation: Measure What Matters Before Production

RAG evaluation measures the quality of the full retrieval-augmented generation pipeline, not just the generator in isolation. It tracks four distinct dimensions: context precision (are the most relevant chunks ranked first?), context recall (is all necessary information being retrieved?), faithfulness (does the generated answer stay grounded in retrieved context?), and answer relevancy (does the response address what the user actually asked?). Standard LLM evaluation tests generation quality given a fixed prompt. RAG evaluation tests the interaction between retrieval and generation, the failure modes that only emerge from the coupling of the two components.

Production RAG systems fail in six predictable patterns: retrieval miss (the relevant document was never surfaced), wrong ranking (the right document is retrieved at position 8 instead of position 1), wrong chunk boundary (the answer spans two chunks and gets fragmented), hallucination (the generator adds claims not supported by retrieved context), generator drift (the model partially ignores context in favor of training data), and the most dangerous pattern, low faithfulness with correct answers, where the model bypasses retrieval entirely and answers from training data. This last pattern appears to work until your knowledge base updates, at which point the model continues producing stale answers with confidence.

RAGAS is an open-source framework that provides reference implementations of the core RAG evaluation metrics: faithfulness, answer relevancy, context precision, and context recall. It is the conceptual standard for component-wise RAG evaluation and includes a synthetic dataset generator for creating initial golden datasets from your document corpus. RAGAS is best suited for exploration and metric familiarization. For production CI/CD integration, frameworks like DeepEval provide tighter engineering workflow integration. For production monitoring, Patronus and Langfuse add observability and bias detection. Most mature RAG evaluation programs use RAGAS for methodology and combine it with other tools for production use.

A golden dataset for RAG evaluation consists of 100 to 200 question-answer pairs that represent the actual distribution of your production workload. For each pair, you need the relevant source documents in your knowledge base clearly identified. Building it well requires three steps: generating a representative question set (from production logs, synthetic generation, or domain expert input), writing ground-truth answers that reflect expert standards, and validating source attributions so you know which documents should have been retrieved. Domain experts must be involved in validation, because questions that look realistic but lack accurate ground-truth answers produce misleading evaluation results.

Faithfulness measures how factually consistent a RAG-generated response is with the retrieved context. It is computed by extracting all claims from the response and checking each against the retrieved documents. The fraction of supported claims gives the faithfulness score, from 0 to 1. For production deployment, a faithfulness score of 0.75 or higher is the standard minimum threshold. Below 0.75, users will regularly encounter hallucinated claims in responses. Above 0.85, the model is reliably grounded. When faithfulness is low despite relevant context being retrieved, the fix is generator-side: stronger system prompts, lower temperature, or a model with better instruction following.

With GPT-4o-mini as the evaluation judge, expect approximately $0.001 to $0.003 per test case across five metrics. A 200-question golden dataset costs under $1 per evaluation run, making it economically trivial to run on every CI/CD build. For production monitoring, the pattern most teams adopt is sampled evaluation: 1 to 5 percent of production traces randomly, plus 100 percent of low-confidence outputs, plus all outputs from new feature deployments during the first 48 hours. For European teams with sovereignty requirements, note that using a US-based judge model sends production data through US infrastructure. EU-sovereign judge models like Mistral are available as alternatives with some quality trade-off.