LLMs and Generative AI β€Ί Embeddings and Vector Search β€Ί Day 265

Day 265: Evaluating Retrieval Quality

Day 265 of 365 β€” Evaluating Retrieval Quality

Master the RAG Triad and quantitative evaluation frameworks for Retrieval-Augmented Generation: evaluate Context Relevance, Faithfulness (groundedness / hallucination detection), and Answer Relevance using RAGAS and TruLens; compute precision, recall, and semantic similarity metrics; automate continuous LLM-as-a-judge CI/CD evaluation pipelines, and implement a production RAG Triad evaluation harness in Python.

Course
LLMs and Generative AI
Category
Embeddings and Vector Search
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-265-evaluating-retrieval-quality

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-265-evaluating-retrieval-quality
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Building a prototype Retrieval-Augmented Generation (RAG) pipeline is easy; verifying that it reliably tells the truth in production is extraordinarily hard. When a RAG system fails, it typically fails in one of three distinct ways:

  1. Retrieval Failure (Low Context Relevance): The vector database retrieves 5 pages of irrelevant technical fluff, confusing the LLM.
  2. Hallucination (Low Faithfulness): The LLM ignores the retrieved documentation and invents plausible-sounding but completely fabricated API parameters.
  3. Question Evasion (Low Answer Relevance): The LLM generates a grammatically perfect paragraph that completely fails to answer the user’s actual question.

Without automated, quantitative evaluation, engineers are forced to rely on β€œvibe checks” β€” manually reading 10 responses and guessing if an embedding change helped or hurt.

Modern RAG evaluation frameworks (such as RAGAS and TruLens) replace guesswork with rigorous, automated mathematics. By implementing the RAG Triad and LLM-as-a-Judge CI/CD testing, you can automatically score retrieval recall, detect hallucinations, and gate pull requests before flawed prompts reach production users.

Today, you will master the RAG Triad, RAGAS reference-free metrics, TruLens feedback functions, and build a production RAG Triad evaluation harness in Python.


The idea in plain language

Think of a court trial with a witness, a folder of evidence documents, and a judge:


Historical background

Diagram: the historical progression of retrieval-augmented generation evaluation

RAG triad evaluation framework diagram

  1. 2002–2004 (BLEU & ROUGE): Early NLP evaluation relied on n-gram overlap with human references, proving brittle for generative language models.
  2. 2023 (TruLens & The RAG Triad): TruEra introduced the RAG Triad, decoupling retrieval quality from generation faithfulness.
  3. 2023 (RAGAS Framework): Es et al. published RAGAS, establishing formal algorithms for computing Context Precision, Context Recall, Faithfulness, and Answer Relevance without requiring ground-truth human annotations.
  4. 2024–2026 (Continuous LLM CI/CD): Industry standard shifted from offline benchmarking to continuous automated pull-request evaluation gates using LLM-as-a-Judge test harnesses.

What it is β€” and what it is not

What RAG Evaluation IS:

What it is NOT:


Why it was created and what problems it solves

Quantitative RAG evaluation solves 4 major engineering bottlenecks:

  1. Elimination of Subjective β€œVibe Testing”: Provides objective numerical metrics (e.g. Faithfulness = 0.94) to guide system optimizations.
  2. Pinpointing Component Failure: Identifies whether a bug originates in the chunker, the embedding retriever, or the generative LLM prompt.
  3. Automated CI/CD Regression Gates: Prevents prompt edits, model swaps, or chunking changes from silently degrading production accuracy.
  4. Reference-Free Evaluation: Scores production queries in real time without needing human-labeled ground-truth answer sheets.

How it works

Let us analyze the mathematical definitions and algorithms behind the RAG Triad and RAGAS metrics.


1. The RAG Triad Architecture

Diagram: the three measurements that locate a failure in a retrieval pipeline

RAGAS automated evaluation pipeline diagram

The RAG Triad isolates the 3 fundamental interactions in any retrieval pipeline:

  1. Query \to Context (Context Relevance): Evaluates if the retrieved chunks are relevant to the query.
  2. Context \to Answer (Faithfulness / Groundedness): Evaluates if every claim in the answer is entailed by the retrieved context.
  3. Query \to Answer (Answer Relevance): Evaluates if the answer directly addresses the query.

2. Mathematical Formulation: Faithfulness (Hallucination Detection)

Diagram: how faithfulness is measured by checking each claim against the sources

Faithfulness measures the fraction of generated factual claims supported by context:

\text{Faithfulness} = \frac{|V|}{|S|}


3. Mathematical Formulation: Context Relevance

Context relevance penalizes irrelevant noise in retrieved passages:

\text{Context Relevance} = \frac{\text{Number of relevant sentences in } C}{\text{Total sentences in } C}


4. Mathematical Formulation: Answer Relevance

Answer relevance measures query alignment without requiring ground truth:

  1. An LLM generates M hypothetical questions that would yield the given answer: \{q_1, q_2, ..., q_m\}.
  2. An embedding model embeds the original query q and each generated question q_i.
  3. Compute the mean cosine similarity: \text{Answer Relevance} = \frac{1}{M} \sum_{i=1}^{M} \cos(E(q), E(q_i))

5. Context Precision and Context Recall (With Ground Truth)

Diagram: retrieval precision compared with retrieval recall

When an evaluation benchmark includes human reference answers:


6. LLM-as-a-Judge Calibration and Bias Mitigation

Diagram: how a model-based evaluation judge is calibrated against human ratings

When using an LLM to evaluate another LLM:


7. Synthetic Test Dataset Generation (Evol-Instruct)

Diagram: how an evaluation set is generated from an existing corpus

Manual curation of 1,000 test Q&A pairs is slow and expensive:


8. TruLens Feedback Functions

TruLens implements feedback functions as decorator primitives:

from trulens_eval.feedback import Feedback, Groundedness

# Define groundedness feedback function
grounded = Groundedness(groundedness_provider=provider)
f_groundedness = Feedback(
    grounded.groundedness_measure_with_cot_reasons
).on(TruApp.context).on_output()

9. Cost and Latency of Evaluation Pipelines

Diagram: exhaustive evaluation compared with sampled continuous evaluation

Evaluating 500 test questions with an LLM judge:


10. Summary: Shifting RAG Engineering from Guesswork to Science


Deep Dive: Comprehensive IR Metrics and Discounted Cumulative Gain

In Information Retrieval and RAG evaluation, measuring retrieval precision requires understanding the mathematical mechanics of binary vs. graded relevance metrics:

  1. Mean Reciprocal Rank (MRR@K): MRR evaluates the system’s ability to return a relevant item at the highest possible rank position. For a query set Q:
MRR = \frac(1)(|Q|) sum_(i=1)^(|Q|) \frac(1)(\text(rank)_i)

Where \text(rank)_i is the position of the first ground-truth relevant document. If no relevant document is retrieved within the top K, the reciprocal rank is 0. MRR is ideal for navigational and factoid Q&A search where a single authoritative document satisfies the user’s information need.

  1. Normalized Discounted Cumulative Gain (NDCG@K): Unlike MRR, NDCG handles graded relevance judgments (e.g. relevance scale 0 to 3). Discounted Cumulative Gain at rank K is defined as:
DCG@K = sum_(i=1)^(K) \frac(2^(rel_i) - 1)(\log_2(i + 1))

The Ideal DCG (IDCG@K) represents the theoretical maximum DCG obtained by sorting all known relevant documents in descending order of relevance. The normalized score is:

NDCG@K = \frac(DCG@K)(IDCG@K)

NDCG penalizes relevant documents appearing lower in the ranking list logarithmically, reflecting user attention drop-off in search interfaces.

  1. Hit Rate (Recall@K): The percentage of evaluation queries for which at least one relevant document appears within the top K retrieved candidates. In multi-stage RAG architectures, Recall@50 at the first retrieval stage determines the theoretical ceiling for the downstream cross-encoder re-ranker.

The Mathematics of Faithfulness and Triad Gating

Automated LLM-as-a-Judge evaluation relies on deterministic claim decomposition:

v(c_i, R) = \begin(cases) 1 & \text(if ) R entails c_i \ 0 & \text(otherwise) \end(cases)
\text(Faithfulness)(A, R) = \frac(sum_(i=1)^n v(c_i, R))(n)

Statistical Significance Testing in Retrieval Evaluations

When evaluating improvements to hybrid search algorithms or new embedding models, observing a small bump in MRR@10 (e.g. from 0.88 to 0.91) can be misleading without statistical rigor. Production engineering teams use paired Student t-tests or Wilcoxon signed-rank tests over golden query sets of at least 200 distinct test cases to verify that p-values remain strictly below 0.05. This ensures that observed retrieval performance gains represent genuine systemic advancements rather than random sampling noise across specific keyword distributions.

An everyday analogy

Diagram: a single overall measurement compared with separate diagnostic measurements

Think of quality control in automobile manufacturing:


Examples in practice

Let us build a RAG Triad Evaluation Harness:

import re
from typing import List, Dict, Any

# Split on sentence terminators, but never on a decimal point.
_SENTENCE_END = re.compile(r'(?:[!?]|(?<!\d)\.|\.(?!\d))\s*')


class RAGTriadEvaluator:
    def __init__(self):
        pass

    def evaluate_faithfulness(self, context: str, answer: str) -> Dict[str, Any]:
        # Calculates factual claim groundedness against retrieved context.
        # Never split on a decimal point: a naive re.split(r'[.!?]') turns
        # "99.99% uptime." into two claims and halves the score of any answer
        # that quotes a figure.
        sentences = [s.strip() for s in re.split(_SENTENCE_END, answer) if s.strip()]
        if not sentences:
            return {"faithfulness_score": 1.0, "total_claims": 0, "supported_claims": 0}

        context_lower = context.lower()
        supported = 0
        claim_results = []

        for s in sentences:
            # Word overlap verification heuristic (simulating LLM judge entailment)
            words = [w for w in re.findall(r'\w+', s.lower()) if len(w) > 3]
            if not words:
                supported += 1
                claim_results.append({"claim": s, "supported": True})
                continue
            
            matches = sum(1 for w in words if w in context_lower)
            is_supported = (matches / len(words)) >= 0.5
            if is_supported:
                supported += 1
            claim_results.append({"claim": s, "supported": is_supported})

        score = supported / len(sentences)
        return {
            "faithfulness_score": score,
            "total_claims": len(sentences),
            "supported_claims": supported,
            "details": claim_results
        }

    def evaluate_context_relevance(self, query: str, context: str) -> float:
        # Measures keyword signal density between query and retrieved context
        q_words = set(re.findall(r'\w+', query.lower()))
        c_sentences = [s.strip() for s in re.split(r'[.!?]', context) if s.strip()]
        if not c_sentences or not q_words:
            return 0.0

        relevant_sentences = 0
        for s in c_sentences:
            s_words = set(re.findall(r'\w+', s.lower()))
            if q_words.intersection(s_words):
                relevant_sentences += 1

        return relevant_sentences / len(c_sentences)

Implications: security, privacy, performance, scalability, and cost

Diagram: the debugging, cost, privacy and process implications of retrieval evaluation

  1. Judge Model Prompt Injection: Ensure student queries evaluated by the judge LLM are formatted inside clear boundary delimiters (<user_query>...</user_query>) to prevent adversarial jailbreaks during automated evaluation runs.
  2. Data Privacy in CI/CD: Do not include real customer PII in your golden evaluation benchmark datasets; use synthetic or anonymized corpora.

Alternatives: free, open source, and commercial

ToolFocusLicense / TypeBest For
RAGASReference-free RAG metricsOpen Source (Apache-2.0)CI/CD evaluation test suites
TruLensRAG Triad & ObservabilityOpen SourceReal-time production tracing
DeepEvalUnit-testing for LLMsOpen SourcePytest integration in DevOps
Arize PhoenixEvals + TracingOpen Source / CloudFull-stack RAG observability

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   EVALUATION FRAMEWORK COMPARISON                      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Framework          β”‚ Metrics         β”‚ Tracing / UI    β”‚ CI/CD Focus   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ RAGAS              β”‚ RAG Triad, MRR  β”‚ Programmatic    β”‚ Excellent     β”‚
β”‚ TruLens            β”‚ Triad + Custom  β”‚ Rich Web UI     β”‚ Very Good     β”‚
β”‚ DeepEval           β”‚ 14+ LLM Metrics β”‚ Pytest-native   β”‚ Maximum       β”‚
β”‚ Traditional BLEU   β”‚ N-gram overlap  β”‚ CLI             β”‚ Poor for RAG  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when formal retrieval evaluation is warranted

When to Use RAG Evaluation:

When NOT to use:


The AI connection

Knowledge check

  1. What are the three vertices of the RAG Triad and what relationship does each test?
  2. How does the Faithfulness metric mathematically quantify hallucination?
  3. Why are BLEU and ROUGE inadequate for evaluating generative RAG answers?
  4. What is Position Bias in LLM-as-a-judge systems and how is it mitigated?
  5. How does Context Relevance help developers diagnose retrieval noise?

Hands-on exercise

In this lab, you will build and test a RAG Triad Evaluation Harness in Python: implement automated Faithfulness claim verification, Context Relevance signal-to-noise scoring, and evaluate a test dataset of query-context-answer triples.

Expected output

[RAG Triad Evaluation Verification Suite]
Testing Faithfulness Claim Verifier:
  Ground-truth grounded answer: Faithfulness = 1.00 [PASS]
  Hallucinated answer: Faithfulness = 0.33 (Detected 2 hallucinations) [PASS]
Testing Context Relevance Scorer:
  High-density context: Relevance = 1.00 [PASS]
  Noisy context with distractors: Relevance = 0.33 [PASS]
Testing Automated Benchmark Runner:
  Evaluated 3 test triples in 0.12s [PASS]
Test Suite: 4 passed in 0.38s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Answer Relevance Evaluation: Build a question generation generator that computes cosine similarity between synthetic queries and the original prompt.
  2. Build a Pytest CI/CD Gate that fails if a pull request drops RAG Triad Faithfulness below 0.90.

Extension challenge

Implement a Self-Consistency Majority Vote Judge:

Quiz

Q1. What does the "Faithfulness" metric evaluate in the RAG Triad?

  1. The token length of the system prompt
  2. Whether the LLM response is polite
  3. The latency of the vector search query
  4. Whether every claim is grounded in the retrieved context
Show answer

Answer: D. Whether every claim is grounded in the retrieved context

Faithfulness directly measures hallucination: it checks if claims are supported by the retrieved context.

Q2. How is "Context Relevance" calculated in RAG evaluation?

  1. The share of retrieved sentences that are actually relevant
  2. By counting the number of characters in the whole database
  3. By checking the price of the embedding API
  4. By calculating the average word length
Show answer

Answer: A. The share of retrieved sentences that are actually relevant

Context relevance ensures that retrieved passages contain high signal-to-noise ratio without irrelevant clutter.

Q3. What is the primary difference between Context Precision and Context Recall in RAGAS?

  1. Context Precision only works on numbers
  2. Precision checks the ranks; recall checks completeness
  3. Context Recall requires a GPU
  4. There is no mathematical difference
Show answer

Answer: B. Precision checks the ranks; recall checks completeness

Precision measures signal ranking quality; Recall measures comprehensive coverage of necessary facts.

Q4. Why is LLM-as-a-judge commonly employed to score RAG outputs rather than traditional string overlap metrics like BLEU or ROUGE?

  1. Because ROUGE cannot parse JSON
  2. Because BLEU is deprecated in Python
  3. They measure n-gram overlap, not meaning or truth
  4. Because LLMs are cheaper than string comparisons
Show answer

Answer: C. They measure n-gram overlap, not meaning or truth

Generative answers can be 100% correct while having zero n-gram overlap with ground truth references.

Q5. How can RAG evaluation be integrated into continuous deployment (CI/CD) pipelines to prevent regressions?

  1. By disabling the re-ranker during testing
  2. By deleting the vector database before every deploy
  3. By manually inspecting all queries in production
  4. Run a golden set in CI and fail below a threshold
Show answer

Answer: D. Run a golden set in CI and fail below a threshold

Automated evaluation gates ensure prompt updates or chunking tweaks do not introduce hallucinations.

Glossary

RAG Triad
An evaluation framework measuring Context Relevance, Grounded Faithfulness, and Answer Relevance.
Faithfulness (Groundedness)
The percentage of claims in the generated response that can be mathematically or logically entailed from the retrieved context.
Context Relevance
The ratio of useful, query-pertinent sentences in the retrieved context compared to total retrieved tokens.
Answer Relevance
How directly and completely the generated output addresses the specific inquiry posed by the user.
Context Precision
A metric evaluating whether the most relevant context chunks are ranked at the highest positions in the retrieval list.
Context Recall
The proportion of ground-truth reference sentences that are successfully contained within the retrieved context.
LLM-as-a-Judge
Using a high-capability language model (e.g. GPT-4o, Claude 3.7) to evaluate and score text based on structured rubrics.
Golden Evaluation Dataset
A curated benchmark suite of test questions, ground-truth contexts, and reference answers used for CI/CD regression testing.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.