LLMs and Generative AI › Embeddings and Vector Search › Day 265

Hands-on lab — Day 265: Evaluating Retrieval Quality

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/rag_eval_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/rag_eval_lib.py
examples/test_rag_eval_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/rag_eval_lib.py
starter/test_rag_eval_lib.py
tests/run_tests.sh
tests/test_rag_eval_lib.py
troubleshooting.md

Lab README

Lab: Day 265 -- Evaluating RAG with RAGAS and TruLens

Lesson

Day number: 265 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: Evaluating RAG with RAGAS, TruLens, and the RAG Triad.

Purpose

Build and test a RAG Triad Evaluation Harness in Python. Implement automated Faithfulness claim verification, Context Relevance signal-to-noise scoring, and evaluate grounding.

Learning objectives

  • Calculate Faithfulness (groundedness) to detect hallucinations mathematically.
  • Measure Context Relevance to identify noisy retrieved chunks.
  • Build automated regression testing harnesses for RAG systems.
  • Explain the role of LLM-as-a-Judge in modern CI/CD pipelines.

Prerequisites

  • Day 264 (Hybrid Search and Re-Ranking).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Pure Python standard libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/rag_eval_lib.py: Student scaffold file.
  • examples/rag_eval_lib.py: Complete reference implementation.
  • tests/test_rag_eval_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/rag_eval_lib.py

What the commands do

  • Evaluates grounded vs. hallucinated answers.
  • Calculates Context Relevance signal density.
  • Runs unit test assertions.

Expected output

Good Answer Faithfulness: 1.00
Hallucinated Answer Faithfulness: 0.00

Validation steps

  1. Verify grounded answers achieve 1.00 Faithfulness.
  2. Confirm hallucinated claims reduce the score proportionally.
  3. Validate Context Relevance reflects sentence relevance ratios.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Missing claim splits: Ensure regex includes periods, question marks, and exclamation points.

Security notes

All evaluation metrics compute locally in memory.

Extension exercises

  1. Implement Answer Relevance cosine scoring against synthetic questions.
  2. Build a Pytest CI/CD regression gate.

Expected output

FIELDS.md

# Expected Output Fields: Day 265

- `Good Answer Faithfulness`: Grounded claim score.
- `Hallucinated Answer Faithfulness`: Ungrounded claim score.

examples-run.txt

Good Answer Faithfulness: 1.00
Hallucinated Answer Faithfulness: 0.00

measured-values.txt

Good Answer Faithfulness: 1.00
Hallucinated Answer Faithfulness: 0.00

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_rag_eval_lib.py::test_faithfulness_grounded_answer PASSED             [ 25%]
tests/test_rag_eval_lib.py::test_faithfulness_hallucinated_claims PASSED         [ 50%]
tests/test_rag_eval_lib.py::test_context_relevance_calculation PASSED           [ 75%]
tests/test_rag_eval_lib.py::test_empty_string_handling PASSED                   [100%]

============================== 4 passed in 0.28s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/rag_eval_lib.py (2801 bytes)
import re
from typing import List, Dict, Any

# Split on sentence terminators, but never on a decimal point: a naive
# re.split(r'[.!?]') turns "99.99% uptime." into two claims and quietly halves
# the faithfulness score of any answer that quotes a figure.
_SENTENCE_END = re.compile(r'(?:[!?]|(?<!\d)\.|\.(?!\d))\s*')


class RAGTriadEvaluator:
    def __init__(self):
        pass

    def evaluate_faithfulness(self, context: str, answer: str) -> Dict[str, Any]:
        sentences = [s.strip() for s in re.split(_SENTENCE_END, answer) if s.strip()]
        if not sentences:
            return {"faithfulness_score": 1.0, "total_claims": 0, "supported_claims": 0, "details": []}

        context_lower = context.lower()
        supported = 0
        claim_results = []

        for s in sentences:
            words = [w for w in re.findall(r'\w+', s.lower()) if len(w) > 3]
            if not words:
                supported += 1
                claim_results.append({"claim": s, "supported": True})
                continue
            
            matches = sum(1 for w in words if w in context_lower)
            is_supported = (matches / len(words)) >= 0.5
            if is_supported:
                supported += 1
            claim_results.append({"claim": s, "supported": is_supported})

        score = supported / len(sentences)
        return {
            "faithfulness_score": score,
            "total_claims": len(sentences),
            "supported_claims": supported,
            "details": claim_results
        }

    def evaluate_context_relevance(self, query: str, context: str) -> float:
        q_words = set(re.findall(r'\w+', query.lower()))
        c_sentences = [s.strip() for s in re.split(r'[.!?]', context) if s.strip()]
        if not c_sentences or not q_words:
            return 0.0

        relevant_sentences = 0
        for s in c_sentences:
            s_words = set(re.findall(r'\w+', s.lower()))
            if q_words.intersection(s_words):
                relevant_sentences += 1

        return relevant_sentences / len(c_sentences)

def run_evaluation_demo():
    evaluator = RAGTriadEvaluator()
    context = "PostgreSQL 16 supports bidirectional logical replication. Max connections default to 100."
    good_answer = "PostgreSQL 16 provides bidirectional logical replication."
    hallucinated_answer = "PostgreSQL 16 was released in 1985 and supports quantum indexing."

    f_good = evaluator.evaluate_faithfulness(context, good_answer)
    f_bad = evaluator.evaluate_faithfulness(context, hallucinated_answer)

    print(f"Good Answer Faithfulness: {f_good['faithfulness_score']:.2f}")
    print(f"Hallucinated Answer Faithfulness: {f_bad['faithfulness_score']:.2f}")
    return f_good, f_bad

if __name__ == "__main__":
    run_evaluation_demo()
examples/test_rag_eval_lib.py (1369 bytes)
import pytest
from examples.rag_eval_lib import (
    RAGTriadEvaluator
)

def test_faithfulness_grounded_answer():
    evaluator = RAGTriadEvaluator()
    context = "The enterprise SLA guarantees 99.99% uptime with 24/7 dedicated support."
    answer = "The enterprise SLA guarantees 99.99% uptime."
    res = evaluator.evaluate_faithfulness(context, answer)
    assert res["faithfulness_score"] == 1.0
    assert res["supported_claims"] == 1

def test_faithfulness_hallucinated_claims():
    evaluator = RAGTriadEvaluator()
    context = "The server runs on port 8080."
    answer = "The server runs on port 8080. It also mines bitcoin automatically."
    res = evaluator.evaluate_faithfulness(context, answer)
    assert res["faithfulness_score"] == 0.5
    assert res["total_claims"] == 2
    assert res["supported_claims"] == 1

def test_context_relevance_calculation():
    evaluator = RAGTriadEvaluator()
    query = "database connection timeout"
    context = "Database connection timeout is 30 seconds. Weather is sunny today. Cats are mammals."
    score = evaluator.evaluate_context_relevance(query, context)
    assert round(score, 2) == 0.33

def test_empty_string_handling():
    evaluator = RAGTriadEvaluator()
    assert evaluator.evaluate_faithfulness("", "")["faithfulness_score"] == 1.0
    assert evaluator.evaluate_context_relevance("", "") == 0.0
metadata.yml (421 bytes)
lesson_id: D265
day: 265
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/rag_eval_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/rag_eval_lib.py (447 bytes)
import re
from typing import List, Dict, Any

class RAGTriadEvaluator:
    def __init__(self):
        pass

    def evaluate_faithfulness(self, context: str, answer: str) -> Dict[str, Any]:
        # TODO: Split answer into claims and verify grounding against context
        pass

    def evaluate_context_relevance(self, query: str, context: str) -> float:
        # TODO: Calculate fraction of context sentences relevant to query
        pass
starter/test_rag_eval_lib.py (1369 bytes)
import pytest
from examples.rag_eval_lib import (
    RAGTriadEvaluator
)

def test_faithfulness_grounded_answer():
    evaluator = RAGTriadEvaluator()
    context = "The enterprise SLA guarantees 99.99% uptime with 24/7 dedicated support."
    answer = "The enterprise SLA guarantees 99.99% uptime."
    res = evaluator.evaluate_faithfulness(context, answer)
    assert res["faithfulness_score"] == 1.0
    assert res["supported_claims"] == 1

def test_faithfulness_hallucinated_claims():
    evaluator = RAGTriadEvaluator()
    context = "The server runs on port 8080."
    answer = "The server runs on port 8080. It also mines bitcoin automatically."
    res = evaluator.evaluate_faithfulness(context, answer)
    assert res["faithfulness_score"] == 0.5
    assert res["total_claims"] == 2
    assert res["supported_claims"] == 1

def test_context_relevance_calculation():
    evaluator = RAGTriadEvaluator()
    query = "database connection timeout"
    context = "Database connection timeout is 30 seconds. Weather is sunny today. Cats are mammals."
    score = evaluator.evaluate_context_relevance(query, context)
    assert round(score, 2) == 0.33

def test_empty_string_handling():
    evaluator = RAGTriadEvaluator()
    assert evaluator.evaluate_faithfulness("", "")["faithfulness_score"] == 1.0
    assert evaluator.evaluate_context_relevance("", "") == 0.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 265 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_rag_eval_lib.py (1369 bytes)
import pytest
from examples.rag_eval_lib import (
    RAGTriadEvaluator
)

def test_faithfulness_grounded_answer():
    evaluator = RAGTriadEvaluator()
    context = "The enterprise SLA guarantees 99.99% uptime with 24/7 dedicated support."
    answer = "The enterprise SLA guarantees 99.99% uptime."
    res = evaluator.evaluate_faithfulness(context, answer)
    assert res["faithfulness_score"] == 1.0
    assert res["supported_claims"] == 1

def test_faithfulness_hallucinated_claims():
    evaluator = RAGTriadEvaluator()
    context = "The server runs on port 8080."
    answer = "The server runs on port 8080. It also mines bitcoin automatically."
    res = evaluator.evaluate_faithfulness(context, answer)
    assert res["faithfulness_score"] == 0.5
    assert res["total_claims"] == 2
    assert res["supported_claims"] == 1

def test_context_relevance_calculation():
    evaluator = RAGTriadEvaluator()
    query = "database connection timeout"
    context = "Database connection timeout is 30 seconds. Weather is sunny today. Cats are mammals."
    score = evaluator.evaluate_context_relevance(query, context)
    assert round(score, 2) == 0.33

def test_empty_string_handling():
    evaluator = RAGTriadEvaluator()
    assert evaluator.evaluate_faithfulness("", "")["faithfulness_score"] == 1.0
    assert evaluator.evaluate_context_relevance("", "") == 0.0

Troubleshooting

Troubleshooting: Day 265 - Evaluating RAG with RAGAS and TruLens

Common Issues

  1. Empty sentence arrays:
    • Cause: Consecutive delimiters in text.
    • Fix: Filter out empty strings after regex splitting.

Security notes

Security & Privacy: Day 265 - Evaluating RAG with RAGAS and TruLens

Security Guidance

  • Do not store unredacted customer PII in persistent evaluation test benchmarks.