LLMs and Generative AI › Embeddings and Vector Search › Day 265
Hands-on lab — Day 265: Evaluating Retrieval Quality
- ← Back to the Day 265 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-265-evaluating-retrieval-quality/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/rag_eval_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/rag_eval_lib.py examples/test_rag_eval_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/rag_eval_lib.py starter/test_rag_eval_lib.py tests/run_tests.sh tests/test_rag_eval_lib.py troubleshooting.md
Lab README
Lab: Day 265 -- Evaluating RAG with RAGAS and TruLens
Lesson
Day number: 265 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: Evaluating RAG with RAGAS, TruLens, and the RAG Triad.
Purpose
Build and test a RAG Triad Evaluation Harness in Python. Implement automated Faithfulness claim verification, Context Relevance signal-to-noise scoring, and evaluate grounding.
Learning objectives
- Calculate Faithfulness (groundedness) to detect hallucinations mathematically.
- Measure Context Relevance to identify noisy retrieved chunks.
- Build automated regression testing harnesses for RAG systems.
- Explain the role of LLM-as-a-Judge in modern CI/CD pipelines.
Prerequisites
- Day 264 (Hybrid Search and Re-Ranking).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 512 MB RAM.
- 50 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Pure Python standard libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/rag_eval_lib.py: Student scaffold file.examples/rag_eval_lib.py: Complete reference implementation.tests/test_rag_eval_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/rag_eval_lib.py
What the commands do
- Evaluates grounded vs. hallucinated answers.
- Calculates Context Relevance signal density.
- Runs unit test assertions.
Expected output
Good Answer Faithfulness: 1.00
Hallucinated Answer Faithfulness: 0.00
Validation steps
- Verify grounded answers achieve 1.00 Faithfulness.
- Confirm hallucinated claims reduce the score proportionally.
- Validate Context Relevance reflects sentence relevance ratios.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Missing claim splits: Ensure regex includes periods, question marks, and exclamation points.
Security notes
All evaluation metrics compute locally in memory.
Extension exercises
- Implement Answer Relevance cosine scoring against synthetic questions.
- Build a Pytest CI/CD regression gate.
Navigation
- Lesson title: Evaluating Retrieval Quality
- Day number: 265 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-265-evaluating-retrieval-quality
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-265-evaluating-retrieval-qualitywhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 265
- `Good Answer Faithfulness`: Grounded claim score.
- `Hallucinated Answer Faithfulness`: Ungrounded claim score.
examples-run.txt
Good Answer Faithfulness: 1.00
Hallucinated Answer Faithfulness: 0.00
measured-values.txt
Good Answer Faithfulness: 1.00
Hallucinated Answer Faithfulness: 0.00
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 4 items
tests/test_rag_eval_lib.py::test_faithfulness_grounded_answer PASSED [ 25%]
tests/test_rag_eval_lib.py::test_faithfulness_hallucinated_claims PASSED [ 50%]
tests/test_rag_eval_lib.py::test_context_relevance_calculation PASSED [ 75%]
tests/test_rag_eval_lib.py::test_empty_string_handling PASSED [100%]
============================== 4 passed in 0.28s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/rag_eval_lib.py (2801 bytes)
import re
from typing import List, Dict, Any
# Split on sentence terminators, but never on a decimal point: a naive
# re.split(r'[.!?]') turns "99.99% uptime." into two claims and quietly halves
# the faithfulness score of any answer that quotes a figure.
_SENTENCE_END = re.compile(r'(?:[!?]|(?<!\d)\.|\.(?!\d))\s*')
class RAGTriadEvaluator:
def __init__(self):
pass
def evaluate_faithfulness(self, context: str, answer: str) -> Dict[str, Any]:
sentences = [s.strip() for s in re.split(_SENTENCE_END, answer) if s.strip()]
if not sentences:
return {"faithfulness_score": 1.0, "total_claims": 0, "supported_claims": 0, "details": []}
context_lower = context.lower()
supported = 0
claim_results = []
for s in sentences:
words = [w for w in re.findall(r'\w+', s.lower()) if len(w) > 3]
if not words:
supported += 1
claim_results.append({"claim": s, "supported": True})
continue
matches = sum(1 for w in words if w in context_lower)
is_supported = (matches / len(words)) >= 0.5
if is_supported:
supported += 1
claim_results.append({"claim": s, "supported": is_supported})
score = supported / len(sentences)
return {
"faithfulness_score": score,
"total_claims": len(sentences),
"supported_claims": supported,
"details": claim_results
}
def evaluate_context_relevance(self, query: str, context: str) -> float:
q_words = set(re.findall(r'\w+', query.lower()))
c_sentences = [s.strip() for s in re.split(r'[.!?]', context) if s.strip()]
if not c_sentences or not q_words:
return 0.0
relevant_sentences = 0
for s in c_sentences:
s_words = set(re.findall(r'\w+', s.lower()))
if q_words.intersection(s_words):
relevant_sentences += 1
return relevant_sentences / len(c_sentences)
def run_evaluation_demo():
evaluator = RAGTriadEvaluator()
context = "PostgreSQL 16 supports bidirectional logical replication. Max connections default to 100."
good_answer = "PostgreSQL 16 provides bidirectional logical replication."
hallucinated_answer = "PostgreSQL 16 was released in 1985 and supports quantum indexing."
f_good = evaluator.evaluate_faithfulness(context, good_answer)
f_bad = evaluator.evaluate_faithfulness(context, hallucinated_answer)
print(f"Good Answer Faithfulness: {f_good['faithfulness_score']:.2f}")
print(f"Hallucinated Answer Faithfulness: {f_bad['faithfulness_score']:.2f}")
return f_good, f_bad
if __name__ == "__main__":
run_evaluation_demo()
examples/test_rag_eval_lib.py (1369 bytes)
import pytest
from examples.rag_eval_lib import (
RAGTriadEvaluator
)
def test_faithfulness_grounded_answer():
evaluator = RAGTriadEvaluator()
context = "The enterprise SLA guarantees 99.99% uptime with 24/7 dedicated support."
answer = "The enterprise SLA guarantees 99.99% uptime."
res = evaluator.evaluate_faithfulness(context, answer)
assert res["faithfulness_score"] == 1.0
assert res["supported_claims"] == 1
def test_faithfulness_hallucinated_claims():
evaluator = RAGTriadEvaluator()
context = "The server runs on port 8080."
answer = "The server runs on port 8080. It also mines bitcoin automatically."
res = evaluator.evaluate_faithfulness(context, answer)
assert res["faithfulness_score"] == 0.5
assert res["total_claims"] == 2
assert res["supported_claims"] == 1
def test_context_relevance_calculation():
evaluator = RAGTriadEvaluator()
query = "database connection timeout"
context = "Database connection timeout is 30 seconds. Weather is sunny today. Cats are mammals."
score = evaluator.evaluate_context_relevance(query, context)
assert round(score, 2) == 0.33
def test_empty_string_handling():
evaluator = RAGTriadEvaluator()
assert evaluator.evaluate_faithfulness("", "")["faithfulness_score"] == 1.0
assert evaluator.evaluate_context_relevance("", "") == 0.0
metadata.yml (421 bytes)
lesson_id: D265
day: 265
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/rag_eval_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/rag_eval_lib.py (447 bytes)
import re
from typing import List, Dict, Any
class RAGTriadEvaluator:
def __init__(self):
pass
def evaluate_faithfulness(self, context: str, answer: str) -> Dict[str, Any]:
# TODO: Split answer into claims and verify grounding against context
pass
def evaluate_context_relevance(self, query: str, context: str) -> float:
# TODO: Calculate fraction of context sentences relevant to query
pass
starter/test_rag_eval_lib.py (1369 bytes)
import pytest
from examples.rag_eval_lib import (
RAGTriadEvaluator
)
def test_faithfulness_grounded_answer():
evaluator = RAGTriadEvaluator()
context = "The enterprise SLA guarantees 99.99% uptime with 24/7 dedicated support."
answer = "The enterprise SLA guarantees 99.99% uptime."
res = evaluator.evaluate_faithfulness(context, answer)
assert res["faithfulness_score"] == 1.0
assert res["supported_claims"] == 1
def test_faithfulness_hallucinated_claims():
evaluator = RAGTriadEvaluator()
context = "The server runs on port 8080."
answer = "The server runs on port 8080. It also mines bitcoin automatically."
res = evaluator.evaluate_faithfulness(context, answer)
assert res["faithfulness_score"] == 0.5
assert res["total_claims"] == 2
assert res["supported_claims"] == 1
def test_context_relevance_calculation():
evaluator = RAGTriadEvaluator()
query = "database connection timeout"
context = "Database connection timeout is 30 seconds. Weather is sunny today. Cats are mammals."
score = evaluator.evaluate_context_relevance(query, context)
assert round(score, 2) == 0.33
def test_empty_string_handling():
evaluator = RAGTriadEvaluator()
assert evaluator.evaluate_faithfulness("", "")["faithfulness_score"] == 1.0
assert evaluator.evaluate_context_relevance("", "") == 0.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 265 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_rag_eval_lib.py (1369 bytes)
import pytest
from examples.rag_eval_lib import (
RAGTriadEvaluator
)
def test_faithfulness_grounded_answer():
evaluator = RAGTriadEvaluator()
context = "The enterprise SLA guarantees 99.99% uptime with 24/7 dedicated support."
answer = "The enterprise SLA guarantees 99.99% uptime."
res = evaluator.evaluate_faithfulness(context, answer)
assert res["faithfulness_score"] == 1.0
assert res["supported_claims"] == 1
def test_faithfulness_hallucinated_claims():
evaluator = RAGTriadEvaluator()
context = "The server runs on port 8080."
answer = "The server runs on port 8080. It also mines bitcoin automatically."
res = evaluator.evaluate_faithfulness(context, answer)
assert res["faithfulness_score"] == 0.5
assert res["total_claims"] == 2
assert res["supported_claims"] == 1
def test_context_relevance_calculation():
evaluator = RAGTriadEvaluator()
query = "database connection timeout"
context = "Database connection timeout is 30 seconds. Weather is sunny today. Cats are mammals."
score = evaluator.evaluate_context_relevance(query, context)
assert round(score, 2) == 0.33
def test_empty_string_handling():
evaluator = RAGTriadEvaluator()
assert evaluator.evaluate_faithfulness("", "")["faithfulness_score"] == 1.0
assert evaluator.evaluate_context_relevance("", "") == 0.0
Troubleshooting
Troubleshooting: Day 265 - Evaluating RAG with RAGAS and TruLens
Common Issues
- Empty sentence arrays:
- Cause: Consecutive delimiters in text.
- Fix: Filter out empty strings after regex splitting.
Security notes
Security & Privacy: Day 265 - Evaluating RAG with RAGAS and TruLens
Security Guidance
- Do not store unredacted customer PII in persistent evaluation test benchmarks.