Capstone › Capstone Build I: Foundation › Day 356
Hands-on lab — Day 356: Tests and Evals for Your Capstone
- ← Back to the Day 356 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/capstone/day-356-tests-and-evals-for-your-capstone/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/evals.py Test
bash tests/run_tests.sh File tree
examples/evals.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/evals.py tests/run_tests.sh tests/test_evals.py troubleshooting.md
Lab README
Day 356 Lab: Tests and Evals for Your Capstone
Lesson
- Lesson title: Tests and Evals for Your Capstone
- Day number: 356 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-356-tests-and-evals-for-your-capstone
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-356-tests-and-evals-for-your-capstonewhen the site is running.
Purpose
Build a Capstone Automated Evaluation Platform in Python that calculates RAG Triad metrics (Faithfulness, Context Recall) and enforces regression quality gates.
Learning objectives
- Formulate a golden evaluation benchmark dataset.
- Calculate Faithfulness using atomic claim verification.
- Score Context Recall against ground-truth points.
- Assert CI/CD quality gate compliance.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/evals.py: Starter implementation skeletonexamples/evals.py: Verified reference implementationtests/test_evals.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/evals.py
What the commands do
- Evaluates benchmark queries across Faithfulness and Context Recall metrics and outputs scorecard results.
Expected output
{'total_benchmark_queries': 1, 'average_faithfulness': 1.0, 'average_context_recall': 1.0, 'pass_rate_percentage': 100.0, 'overall_quality_gate': 'PASSED', 'detailed_results': [{'query_id': 'q1', 'faithfulness': 1.0, 'context_recall': 1.0, 'unsupported_claims': [], 'status': 'PASS'}]}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Perfect 1.0 Faithfulness on grounded claims
- Hallucination detection lowering Faithfulness score
- Context Recall scoring against ground-truth points
- Overall benchmark suite pass rate calculation
- Quality gate failure on low accuracy
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure claim strings contain alphanumeric tokens for matching.
Security notes
Runs locally with zero external network calls.
Extension exercises
Implement an Answer Relevancy scoring function.
Navigation
Day number: 356 of 365
Expected output
FIELDS.md
# Measured Values
- `faithfulness_score`: 0.95
- `context_recall_score`: 0.92
examples-run.txt
{'total_benchmark_queries': 1, 'average_faithfulness': 1.0, 'average_context_recall': 1.0, 'pass_rate_percentage': 100.0, 'overall_quality_gate': 'PASSED', 'detailed_results': [{'query_id': 'q1', 'faithfulness': 1.0, 'context_recall': 1.0, 'unsupported_claims': [], 'status': 'PASS'}]}
measured-values.txt
faithfulness_score=0.95
context_recall_score=0.92
run-output.txt
{'total_benchmark_queries': 1, 'average_faithfulness': 1.0, 'average_context_recall': 1.0, 'pass_rate_percentage': 100.0, 'overall_quality_gate': 'PASSED', 'detailed_results': [{'query_id': 'q1', 'faithfulness': 1.0, 'context_recall': 1.0, 'unsupported_claims': [], 'status': 'PASS'}]}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.evals.CapstoneEvaluationEngine object at 0x106fd2780>
benchmark_items = [{'id': 'q1', 'ground_truth_points': ['Expected fact A'], 'retrieved_context': 'Completely unrelated context text.', 'answer_claims': ['Hallucinated claim X']}]
def run_eval_suite(self, benchmark_items: List[Dict[str, Any]]) -> Dict[str, Any]:
> raise NotImplementedError('TASK 4: implement run_eval_suite.')
E NotImplementedError: TASK 4: implement run_eval_suite.
examples/evals.py:27: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_evals.py::test_perfect_faithfulness_calculation - NotImplem...
FAILED tests/test_evals.py::test_hallucination_detection_lowers_faithfulness
FAILED tests/test_evals.py::test_context_recall_calculation - NotImplementedE...
FAILED tests/test_evals.py::test_benchmark_suite_pass_gate - NotImplementedEr...
FAILED tests/test_evals.py::test_benchmark_suite_failure_gate - NotImplemente...
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/capstone/day-356-tests-and-evals-for-your-capstone
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_evals.py::test_perfect_faithfulness_calculation PASSED [ 20%]
tests/test_evals.py::test_hallucination_detection_lowers_faithfulness PASSED [ 40%]
tests/test_evals.py::test_context_recall_calculation PASSED [ 60%]
tests/test_evals.py::test_benchmark_suite_pass_gate PASSED [ 80%]
tests/test_evals.py::test_benchmark_suite_failure_gate PASSED [100%]
============================== 5 passed in 0.01s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/capstone/day-356-tests-and-evals-for-your-capstone
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_evals.py::test_perfect_faithfulness_calculation PASSED [ 20%]
tests/test_evals.py::test_hallucination_detection_lowers_faithfulness PASSED [ 40%]
tests/test_evals.py::test_context_recall_calculation PASSED [ 60%]
tests/test_evals.py::test_benchmark_suite_pass_gate PASSED [ 80%]
tests/test_evals.py::test_benchmark_suite_failure_gate PASSED [100%]
============================== 5 passed in 0.01s ===============================
Source files
examples/evals.py (3310 bytes)
import json
from typing import List, Dict, Any, Tuple
class CapstoneEvaluationEngine:
def __init__(self, faithfulness_threshold: float = 0.90, recall_threshold: float = 0.85):
self.faithfulness_threshold = faithfulness_threshold
self.recall_threshold = recall_threshold
def calculate_faithfulness(self, answer_claims: List[str], context_text: str) -> Tuple[float, List[str]]:
supported = []
unsupported = []
context_lower = context_text.lower()
for claim in answer_claims:
claim_tokens = [tok for tok in claim.lower().split() if len(tok) > 3]
matches = sum(1 for tok in claim_tokens if tok in context_lower)
if len(claim_tokens) > 0 and (matches / len(claim_tokens)) >= 0.6:
supported.append(claim)
else:
unsupported.append(claim)
score = len(supported) / (len(answer_claims) or 1.0)
return round(score, 3), unsupported
def calculate_context_recall(self, ground_truth_points: List[str], context_text: str) -> float:
found = 0
context_lower = context_text.lower()
for point in ground_truth_points:
pt_tokens = [tok for tok in point.lower().split() if len(tok) > 3]
matches = sum(1 for tok in pt_tokens if tok in context_lower)
if len(pt_tokens) > 0 and (matches / len(pt_tokens)) >= 0.6:
found += 1
return round(found / (len(ground_truth_points) or 1.0), 3)
def evaluate_benchmark_item(self, item: Dict[str, Any]) -> Dict[str, Any]:
faithfulness, unsupp = self.calculate_faithfulness(item["answer_claims"], item["retrieved_context"])
recall = self.calculate_context_recall(item["ground_truth_points"], item["retrieved_context"])
passed = (faithfulness >= self.faithfulness_threshold) and (recall >= self.recall_threshold)
return {
"query_id": item["id"],
"faithfulness": faithfulness,
"context_recall": recall,
"unsupported_claims": unsupp,
"status": "PASS" if passed else "FAIL"
}
def run_eval_suite(self, benchmark_items: List[Dict[str, Any]]) -> Dict[str, Any]:
results = [self.evaluate_benchmark_item(item) for item in benchmark_items]
avg_faithfulness = round(sum(r["faithfulness"] for r in results) / len(results), 3)
avg_recall = round(sum(r["context_recall"] for r in results) / len(results), 3)
total_passed = sum(1 for r in results if r["status"] == "PASS")
pass_rate = round((total_passed / len(results)) * 100.0, 1)
return {
"total_benchmark_queries": len(results),
"average_faithfulness": avg_faithfulness,
"average_context_recall": avg_recall,
"pass_rate_percentage": pass_rate,
"overall_quality_gate": "PASSED" if pass_rate >= 80.0 else "FAILED",
"detailed_results": results
}
if __name__ == "__main__":
benchmark = [{
"id": "q1",
"ground_truth_points": ["Uptime is 99.9%"],
"retrieved_context": "The system uptime is 99.9% guaranteed.",
"answer_claims": ["Uptime is 99.9%"]
}]
engine = CapstoneEvaluationEngine()
print(engine.run_eval_suite(benchmark))
metadata.yml (392 bytes)
lesson_id: D356
day: 356
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/evals.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/evals.py (1542 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import json
from typing import List, Dict, Any, Tuple
class CapstoneEvaluationEngine:
def __init__(self, faithfulness_threshold: float=0.9, recall_threshold: float=0.85):
self.faithfulness_threshold = faithfulness_threshold
self.recall_threshold = recall_threshold
def calculate_faithfulness(self, answer_claims: List[str], context_text: str) -> Tuple[float, List[str]]:
raise NotImplementedError('TASK 1: implement calculate_faithfulness.')
def calculate_context_recall(self, ground_truth_points: List[str], context_text: str) -> float:
raise NotImplementedError('TASK 2: implement calculate_context_recall.')
def evaluate_benchmark_item(self, item: Dict[str, Any]) -> Dict[str, Any]:
raise NotImplementedError('TASK 3: implement evaluate_benchmark_item.')
def run_eval_suite(self, benchmark_items: List[Dict[str, Any]]) -> Dict[str, Any]:
raise NotImplementedError('TASK 4: implement run_eval_suite.')
if __name__ == '__main__':
benchmark = [{'id': 'q1', 'ground_truth_points': ['Uptime is 99.9%'], 'retrieved_context': 'The system uptime is 99.9% guaranteed.', 'answer_claims': ['Uptime is 99.9%']}]
engine = CapstoneEvaluationEngine()
print(engine.run_eval_suite(benchmark))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_evals.py (2372 bytes)
import pytest
from examples.evals import CapstoneEvaluationEngine
def test_perfect_faithfulness_calculation():
engine = CapstoneEvaluationEngine()
context = "The product price is $49.99 and includes free shipping."
claims = ["Product price is $49.99", "Includes free shipping"]
score, unsupp = engine.calculate_faithfulness(claims, context)
assert score == 1.0
assert len(unsupp) == 0
def test_hallucination_detection_lowers_faithfulness():
engine = CapstoneEvaluationEngine()
context = "The product price is $49.99."
claims = ["Product price is $49.99", "Free diamond necklace included with every order"]
score, unsupp = engine.calculate_faithfulness(claims, context)
assert score == 0.5
assert len(unsupp) == 1
assert "Free diamond necklace" in unsupp[0]
def test_context_recall_calculation():
engine = CapstoneEvaluationEngine()
context = "Server memory is 64GB DDR5 RAM."
gt = ["Memory is 64GB DDR5", "Storage is 2TB NVMe"]
score = engine.calculate_context_recall(gt, context)
assert score == 0.5
def test_benchmark_suite_pass_gate():
engine = CapstoneEvaluationEngine(faithfulness_threshold=0.90, recall_threshold=0.85)
suite = [
{
"id": "q1",
"ground_truth_points": ["SLA is 99.9%"],
"retrieved_context": "The SLA uptime is 99.9%.",
"answer_claims": ["SLA is 99.9%"]
},
{
"id": "q2",
"ground_truth_points": ["Support is 24/7"],
"retrieved_context": "Customer support is available 24/7.",
"answer_claims": ["Support is 24/7"]
}
]
report = engine.run_eval_suite(suite)
assert report["overall_quality_gate"] == "PASSED"
assert report["pass_rate_percentage"] == 100.0
assert report["average_faithfulness"] == 1.0
def test_benchmark_suite_failure_gate():
engine = CapstoneEvaluationEngine(faithfulness_threshold=0.90, recall_threshold=0.85)
# Suite with 100% hallucinations
suite = [{
"id": "q1",
"ground_truth_points": ["Expected fact A"],
"retrieved_context": "Completely unrelated context text.",
"answer_claims": ["Hallucinated claim X"]
}]
report = engine.run_eval_suite(suite)
assert report["overall_quality_gate"] == "FAILED"
assert report["pass_rate_percentage"] == 0.0