Capstone › Capstone Build I: Foundation › Day 356

Hands-on lab — Day 356: Tests and Evals for Your Capstone

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/evals.py

Test

bash tests/run_tests.sh

File tree

examples/evals.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/evals.py
tests/run_tests.sh
tests/test_evals.py
troubleshooting.md

Lab README

Day 356 Lab: Tests and Evals for Your Capstone

Lesson

Purpose

Build a Capstone Automated Evaluation Platform in Python that calculates RAG Triad metrics (Faithfulness, Context Recall) and enforces regression quality gates.

Learning objectives

  • Formulate a golden evaluation benchmark dataset.
  • Calculate Faithfulness using atomic claim verification.
  • Score Context Recall against ground-truth points.
  • Assert CI/CD quality gate compliance.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/evals.py: Starter implementation skeleton
  • examples/evals.py: Verified reference implementation
  • tests/test_evals.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/evals.py

What the commands do

  • Evaluates benchmark queries across Faithfulness and Context Recall metrics and outputs scorecard results.

Expected output

{'total_benchmark_queries': 1, 'average_faithfulness': 1.0, 'average_context_recall': 1.0, 'pass_rate_percentage': 100.0, 'overall_quality_gate': 'PASSED', 'detailed_results': [{'query_id': 'q1', 'faithfulness': 1.0, 'context_recall': 1.0, 'unsupported_claims': [], 'status': 'PASS'}]}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Perfect 1.0 Faithfulness on grounded claims
  • Hallucination detection lowering Faithfulness score
  • Context Recall scoring against ground-truth points
  • Overall benchmark suite pass rate calculation
  • Quality gate failure on low accuracy

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure claim strings contain alphanumeric tokens for matching.

Security notes

Runs locally with zero external network calls.

Extension exercises

Implement an Answer Relevancy scoring function.

Day number: 356 of 365

Expected output

FIELDS.md

# Measured Values
- `faithfulness_score`: 0.95
- `context_recall_score`: 0.92

examples-run.txt

{'total_benchmark_queries': 1, 'average_faithfulness': 1.0, 'average_context_recall': 1.0, 'pass_rate_percentage': 100.0, 'overall_quality_gate': 'PASSED', 'detailed_results': [{'query_id': 'q1', 'faithfulness': 1.0, 'context_recall': 1.0, 'unsupported_claims': [], 'status': 'PASS'}]}

measured-values.txt

faithfulness_score=0.95
context_recall_score=0.92

run-output.txt

{'total_benchmark_queries': 1, 'average_faithfulness': 1.0, 'average_context_recall': 1.0, 'pass_rate_percentage': 100.0, 'overall_quality_gate': 'PASSED', 'detailed_results': [{'query_id': 'q1', 'faithfulness': 1.0, 'context_recall': 1.0, 'unsupported_claims': [], 'status': 'PASS'}]}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


self = <examples.evals.CapstoneEvaluationEngine object at 0x106fd2780>
benchmark_items = [{'id': 'q1', 'ground_truth_points': ['Expected fact A'], 'retrieved_context': 'Completely unrelated context text.', 'answer_claims': ['Hallucinated claim X']}]

    def run_eval_suite(self, benchmark_items: List[Dict[str, Any]]) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 4: implement run_eval_suite.')
E       NotImplementedError: TASK 4: implement run_eval_suite.

examples/evals.py:27: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_evals.py::test_perfect_faithfulness_calculation - NotImplem...
FAILED tests/test_evals.py::test_hallucination_detection_lowers_faithfulness
FAILED tests/test_evals.py::test_context_recall_calculation - NotImplementedE...
FAILED tests/test_evals.py::test_benchmark_suite_pass_gate - NotImplementedEr...
FAILED tests/test_evals.py::test_benchmark_suite_failure_gate - NotImplemente...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/capstone/day-356-tests-and-evals-for-your-capstone
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_evals.py::test_perfect_faithfulness_calculation PASSED        [ 20%]
tests/test_evals.py::test_hallucination_detection_lowers_faithfulness PASSED [ 40%]
tests/test_evals.py::test_context_recall_calculation PASSED              [ 60%]
tests/test_evals.py::test_benchmark_suite_pass_gate PASSED               [ 80%]
tests/test_evals.py::test_benchmark_suite_failure_gate PASSED            [100%]

============================== 5 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/capstone/day-356-tests-and-evals-for-your-capstone
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_evals.py::test_perfect_faithfulness_calculation PASSED        [ 20%]
tests/test_evals.py::test_hallucination_detection_lowers_faithfulness PASSED [ 40%]
tests/test_evals.py::test_context_recall_calculation PASSED              [ 60%]
tests/test_evals.py::test_benchmark_suite_pass_gate PASSED               [ 80%]
tests/test_evals.py::test_benchmark_suite_failure_gate PASSED            [100%]

============================== 5 passed in 0.01s ===============================

Source files

examples/evals.py (3310 bytes)
import json
from typing import List, Dict, Any, Tuple

class CapstoneEvaluationEngine:
    def __init__(self, faithfulness_threshold: float = 0.90, recall_threshold: float = 0.85):
        self.faithfulness_threshold = faithfulness_threshold
        self.recall_threshold = recall_threshold

    def calculate_faithfulness(self, answer_claims: List[str], context_text: str) -> Tuple[float, List[str]]:
        supported = []
        unsupported = []
        context_lower = context_text.lower()

        for claim in answer_claims:
            claim_tokens = [tok for tok in claim.lower().split() if len(tok) > 3]
            matches = sum(1 for tok in claim_tokens if tok in context_lower)
            if len(claim_tokens) > 0 and (matches / len(claim_tokens)) >= 0.6:
                supported.append(claim)
            else:
                unsupported.append(claim)

        score = len(supported) / (len(answer_claims) or 1.0)
        return round(score, 3), unsupported

    def calculate_context_recall(self, ground_truth_points: List[str], context_text: str) -> float:
        found = 0
        context_lower = context_text.lower()
        for point in ground_truth_points:
            pt_tokens = [tok for tok in point.lower().split() if len(tok) > 3]
            matches = sum(1 for tok in pt_tokens if tok in context_lower)
            if len(pt_tokens) > 0 and (matches / len(pt_tokens)) >= 0.6:
                found += 1
        return round(found / (len(ground_truth_points) or 1.0), 3)

    def evaluate_benchmark_item(self, item: Dict[str, Any]) -> Dict[str, Any]:
        faithfulness, unsupp = self.calculate_faithfulness(item["answer_claims"], item["retrieved_context"])
        recall = self.calculate_context_recall(item["ground_truth_points"], item["retrieved_context"])
        
        passed = (faithfulness >= self.faithfulness_threshold) and (recall >= self.recall_threshold)
        return {
            "query_id": item["id"],
            "faithfulness": faithfulness,
            "context_recall": recall,
            "unsupported_claims": unsupp,
            "status": "PASS" if passed else "FAIL"
        }

    def run_eval_suite(self, benchmark_items: List[Dict[str, Any]]) -> Dict[str, Any]:
        results = [self.evaluate_benchmark_item(item) for item in benchmark_items]
        avg_faithfulness = round(sum(r["faithfulness"] for r in results) / len(results), 3)
        avg_recall = round(sum(r["context_recall"] for r in results) / len(results), 3)
        total_passed = sum(1 for r in results if r["status"] == "PASS")
        pass_rate = round((total_passed / len(results)) * 100.0, 1)

        return {
            "total_benchmark_queries": len(results),
            "average_faithfulness": avg_faithfulness,
            "average_context_recall": avg_recall,
            "pass_rate_percentage": pass_rate,
            "overall_quality_gate": "PASSED" if pass_rate >= 80.0 else "FAILED",
            "detailed_results": results
        }

if __name__ == "__main__":
    benchmark = [{
        "id": "q1",
        "ground_truth_points": ["Uptime is 99.9%"],
        "retrieved_context": "The system uptime is 99.9% guaranteed.",
        "answer_claims": ["Uptime is 99.9%"]
    }]
    engine = CapstoneEvaluationEngine()
    print(engine.run_eval_suite(benchmark))
metadata.yml (392 bytes)
lesson_id: D356
day: 356
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/evals.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/evals.py (1542 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import json
from typing import List, Dict, Any, Tuple

class CapstoneEvaluationEngine:

    def __init__(self, faithfulness_threshold: float=0.9, recall_threshold: float=0.85):
        self.faithfulness_threshold = faithfulness_threshold
        self.recall_threshold = recall_threshold

    def calculate_faithfulness(self, answer_claims: List[str], context_text: str) -> Tuple[float, List[str]]:
        raise NotImplementedError('TASK 1: implement calculate_faithfulness.')

    def calculate_context_recall(self, ground_truth_points: List[str], context_text: str) -> float:
        raise NotImplementedError('TASK 2: implement calculate_context_recall.')

    def evaluate_benchmark_item(self, item: Dict[str, Any]) -> Dict[str, Any]:
        raise NotImplementedError('TASK 3: implement evaluate_benchmark_item.')

    def run_eval_suite(self, benchmark_items: List[Dict[str, Any]]) -> Dict[str, Any]:
        raise NotImplementedError('TASK 4: implement run_eval_suite.')
if __name__ == '__main__':
    benchmark = [{'id': 'q1', 'ground_truth_points': ['Uptime is 99.9%'], 'retrieved_context': 'The system uptime is 99.9% guaranteed.', 'answer_claims': ['Uptime is 99.9%']}]
    engine = CapstoneEvaluationEngine()
    print(engine.run_eval_suite(benchmark))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_evals.py (2372 bytes)
import pytest
from examples.evals import CapstoneEvaluationEngine

def test_perfect_faithfulness_calculation():
    engine = CapstoneEvaluationEngine()
    context = "The product price is $49.99 and includes free shipping."
    claims = ["Product price is $49.99", "Includes free shipping"]
    score, unsupp = engine.calculate_faithfulness(claims, context)
    assert score == 1.0
    assert len(unsupp) == 0

def test_hallucination_detection_lowers_faithfulness():
    engine = CapstoneEvaluationEngine()
    context = "The product price is $49.99."
    claims = ["Product price is $49.99", "Free diamond necklace included with every order"]
    score, unsupp = engine.calculate_faithfulness(claims, context)
    assert score == 0.5
    assert len(unsupp) == 1
    assert "Free diamond necklace" in unsupp[0]

def test_context_recall_calculation():
    engine = CapstoneEvaluationEngine()
    context = "Server memory is 64GB DDR5 RAM."
    gt = ["Memory is 64GB DDR5", "Storage is 2TB NVMe"]
    score = engine.calculate_context_recall(gt, context)
    assert score == 0.5

def test_benchmark_suite_pass_gate():
    engine = CapstoneEvaluationEngine(faithfulness_threshold=0.90, recall_threshold=0.85)
    suite = [
        {
            "id": "q1",
            "ground_truth_points": ["SLA is 99.9%"],
            "retrieved_context": "The SLA uptime is 99.9%.",
            "answer_claims": ["SLA is 99.9%"]
        },
        {
            "id": "q2",
            "ground_truth_points": ["Support is 24/7"],
            "retrieved_context": "Customer support is available 24/7.",
            "answer_claims": ["Support is 24/7"]
        }
    ]
    report = engine.run_eval_suite(suite)
    assert report["overall_quality_gate"] == "PASSED"
    assert report["pass_rate_percentage"] == 100.0
    assert report["average_faithfulness"] == 1.0

def test_benchmark_suite_failure_gate():
    engine = CapstoneEvaluationEngine(faithfulness_threshold=0.90, recall_threshold=0.85)
    # Suite with 100% hallucinations
    suite = [{
        "id": "q1",
        "ground_truth_points": ["Expected fact A"],
        "retrieved_context": "Completely unrelated context text.",
        "answer_claims": ["Hallucinated claim X"]
    }]
    report = engine.run_eval_suite(suite)
    assert report["overall_quality_gate"] == "FAILED"
    assert report["pass_rate_percentage"] == 0.0

Troubleshooting

Troubleshooting - Day 356\nVerify ground-truth semantic tokens exist in retrieved context passages.\n

Security notes

Security - Day 356\nEvaluation platform isolates test prompts and verifies factual grounding without data leakage.\n