AI Engineering: Agents and Applications › Evaluation and Reliability › Day 315

Hands-on lab — Day 315: An Evaluation Harness

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/eval_harness.py

Test

bash tests/run_tests.sh

File tree

examples/eval_harness.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/eval_harness.py
tests/run_tests.sh
tests/test_eval_harness.py
troubleshooting.md

Lab README

Day 315 Lab: An Evaluation Harness

Lesson

  • Lesson title: An Evaluation Harness
  • Day number: 315 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-315-an-evaluation-harness
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-315-an-evaluation-harness when the site is running.

Purpose

Build a Unified End-to-End AI Evaluation Harness in Python supporting multi-metric scoring (Exact Match and Overlap F1), composite score aggregation, golden invariant tracking, and CI regression gate reporting.

Learning objectives

  • Implement multi-metric scoring across test predictions.
  • Calculate composite weighted scores.
  • Track critical golden invariant test cases.
  • Compute metric deltas against pinned baselines and enforce CI deployment gates.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/eval_harness.py: Starter implementation skeleton
  • examples/eval_harness.py: Verified reference implementation
  • tests/test_eval_harness.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/eval_harness.py

What the commands do

  • Executes batch evaluation scoring and outputs CI gate status reports.

Expected output

Report: {'total_cases': 1, 'baseline_score': 0.8, 'candidate_score': 1.0, 'delta': 0.2, 'failed_golden_cases': [], 'gate_passed': True, 'status': 'APPROVED'}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Single case exact match and overlap scoring
  • Composite score aggregation across batch runs
  • Approval on positive delta improvement
  • Rejection on severe accuracy regressions exceeding tolerance
  • Rejection when critical golden invariant cases fail

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure test cases specify is_golden appropriately.

Security notes

Runs completely offline on local CPU.

Extension exercises

Add category stratification breakdown to the final report.

Day number: 315 of 365

Expected output

FIELDS.md

# Measured Values
- `cases_evaluated`: 20
- `composite_score`: 0.925

examples-run.txt

Report: {'total_cases': 1, 'baseline_score': 0.8, 'candidate_score': 1.0, 'delta': 0.2, 'failed_golden_cases': [], 'gate_passed': True, 'status': 'APPROVED'}

measured-values.txt

cases_evaluated=20
composite_score=0.925

run-output.txt

Report: {'total_cases': 1, 'baseline_score': 0.8, 'candidate_score': 1.0, 'delta': 0.2, 'failed_golden_cases': [], 'gate_passed': True, 'status': 'APPROVED'}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

self = <examples.eval_harness.CompleteEvalHarness object at 0x1094528b0>
case_id = 'golden_01', category = 'happy_path', prediction = 'Wrong Answer'
ground_truth = 'Correct Ground Truth', is_golden = True

    def evaluate_case(self, case_id: str, category: str, prediction: str, ground_truth: str, is_golden: bool=False) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 1: implement evaluate_case.')
E       NotImplementedError: TASK 1: implement evaluate_case.

examples/eval_harness.py:19: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_eval_harness.py::test_evaluate_case_perfect_match - NotImpl...
FAILED tests/test_eval_harness.py::test_evaluate_case_partial_match - NotImpl...
FAILED tests/test_eval_harness.py::test_report_approved_on_improvement - NotI...
FAILED tests/test_eval_harness.py::test_report_rejected_on_regression - NotIm...
FAILED tests/test_eval_harness.py::test_report_rejected_on_golden_failure - N...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-315-an-evaluation-harness
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_eval_harness.py::test_evaluate_case_perfect_match PASSED      [ 20%]
tests/test_eval_harness.py::test_evaluate_case_partial_match PASSED      [ 40%]
tests/test_eval_harness.py::test_report_approved_on_improvement PASSED   [ 60%]
tests/test_eval_harness.py::test_report_rejected_on_regression PASSED    [ 80%]
tests/test_eval_harness.py::test_report_rejected_on_golden_failure PASSED [100%]

============================== 5 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-315-an-evaluation-harness
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_eval_harness.py::test_evaluate_case_perfect_match PASSED      [ 20%]
tests/test_eval_harness.py::test_evaluate_case_partial_match PASSED      [ 40%]
tests/test_eval_harness.py::test_report_approved_on_improvement PASSED   [ 60%]
tests/test_eval_harness.py::test_report_rejected_on_regression PASSED    [ 80%]
tests/test_eval_harness.py::test_report_rejected_on_golden_failure PASSED [100%]

============================== 5 passed in 0.01s ===============================

Source files

examples/eval_harness.py (2559 bytes)
import json
from typing import List, Dict, Any, Optional

class CompleteEvalHarness:
    def __init__(self, baseline_composite_score: float = 0.85, tolerance_delta: float = -0.02):
        self.baseline_score = float(baseline_composite_score)
        self.tolerance_delta = float(tolerance_delta)
        self.results: List[Dict[str, Any]] = []
        
    def evaluate_case(self, case_id: str, category: str, prediction: str, ground_truth: str, is_golden: bool = False) -> Dict[str, Any]:
        norm_pred = str(prediction).strip().lower()
        norm_gt = str(ground_truth).strip().lower()
        
        exact_match = 1.0 if norm_pred == norm_gt else 0.0
        
        pred_words = set(norm_pred.split())
        gt_words = set(norm_gt.split())
        common = pred_words.intersection(gt_words)
        
        if pred_words and gt_words and common:
            p = len(common) / len(pred_words)
            r = len(common) / len(gt_words)
            overlap_f1 = (2 * p * r) / (p + r)
        else:
            overlap_f1 = 0.0
            
        score = (0.5 * exact_match) + (0.5 * overlap_f1)
        passed = score >= 0.70
        
        result = {
            "case_id": case_id,
            "category": category,
            "score": round(score, 4),
            "passed": passed,
            "is_golden": is_golden
        }
        self.results.append(result)
        return result
        
    def generate_report(self) -> Dict[str, Any]:
        if not self.results:
            return {"status": "EMPTY", "gate_passed": False}
            
        total_score = sum(r["score"] for r in self.results)
        candidate_composite = round(total_score / len(self.results), 4)
        delta = round(candidate_composite - self.baseline_score, 4)
        
        failed_golden = [r["case_id"] for r in self.results if r["is_golden"] and not r["passed"]]
        
        gate_passed = (delta >= self.tolerance_delta) and (len(failed_golden) == 0)
        
        return {
            "total_cases": len(self.results),
            "baseline_score": self.baseline_score,
            "candidate_score": candidate_composite,
            "delta": delta,
            "failed_golden_cases": failed_golden,
            "gate_passed": gate_passed,
            "status": "APPROVED" if gate_passed else "REJECTED"
        }

if __name__ == "__main__":
    harness = CompleteEvalHarness(baseline_composite_score=0.80)
    harness.evaluate_case("c1", "happy_path", "Paris", "Paris", is_golden=True)
    print("Report:", harness.generate_report())
metadata.yml (399 bytes)
lesson_id: D315
day: 315
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/eval_harness.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/eval_harness.py (1166 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import json
from typing import List, Dict, Any, Optional

class CompleteEvalHarness:

    def __init__(self, baseline_composite_score: float=0.85, tolerance_delta: float=-0.02):
        self.baseline_score = float(baseline_composite_score)
        self.tolerance_delta = float(tolerance_delta)
        self.results: List[Dict[str, Any]] = []

    def evaluate_case(self, case_id: str, category: str, prediction: str, ground_truth: str, is_golden: bool=False) -> Dict[str, Any]:
        raise NotImplementedError('TASK 1: implement evaluate_case.')

    def generate_report(self) -> Dict[str, Any]:
        raise NotImplementedError('TASK 2: implement generate_report.')
if __name__ == '__main__':
    harness = CompleteEvalHarness(baseline_composite_score=0.8)
    harness.evaluate_case('c1', 'happy_path', 'Paris', 'Paris', is_golden=True)
    print('Report:', harness.generate_report())
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_eval_harness.py (1904 bytes)
import pytest
from examples.eval_harness import CompleteEvalHarness

def test_evaluate_case_perfect_match():
    harness = CompleteEvalHarness()
    res = harness.evaluate_case("c1", "happy_path", "San Francisco", "San Francisco", is_golden=False)
    assert res["score"] == 1.0
    assert res["passed"] is True

def test_evaluate_case_partial_match():
    harness = CompleteEvalHarness()
    res = harness.evaluate_case("c2", "happy_path", "The capital is Paris", "Paris is the capital", is_golden=False)
    assert res["score"] > 0.4
    assert len(harness.results) == 1

def test_report_approved_on_improvement():
    harness = CompleteEvalHarness(baseline_composite_score=0.80, tolerance_delta=-0.02)
    harness.evaluate_case("c1", "happy_path", "Exact", "Exact", is_golden=True)
    harness.evaluate_case("c2", "happy_path", "Exact", "Exact", is_golden=False)
    report = harness.generate_report()
    assert report["gate_passed"] is True
    assert report["status"] == "APPROVED"
    assert report["delta"] == 0.20

def test_report_rejected_on_regression():
    harness = CompleteEvalHarness(baseline_composite_score=0.95, tolerance_delta=-0.02)
    harness.evaluate_case("c1", "happy_path", "Wrong", "Right", is_golden=False)
    report = harness.generate_report()
    assert report["gate_passed"] is False
    assert report["status"] == "REJECTED"
    assert report["delta"] < -0.02

def test_report_rejected_on_golden_failure():
    harness = CompleteEvalHarness(baseline_composite_score=0.40, tolerance_delta=-0.02)
    harness.evaluate_case("golden_01", "happy_path", "Wrong Answer", "Correct Ground Truth", is_golden=True)
    harness.evaluate_case("normal_02", "happy_path", "Exact", "Exact", is_golden=False)
    report = harness.generate_report()
    assert report["gate_passed"] is False
    assert "golden_01" in report["failed_golden_cases"]
    assert report["status"] == "REJECTED"

Troubleshooting

Troubleshooting - Day 315\nEnsure score weighting fractions sum up to 1.0.\n

Security notes

Security - Day 315\nEvaluation harnesses execute in isolated environments with zero production write permissions.\n