AI Engineering: Agents and Applications › Evaluation and Reliability › Day 312
Hands-on lab — Day 312: Regression Testing for Prompts and Models
- ← Back to the Day 312 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-312-regression-testing-for-prompts-and-models/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/regression_runner.py Test
bash tests/run_tests.sh File tree
examples/regression_runner.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/regression_runner.py tests/run_tests.sh tests/test_regression_runner.py troubleshooting.md
Lab README
Day 312 Lab: Regression Testing for Prompts and Models
Lesson
- Lesson title: Regression Testing for Prompts and Models
- Day number: 312 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-312-regression-testing-for-prompts-and-models
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-312-regression-testing-for-prompts-and-modelswhen the site is running.
Purpose
Build an Automated AI Regression Test Runner in Python that compares candidate benchmark evaluation metrics against baseline benchmarks and enforces CI/CD deployment gates.
Learning objectives
- Calculate metric deltas between baseline and candidate evaluation runs.
- Enforce configurable tolerance thresholds and schema validity invariants.
- Check for failures on critical golden invariant test cases.
- Generate structured pass/fail decisions for CI automation.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/regression_runner.py: Starter implementation skeletonexamples/regression_runner.py: Verified reference implementationtests/test_regression_runner.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/regression_runner.py
What the commands do
- Evaluates regression deltas and outputs CI gate status reports.
Expected output
Report: {'gate_passed': True, 'baseline_accuracy': 0.9, 'candidate_accuracy': 0.92, 'accuracy_delta': 0.02, 'schema_validity_passed': True, 'golden_invariants_passed': True, 'failed_golden_count': 0, 'status': 'APPROVED'}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Approved gate for improved candidate accuracy
- Approved gate for slight drop within tolerance threshold
- Rejected gate for severe accuracy regression exceeding tolerance
- Rejected gate for broken schema validity (even with high accuracy)
- Rejected gate when critical golden invariant cases fail
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure accuracy values are between 0.0 and 1.0.
Security notes
Runs completely offline on local CPU.
Extension exercises
Generate formatted markdown summary comments for GitHub Actions.
Navigation
Day number: 312 of 365
Expected output
FIELDS.md
# Measured Values
- `regressions_evaluated`: 5
- `gates_passed`: 1
examples-run.txt
Report: {'gate_passed': True, 'baseline_accuracy': 0.9, 'candidate_accuracy': 0.92, 'accuracy_delta': 0.02, 'schema_validity_passed': True, 'golden_invariants_passed': True, 'failed_golden_count': 0, 'status': 'APPROVED'}
measured-values.txt
regressions_evaluated=5
gates_passed=1
run-output.txt
Report: {'gate_passed': True, 'baseline_accuracy': 0.9, 'candidate_accuracy': 0.92, 'accuracy_delta': 0.02, 'schema_validity_passed': True, 'golden_invariants_passed': True, 'failed_golden_count': 0, 'status': 'APPROVED'}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.regression_runner.RegressionTestRunner object at 0x106f6e780>
baseline_results = {'accuracy': 0.85, 'schema_validity': 1.0, 'failed_golden_cases': []}
candidate_results = {'accuracy': 0.9, 'schema_validity': 1.0, 'failed_golden_cases': ['case_auth_01']}
def evaluate_regression(self, baseline_results: Dict[str, Any], candidate_results: Dict[str, Any]) -> Dict[str, Any]:
> raise NotImplementedError('TASK 1: implement evaluate_regression.')
E NotImplementedError: TASK 1: implement evaluate_regression.
examples/regression_runner.py:16: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_regression_runner.py::test_regression_gate_approved_on_improvement
FAILED tests/test_regression_runner.py::test_regression_gate_approved_within_tolerance
FAILED tests/test_regression_runner.py::test_regression_gate_rejected_on_severe_accuracy_drop
FAILED tests/test_regression_runner.py::test_regression_gate_rejected_on_schema_invalidity
FAILED tests/test_regression_runner.py::test_regression_gate_rejected_on_golden_failure
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-312-regression-testing-for-prompts-and-models
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_regression_runner.py::test_regression_gate_approved_on_improvement PASSED [ 20%]
tests/test_regression_runner.py::test_regression_gate_approved_within_tolerance PASSED [ 40%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_severe_accuracy_drop PASSED [ 60%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_schema_invalidity PASSED [ 80%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_golden_failure PASSED [100%]
============================== 5 passed in 0.01s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-312-regression-testing-for-prompts-and-models
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_regression_runner.py::test_regression_gate_approved_on_improvement PASSED [ 20%]
tests/test_regression_runner.py::test_regression_gate_approved_within_tolerance PASSED [ 40%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_severe_accuracy_drop PASSED [ 60%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_schema_invalidity PASSED [ 80%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_golden_failure PASSED [100%]
============================== 5 passed in 0.01s ===============================
Source files
examples/regression_runner.py (1800 bytes)
from typing import Dict, Any, List
class RegressionTestRunner:
def __init__(self, tolerance_delta: float = -0.02):
self.tolerance_delta = tolerance_delta
def evaluate_regression(
self,
baseline_results: Dict[str, Any],
candidate_results: Dict[str, Any]
) -> Dict[str, Any]:
base_accuracy = float(baseline_results.get("accuracy", 0.0))
cand_accuracy = float(candidate_results.get("accuracy", 0.0))
delta = round(cand_accuracy - base_accuracy, 4)
base_schema_validity = float(baseline_results.get("schema_validity", 1.0))
cand_schema_validity = float(candidate_results.get("schema_validity", 1.0))
passed_schema = cand_schema_validity >= 1.0
passed_tolerance = delta >= self.tolerance_delta
cand_failed_golden = candidate_results.get("failed_golden_cases", [])
passed_golden = len(cand_failed_golden) == 0
gate_passed = passed_schema and passed_tolerance and passed_golden
return {
"gate_passed": gate_passed,
"baseline_accuracy": base_accuracy,
"candidate_accuracy": cand_accuracy,
"accuracy_delta": delta,
"schema_validity_passed": passed_schema,
"golden_invariants_passed": passed_golden,
"failed_golden_count": len(cand_failed_golden),
"status": "APPROVED" if gate_passed else "REJECTED"
}
if __name__ == "__main__":
runner = RegressionTestRunner(tolerance_delta=-0.02)
base = {"accuracy": 0.90, "schema_validity": 1.0, "failed_golden_cases": []}
cand = {"accuracy": 0.92, "schema_validity": 1.0, "failed_golden_cases": []}
print("Report:", runner.evaluate_regression(base, cand))
metadata.yml (404 bytes)
lesson_id: D312
day: 312
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/regression_runner.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/regression_runner.py (948 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
from typing import Dict, Any, List
class RegressionTestRunner:
def __init__(self, tolerance_delta: float=-0.02):
self.tolerance_delta = tolerance_delta
def evaluate_regression(self, baseline_results: Dict[str, Any], candidate_results: Dict[str, Any]) -> Dict[str, Any]:
raise NotImplementedError('TASK 1: implement evaluate_regression.')
if __name__ == '__main__':
runner = RegressionTestRunner(tolerance_delta=-0.02)
base = {'accuracy': 0.9, 'schema_validity': 1.0, 'failed_golden_cases': []}
cand = {'accuracy': 0.92, 'schema_validity': 1.0, 'failed_golden_cases': []}
print('Report:', runner.evaluate_regression(base, cand))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_regression_runner.py (2372 bytes)
import pytest
from examples.regression_runner import RegressionTestRunner
def test_regression_gate_approved_on_improvement():
runner = RegressionTestRunner(tolerance_delta=-0.02)
base = {"accuracy": 0.85, "schema_validity": 1.0, "failed_golden_cases": []}
cand = {"accuracy": 0.89, "schema_validity": 1.0, "failed_golden_cases": []}
res = runner.evaluate_regression(base, cand)
assert res["gate_passed"] is True
assert res["status"] == "APPROVED"
assert res["accuracy_delta"] == 0.04
def test_regression_gate_approved_within_tolerance():
runner = RegressionTestRunner(tolerance_delta=-0.02)
base = {"accuracy": 0.90, "schema_validity": 1.0, "failed_golden_cases": []}
cand = {"accuracy": 0.89, "schema_validity": 1.0, "failed_golden_cases": []}
res = runner.evaluate_regression(base, cand)
assert res["gate_passed"] is True
assert res["status"] == "APPROVED"
assert res["accuracy_delta"] == -0.01
def test_regression_gate_rejected_on_severe_accuracy_drop():
runner = RegressionTestRunner(tolerance_delta=-0.02)
base = {"accuracy": 0.90, "schema_validity": 1.0, "failed_golden_cases": []}
cand = {"accuracy": 0.84, "schema_validity": 1.0, "failed_golden_cases": []}
res = runner.evaluate_regression(base, cand)
assert res["gate_passed"] is False
assert res["status"] == "REJECTED"
assert res["accuracy_delta"] == -0.06
def test_regression_gate_rejected_on_schema_invalidity():
runner = RegressionTestRunner(tolerance_delta=-0.02)
base = {"accuracy": 0.80, "schema_validity": 1.0, "failed_golden_cases": []}
cand = {"accuracy": 0.95, "schema_validity": 0.90, "failed_golden_cases": []}
res = runner.evaluate_regression(base, cand)
assert res["gate_passed"] is False
assert res["schema_validity_passed"] is False
assert res["status"] == "REJECTED"
def test_regression_gate_rejected_on_golden_failure():
runner = RegressionTestRunner(tolerance_delta=-0.02)
base = {"accuracy": 0.85, "schema_validity": 1.0, "failed_golden_cases": []}
cand = {"accuracy": 0.90, "schema_validity": 1.0, "failed_golden_cases": ["case_auth_01"]}
res = runner.evaluate_regression(base, cand)
assert res["gate_passed"] is False
assert res["golden_invariants_passed"] is False
assert res["failed_golden_count"] == 1
assert res["status"] == "REJECTED"
Troubleshooting
Troubleshooting - Day 312
Verify accuracy numbers are normalized floats between 0.0 and 1.0.
Security notes
Security - Day 312
Regression test suites must isolate benchmark secrets and API keys.