AI Engineering: Agents and Applications › Evaluation and Reliability › Day 312

Hands-on lab — Day 312: Regression Testing for Prompts and Models

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/regression_runner.py

Test

bash tests/run_tests.sh

File tree

examples/regression_runner.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/regression_runner.py
tests/run_tests.sh
tests/test_regression_runner.py
troubleshooting.md

Lab README

Day 312 Lab: Regression Testing for Prompts and Models

Lesson

Purpose

Build an Automated AI Regression Test Runner in Python that compares candidate benchmark evaluation metrics against baseline benchmarks and enforces CI/CD deployment gates.

Learning objectives

  • Calculate metric deltas between baseline and candidate evaluation runs.
  • Enforce configurable tolerance thresholds and schema validity invariants.
  • Check for failures on critical golden invariant test cases.
  • Generate structured pass/fail decisions for CI automation.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/regression_runner.py: Starter implementation skeleton
  • examples/regression_runner.py: Verified reference implementation
  • tests/test_regression_runner.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/regression_runner.py

What the commands do

  • Evaluates regression deltas and outputs CI gate status reports.

Expected output

Report: {'gate_passed': True, 'baseline_accuracy': 0.9, 'candidate_accuracy': 0.92, 'accuracy_delta': 0.02, 'schema_validity_passed': True, 'golden_invariants_passed': True, 'failed_golden_count': 0, 'status': 'APPROVED'}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Approved gate for improved candidate accuracy
  • Approved gate for slight drop within tolerance threshold
  • Rejected gate for severe accuracy regression exceeding tolerance
  • Rejected gate for broken schema validity (even with high accuracy)
  • Rejected gate when critical golden invariant cases fail

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure accuracy values are between 0.0 and 1.0.

Security notes

Runs completely offline on local CPU.

Extension exercises

Generate formatted markdown summary comments for GitHub Actions.

Day number: 312 of 365

Expected output

FIELDS.md

# Measured Values
- `regressions_evaluated`: 5
- `gates_passed`: 1

examples-run.txt

Report: {'gate_passed': True, 'baseline_accuracy': 0.9, 'candidate_accuracy': 0.92, 'accuracy_delta': 0.02, 'schema_validity_passed': True, 'golden_invariants_passed': True, 'failed_golden_count': 0, 'status': 'APPROVED'}

measured-values.txt

regressions_evaluated=5
gates_passed=1

run-output.txt

Report: {'gate_passed': True, 'baseline_accuracy': 0.9, 'candidate_accuracy': 0.92, 'accuracy_delta': 0.02, 'schema_validity_passed': True, 'golden_invariants_passed': True, 'failed_golden_count': 0, 'status': 'APPROVED'}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

self = <examples.regression_runner.RegressionTestRunner object at 0x106f6e780>
baseline_results = {'accuracy': 0.85, 'schema_validity': 1.0, 'failed_golden_cases': []}
candidate_results = {'accuracy': 0.9, 'schema_validity': 1.0, 'failed_golden_cases': ['case_auth_01']}

    def evaluate_regression(self, baseline_results: Dict[str, Any], candidate_results: Dict[str, Any]) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 1: implement evaluate_regression.')
E       NotImplementedError: TASK 1: implement evaluate_regression.

examples/regression_runner.py:16: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_regression_runner.py::test_regression_gate_approved_on_improvement
FAILED tests/test_regression_runner.py::test_regression_gate_approved_within_tolerance
FAILED tests/test_regression_runner.py::test_regression_gate_rejected_on_severe_accuracy_drop
FAILED tests/test_regression_runner.py::test_regression_gate_rejected_on_schema_invalidity
FAILED tests/test_regression_runner.py::test_regression_gate_rejected_on_golden_failure
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-312-regression-testing-for-prompts-and-models
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_regression_runner.py::test_regression_gate_approved_on_improvement PASSED [ 20%]
tests/test_regression_runner.py::test_regression_gate_approved_within_tolerance PASSED [ 40%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_severe_accuracy_drop PASSED [ 60%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_schema_invalidity PASSED [ 80%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_golden_failure PASSED [100%]

============================== 5 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-312-regression-testing-for-prompts-and-models
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_regression_runner.py::test_regression_gate_approved_on_improvement PASSED [ 20%]
tests/test_regression_runner.py::test_regression_gate_approved_within_tolerance PASSED [ 40%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_severe_accuracy_drop PASSED [ 60%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_schema_invalidity PASSED [ 80%]
tests/test_regression_runner.py::test_regression_gate_rejected_on_golden_failure PASSED [100%]

============================== 5 passed in 0.01s ===============================

Source files

examples/regression_runner.py (1800 bytes)
from typing import Dict, Any, List

class RegressionTestRunner:
    def __init__(self, tolerance_delta: float = -0.02):
        self.tolerance_delta = tolerance_delta
        
    def evaluate_regression(
        self,
        baseline_results: Dict[str, Any],
        candidate_results: Dict[str, Any]
    ) -> Dict[str, Any]:
        base_accuracy = float(baseline_results.get("accuracy", 0.0))
        cand_accuracy = float(candidate_results.get("accuracy", 0.0))
        
        delta = round(cand_accuracy - base_accuracy, 4)
        
        base_schema_validity = float(baseline_results.get("schema_validity", 1.0))
        cand_schema_validity = float(candidate_results.get("schema_validity", 1.0))
        
        passed_schema = cand_schema_validity >= 1.0
        passed_tolerance = delta >= self.tolerance_delta
        
        cand_failed_golden = candidate_results.get("failed_golden_cases", [])
        passed_golden = len(cand_failed_golden) == 0
        
        gate_passed = passed_schema and passed_tolerance and passed_golden
        
        return {
            "gate_passed": gate_passed,
            "baseline_accuracy": base_accuracy,
            "candidate_accuracy": cand_accuracy,
            "accuracy_delta": delta,
            "schema_validity_passed": passed_schema,
            "golden_invariants_passed": passed_golden,
            "failed_golden_count": len(cand_failed_golden),
            "status": "APPROVED" if gate_passed else "REJECTED"
        }

if __name__ == "__main__":
    runner = RegressionTestRunner(tolerance_delta=-0.02)
    base = {"accuracy": 0.90, "schema_validity": 1.0, "failed_golden_cases": []}
    cand = {"accuracy": 0.92, "schema_validity": 1.0, "failed_golden_cases": []}
    print("Report:", runner.evaluate_regression(base, cand))
metadata.yml (404 bytes)
lesson_id: D312
day: 312
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/regression_runner.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/regression_runner.py (948 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

from typing import Dict, Any, List

class RegressionTestRunner:

    def __init__(self, tolerance_delta: float=-0.02):
        self.tolerance_delta = tolerance_delta

    def evaluate_regression(self, baseline_results: Dict[str, Any], candidate_results: Dict[str, Any]) -> Dict[str, Any]:
        raise NotImplementedError('TASK 1: implement evaluate_regression.')
if __name__ == '__main__':
    runner = RegressionTestRunner(tolerance_delta=-0.02)
    base = {'accuracy': 0.9, 'schema_validity': 1.0, 'failed_golden_cases': []}
    cand = {'accuracy': 0.92, 'schema_validity': 1.0, 'failed_golden_cases': []}
    print('Report:', runner.evaluate_regression(base, cand))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_regression_runner.py (2372 bytes)
import pytest
from examples.regression_runner import RegressionTestRunner

def test_regression_gate_approved_on_improvement():
    runner = RegressionTestRunner(tolerance_delta=-0.02)
    base = {"accuracy": 0.85, "schema_validity": 1.0, "failed_golden_cases": []}
    cand = {"accuracy": 0.89, "schema_validity": 1.0, "failed_golden_cases": []}
    res = runner.evaluate_regression(base, cand)
    assert res["gate_passed"] is True
    assert res["status"] == "APPROVED"
    assert res["accuracy_delta"] == 0.04

def test_regression_gate_approved_within_tolerance():
    runner = RegressionTestRunner(tolerance_delta=-0.02)
    base = {"accuracy": 0.90, "schema_validity": 1.0, "failed_golden_cases": []}
    cand = {"accuracy": 0.89, "schema_validity": 1.0, "failed_golden_cases": []}
    res = runner.evaluate_regression(base, cand)
    assert res["gate_passed"] is True
    assert res["status"] == "APPROVED"
    assert res["accuracy_delta"] == -0.01

def test_regression_gate_rejected_on_severe_accuracy_drop():
    runner = RegressionTestRunner(tolerance_delta=-0.02)
    base = {"accuracy": 0.90, "schema_validity": 1.0, "failed_golden_cases": []}
    cand = {"accuracy": 0.84, "schema_validity": 1.0, "failed_golden_cases": []}
    res = runner.evaluate_regression(base, cand)
    assert res["gate_passed"] is False
    assert res["status"] == "REJECTED"
    assert res["accuracy_delta"] == -0.06

def test_regression_gate_rejected_on_schema_invalidity():
    runner = RegressionTestRunner(tolerance_delta=-0.02)
    base = {"accuracy": 0.80, "schema_validity": 1.0, "failed_golden_cases": []}
    cand = {"accuracy": 0.95, "schema_validity": 0.90, "failed_golden_cases": []}
    res = runner.evaluate_regression(base, cand)
    assert res["gate_passed"] is False
    assert res["schema_validity_passed"] is False
    assert res["status"] == "REJECTED"

def test_regression_gate_rejected_on_golden_failure():
    runner = RegressionTestRunner(tolerance_delta=-0.02)
    base = {"accuracy": 0.85, "schema_validity": 1.0, "failed_golden_cases": []}
    cand = {"accuracy": 0.90, "schema_validity": 1.0, "failed_golden_cases": ["case_auth_01"]}
    res = runner.evaluate_regression(base, cand)
    assert res["gate_passed"] is False
    assert res["golden_invariants_passed"] is False
    assert res["failed_golden_count"] == 1
    assert res["status"] == "REJECTED"

Troubleshooting

Troubleshooting - Day 312

Verify accuracy numbers are normalized floats between 0.0 and 1.0.

Security notes

Security - Day 312

Regression test suites must isolate benchmark secrets and API keys.