LLMs and Generative AI › Prompt Engineering › Day 252

Hands-on lab — Day 252: A Tested Prompt Library

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/a_tested_prompt_library_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/a_tested_prompt_library_lib.py
examples/test_a_tested_prompt_library_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/a_tested_prompt_library_lib.py
starter/test_a_tested_prompt_library_lib.py
tests/run_tests.sh
tests/test_a_tested_prompt_library_lib.py
troubleshooting.md

Lab README

Lab: Day 252 -- A Tested Prompt Library

Lesson

Day number: 252 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Tested Prompt Libraries, Regression Test Harnesses, and CI/CD Evals.

Purpose

Build and test a comprehensive Prompt Evaluation and Regression Test Harness in Python. Construct a multi-case Golden Dataset with deterministic JSON and substring assertions, and evaluate batch pass rates.

Learning objectives

  • Architect an automated prompt regression test harness.
  • Implement deterministic assertions for schema validation and preamble bans.
  • Benchmark prompt latency and pass-rate statistics across test runs.
  • Structure prompt evaluation pipelines for CI/CD integration.

Prerequisites

  • Day 251 (Prompt Injection and Safe Prompting).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python standard json and time libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/a_tested_prompt_library_lib.py: Student scaffold file.
  • examples/a_tested_prompt_library_lib.py: Complete reference implementation.
  • tests/test_a_tested_prompt_library_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/a_tested_prompt_library_lib.py

What the commands do

  • Executes test cases in batch.
  • Asserts JSON format and substring rules.
  • Runs unit test assertions.

Expected output

Prompt Evaluation Demo Completed. Pass Rate: 1.0

Validation steps

  1. Verify run_evaluations processes all test cases.
  2. Confirm is_json assertion identifies malformed JSON.
  3. Validate failure reasons are captured in the report.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Assertion failure: Verify expected substring values match the runner outputs.

Security notes

All prompt testing executes locally in memory.

Extension exercises

  1. Implement an automated LLM-as-a-Judge scoring rubric.
  2. Build an automated GitHub Actions CI workflow for Promptfoo.
  • Lesson title: A Tested Prompt Library
  • Day number: 252 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-252-a-tested-prompt-library
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-252-a-tested-prompt-library when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 252

- `Prompt Evaluation Demo`: Status of prompt evaluation harness execution.

examples-run.txt

Prompt Evaluation Demo Completed. Pass Rate: 1.0

measured-values.txt

Prompt Evaluation Demo: Success
Total Tests: 2
Passed Tests: 2
Pass Rate: 1.0

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_a_tested_prompt_library_lib.py::test_evaluation_harness_pass_rate PASSED [ 33%]
tests/test_a_tested_prompt_library_lib.py::test_assertion_failure_tracking PASSED [ 66%]
tests/test_a_tested_prompt_library_lib.py::test_empty_test_suite_handling PASSED [100%]

============================== 3 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/a_tested_prompt_library_lib.py (3423 bytes)
from typing import List, Dict, Any, Callable
import json
import time

class TestCase:
    def __init__(self, test_id: str, inputs: Dict[str, Any], assertions: List[Dict[str, Any]]):
        self.test_id = test_id
        self.inputs = inputs
        self.assertions = assertions

class PromptEvaluationHarness:
    def __init__(self, prompt_runner_fn: Callable[[Dict[str, Any]], str]):
        self.prompt_runner = prompt_runner_fn
        self.test_suite: List[TestCase] = []

    def add_test_case(self, test_id: str, inputs: Dict[str, Any], assertions: List[Dict[str, Any]]) -> "PromptEvaluationHarness":
        self.test_suite.append(TestCase(test_id, inputs, assertions))
        return self

    def run_evaluations(self) -> Dict[str, Any]:
        results = []
        passed_count = 0

        for case in self.test_suite:
            start_time = time.time()
            output = self.prompt_runner(case.inputs)
            latency_ms = (time.time() - start_time) * 1000

            case_passed = True
            failure_reasons = []

            for assertion in case.assertions:
                assert_type = assertion["type"]
                if assert_type == "is_json":
                    try:
                        json.loads(output)
                    except Exception:
                        case_passed = False
                        failure_reasons.append("Output failed valid JSON deserialization.")
                elif assert_type == "contains":
                    if assertion["value"] not in output:
                        case_passed = False
                        failure_reasons.append(f"Missing required substring: '{assertion['value']}'")
                elif assert_type == "contains_no":
                    if assertion["value"] in output:
                        case_passed = False
                        failure_reasons.append(f"Forbidden substring detected: '{assertion['value']}'")

            if case_passed:
                passed_count += 1

            results.append({
                "test_id": case.test_id,
                "passed": case_passed,
                "latency_ms": latency_ms,
                "failures": failure_reasons
            })

        total = len(self.test_suite)
        pass_rate = (passed_count / total) if total > 0 else 0.0

        return {
            "total_tests": total,
            "passed_tests": passed_count,
            "pass_rate": pass_rate,
            "results": results
        }

def mock_prompt_runner(inputs: Dict[str, Any]) -> str:
    query = inputs.get("query", "")
    if "error" in query:
        return '{"status": "ERROR", "code": 500}'
    return '{"status": "SUCCESS", "result": "Calculated value"}'

def run_evaluation_demo():
    harness = PromptEvaluationHarness(mock_prompt_runner)
    harness.add_test_case(
        test_id="case_happy_path",
        inputs={"query": "process transaction"},
        assertions=[{"type": "is_json"}, {"type": "contains", "value": "SUCCESS"}, {"type": "contains_no", "value": "Sure!"}]
    )
    harness.add_test_case(
        test_id="case_error_path",
        inputs={"query": "trigger error"},
        assertions=[{"type": "is_json"}, {"type": "contains", "value": "ERROR"}]
    )

    report = harness.run_evaluations()
    print("Prompt Evaluation Demo Completed. Pass Rate:", report["pass_rate"])
    return report

if __name__ == "__main__":
    run_evaluation_demo()
examples/test_a_tested_prompt_library_lib.py (1265 bytes)
import pytest
from examples.a_tested_prompt_library_lib import PromptEvaluationHarness

def test_evaluation_harness_pass_rate():
    def mock_runner(inputs):
        return '{"status": "OK", "id": 100}'

    harness = PromptEvaluationHarness(mock_runner)
    harness.add_test_case("test_1", {"query": "q1"}, [{"type": "is_json"}, {"type": "contains", "value": "OK"}])
    harness.add_test_case("test_2", {"query": "q2"}, [{"type": "contains_no", "value": "INVALID"}])

    report = harness.run_evaluations()
    assert report["total_tests"] == 2
    assert report["passed_tests"] == 2
    assert report["pass_rate"] == 1.0

def test_assertion_failure_tracking():
    def mock_runner(inputs):
        return "Sure! Here is your invalid text output."

    harness = PromptEvaluationHarness(mock_runner)
    harness.add_test_case("test_fail", {"query": "fail"}, [{"type": "is_json"}, {"type": "contains_no", "value": "Sure!"}])

    report = harness.run_evaluations()
    assert report["passed_tests"] == 0
    assert len(report["results"][0]["failures"]) == 2

def test_empty_test_suite_handling():
    harness = PromptEvaluationHarness(lambda x: "")
    report = harness.run_evaluations()

    assert report["total_tests"] == 0
    assert report["pass_rate"] == 0.0
metadata.yml (436 bytes)
lesson_id: D252
day: 252
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/a_tested_prompt_library_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/a_tested_prompt_library_lib.py (765 bytes)
from typing import List, Dict, Any, Callable
import json
import time

class TestCase:
    def __init__(self, test_id: str, inputs: Dict[str, Any], assertions: List[Dict[str, Any]]):
        self.test_id = test_id
        self.inputs = inputs
        self.assertions = assertions

class PromptEvaluationHarness:
    def __init__(self, prompt_runner_fn: Callable[[Dict[str, Any]], str]):
        self.prompt_runner = prompt_runner_fn
        self.test_suite: List[TestCase] = []

    def add_test_case(self, test_id: str, inputs: Dict[str, Any], assertions: List[Dict[str, Any]]) -> "PromptEvaluationHarness":
        # TODO: Add test case
        pass

    def run_evaluations(self) -> Dict[str, Any]:
        # TODO: Run evaluations and return summary
        pass
starter/test_a_tested_prompt_library_lib.py (1265 bytes)
import pytest
from examples.a_tested_prompt_library_lib import PromptEvaluationHarness

def test_evaluation_harness_pass_rate():
    def mock_runner(inputs):
        return '{"status": "OK", "id": 100}'

    harness = PromptEvaluationHarness(mock_runner)
    harness.add_test_case("test_1", {"query": "q1"}, [{"type": "is_json"}, {"type": "contains", "value": "OK"}])
    harness.add_test_case("test_2", {"query": "q2"}, [{"type": "contains_no", "value": "INVALID"}])

    report = harness.run_evaluations()
    assert report["total_tests"] == 2
    assert report["passed_tests"] == 2
    assert report["pass_rate"] == 1.0

def test_assertion_failure_tracking():
    def mock_runner(inputs):
        return "Sure! Here is your invalid text output."

    harness = PromptEvaluationHarness(mock_runner)
    harness.add_test_case("test_fail", {"query": "fail"}, [{"type": "is_json"}, {"type": "contains_no", "value": "Sure!"}])

    report = harness.run_evaluations()
    assert report["passed_tests"] == 0
    assert len(report["results"][0]["failures"]) == 2

def test_empty_test_suite_handling():
    harness = PromptEvaluationHarness(lambda x: "")
    report = harness.run_evaluations()

    assert report["total_tests"] == 0
    assert report["pass_rate"] == 0.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 252 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_a_tested_prompt_library_lib.py (1265 bytes)
import pytest
from examples.a_tested_prompt_library_lib import PromptEvaluationHarness

def test_evaluation_harness_pass_rate():
    def mock_runner(inputs):
        return '{"status": "OK", "id": 100}'

    harness = PromptEvaluationHarness(mock_runner)
    harness.add_test_case("test_1", {"query": "q1"}, [{"type": "is_json"}, {"type": "contains", "value": "OK"}])
    harness.add_test_case("test_2", {"query": "q2"}, [{"type": "contains_no", "value": "INVALID"}])

    report = harness.run_evaluations()
    assert report["total_tests"] == 2
    assert report["passed_tests"] == 2
    assert report["pass_rate"] == 1.0

def test_assertion_failure_tracking():
    def mock_runner(inputs):
        return "Sure! Here is your invalid text output."

    harness = PromptEvaluationHarness(mock_runner)
    harness.add_test_case("test_fail", {"query": "fail"}, [{"type": "is_json"}, {"type": "contains_no", "value": "Sure!"}])

    report = harness.run_evaluations()
    assert report["passed_tests"] == 0
    assert len(report["results"][0]["failures"]) == 2

def test_empty_test_suite_handling():
    harness = PromptEvaluationHarness(lambda x: "")
    report = harness.run_evaluations()

    assert report["total_tests"] == 0
    assert report["pass_rate"] == 0.0

Troubleshooting

Troubleshooting: Day 252 - A Tested Prompt Library

Common Issues

  1. Assertion failure in test harness:
    • Cause: Output string does not match assertion condition.
    • Fix: Check failure reasons array in report output.

Security notes

Security & Privacy: Day 252 - A Tested Prompt Library

Security Guidance

  • Automated regression testing prevents prompt injection bypasses from entering production.