LLMs and Generative AI › Prompt Engineering › Day 252
Hands-on lab — Day 252: A Tested Prompt Library
- ← Back to the Day 252 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-252-a-tested-prompt-library/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/a_tested_prompt_library_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/a_tested_prompt_library_lib.py examples/test_a_tested_prompt_library_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/a_tested_prompt_library_lib.py starter/test_a_tested_prompt_library_lib.py tests/run_tests.sh tests/test_a_tested_prompt_library_lib.py troubleshooting.md
Lab README
Lab: Day 252 -- A Tested Prompt Library
Lesson
Day number: 252 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Tested Prompt Libraries, Regression Test Harnesses, and CI/CD Evals.
Purpose
Build and test a comprehensive Prompt Evaluation and Regression Test Harness in Python. Construct a multi-case Golden Dataset with deterministic JSON and substring assertions, and evaluate batch pass rates.
Learning objectives
- Architect an automated prompt regression test harness.
- Implement deterministic assertions for schema validation and preamble bans.
- Benchmark prompt latency and pass-rate statistics across test runs.
- Structure prompt evaluation pipelines for CI/CD integration.
Prerequisites
- Day 251 (Prompt Injection and Safe Prompting).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python standard json and time libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/a_tested_prompt_library_lib.py: Student scaffold file.examples/a_tested_prompt_library_lib.py: Complete reference implementation.tests/test_a_tested_prompt_library_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/a_tested_prompt_library_lib.py
What the commands do
- Executes test cases in batch.
- Asserts JSON format and substring rules.
- Runs unit test assertions.
Expected output
Prompt Evaluation Demo Completed. Pass Rate: 1.0
Validation steps
- Verify
run_evaluationsprocesses all test cases. - Confirm
is_jsonassertion identifies malformed JSON. - Validate failure reasons are captured in the report.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Assertion failure: Verify expected substring values match the runner outputs.
Security notes
All prompt testing executes locally in memory.
Extension exercises
- Implement an automated LLM-as-a-Judge scoring rubric.
- Build an automated GitHub Actions CI workflow for Promptfoo.
Navigation
- Lesson title: A Tested Prompt Library
- Day number: 252 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-252-a-tested-prompt-library
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-252-a-tested-prompt-librarywhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 252
- `Prompt Evaluation Demo`: Status of prompt evaluation harness execution.
examples-run.txt
Prompt Evaluation Demo Completed. Pass Rate: 1.0
measured-values.txt
Prompt Evaluation Demo: Success
Total Tests: 2
Passed Tests: 2
Pass Rate: 1.0
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_a_tested_prompt_library_lib.py::test_evaluation_harness_pass_rate PASSED [ 33%]
tests/test_a_tested_prompt_library_lib.py::test_assertion_failure_tracking PASSED [ 66%]
tests/test_a_tested_prompt_library_lib.py::test_empty_test_suite_handling PASSED [100%]
============================== 3 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/a_tested_prompt_library_lib.py (3423 bytes)
from typing import List, Dict, Any, Callable
import json
import time
class TestCase:
def __init__(self, test_id: str, inputs: Dict[str, Any], assertions: List[Dict[str, Any]]):
self.test_id = test_id
self.inputs = inputs
self.assertions = assertions
class PromptEvaluationHarness:
def __init__(self, prompt_runner_fn: Callable[[Dict[str, Any]], str]):
self.prompt_runner = prompt_runner_fn
self.test_suite: List[TestCase] = []
def add_test_case(self, test_id: str, inputs: Dict[str, Any], assertions: List[Dict[str, Any]]) -> "PromptEvaluationHarness":
self.test_suite.append(TestCase(test_id, inputs, assertions))
return self
def run_evaluations(self) -> Dict[str, Any]:
results = []
passed_count = 0
for case in self.test_suite:
start_time = time.time()
output = self.prompt_runner(case.inputs)
latency_ms = (time.time() - start_time) * 1000
case_passed = True
failure_reasons = []
for assertion in case.assertions:
assert_type = assertion["type"]
if assert_type == "is_json":
try:
json.loads(output)
except Exception:
case_passed = False
failure_reasons.append("Output failed valid JSON deserialization.")
elif assert_type == "contains":
if assertion["value"] not in output:
case_passed = False
failure_reasons.append(f"Missing required substring: '{assertion['value']}'")
elif assert_type == "contains_no":
if assertion["value"] in output:
case_passed = False
failure_reasons.append(f"Forbidden substring detected: '{assertion['value']}'")
if case_passed:
passed_count += 1
results.append({
"test_id": case.test_id,
"passed": case_passed,
"latency_ms": latency_ms,
"failures": failure_reasons
})
total = len(self.test_suite)
pass_rate = (passed_count / total) if total > 0 else 0.0
return {
"total_tests": total,
"passed_tests": passed_count,
"pass_rate": pass_rate,
"results": results
}
def mock_prompt_runner(inputs: Dict[str, Any]) -> str:
query = inputs.get("query", "")
if "error" in query:
return '{"status": "ERROR", "code": 500}'
return '{"status": "SUCCESS", "result": "Calculated value"}'
def run_evaluation_demo():
harness = PromptEvaluationHarness(mock_prompt_runner)
harness.add_test_case(
test_id="case_happy_path",
inputs={"query": "process transaction"},
assertions=[{"type": "is_json"}, {"type": "contains", "value": "SUCCESS"}, {"type": "contains_no", "value": "Sure!"}]
)
harness.add_test_case(
test_id="case_error_path",
inputs={"query": "trigger error"},
assertions=[{"type": "is_json"}, {"type": "contains", "value": "ERROR"}]
)
report = harness.run_evaluations()
print("Prompt Evaluation Demo Completed. Pass Rate:", report["pass_rate"])
return report
if __name__ == "__main__":
run_evaluation_demo()
examples/test_a_tested_prompt_library_lib.py (1265 bytes)
import pytest
from examples.a_tested_prompt_library_lib import PromptEvaluationHarness
def test_evaluation_harness_pass_rate():
def mock_runner(inputs):
return '{"status": "OK", "id": 100}'
harness = PromptEvaluationHarness(mock_runner)
harness.add_test_case("test_1", {"query": "q1"}, [{"type": "is_json"}, {"type": "contains", "value": "OK"}])
harness.add_test_case("test_2", {"query": "q2"}, [{"type": "contains_no", "value": "INVALID"}])
report = harness.run_evaluations()
assert report["total_tests"] == 2
assert report["passed_tests"] == 2
assert report["pass_rate"] == 1.0
def test_assertion_failure_tracking():
def mock_runner(inputs):
return "Sure! Here is your invalid text output."
harness = PromptEvaluationHarness(mock_runner)
harness.add_test_case("test_fail", {"query": "fail"}, [{"type": "is_json"}, {"type": "contains_no", "value": "Sure!"}])
report = harness.run_evaluations()
assert report["passed_tests"] == 0
assert len(report["results"][0]["failures"]) == 2
def test_empty_test_suite_handling():
harness = PromptEvaluationHarness(lambda x: "")
report = harness.run_evaluations()
assert report["total_tests"] == 0
assert report["pass_rate"] == 0.0
metadata.yml (436 bytes)
lesson_id: D252
day: 252
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/a_tested_prompt_library_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/a_tested_prompt_library_lib.py (765 bytes)
from typing import List, Dict, Any, Callable
import json
import time
class TestCase:
def __init__(self, test_id: str, inputs: Dict[str, Any], assertions: List[Dict[str, Any]]):
self.test_id = test_id
self.inputs = inputs
self.assertions = assertions
class PromptEvaluationHarness:
def __init__(self, prompt_runner_fn: Callable[[Dict[str, Any]], str]):
self.prompt_runner = prompt_runner_fn
self.test_suite: List[TestCase] = []
def add_test_case(self, test_id: str, inputs: Dict[str, Any], assertions: List[Dict[str, Any]]) -> "PromptEvaluationHarness":
# TODO: Add test case
pass
def run_evaluations(self) -> Dict[str, Any]:
# TODO: Run evaluations and return summary
pass
starter/test_a_tested_prompt_library_lib.py (1265 bytes)
import pytest
from examples.a_tested_prompt_library_lib import PromptEvaluationHarness
def test_evaluation_harness_pass_rate():
def mock_runner(inputs):
return '{"status": "OK", "id": 100}'
harness = PromptEvaluationHarness(mock_runner)
harness.add_test_case("test_1", {"query": "q1"}, [{"type": "is_json"}, {"type": "contains", "value": "OK"}])
harness.add_test_case("test_2", {"query": "q2"}, [{"type": "contains_no", "value": "INVALID"}])
report = harness.run_evaluations()
assert report["total_tests"] == 2
assert report["passed_tests"] == 2
assert report["pass_rate"] == 1.0
def test_assertion_failure_tracking():
def mock_runner(inputs):
return "Sure! Here is your invalid text output."
harness = PromptEvaluationHarness(mock_runner)
harness.add_test_case("test_fail", {"query": "fail"}, [{"type": "is_json"}, {"type": "contains_no", "value": "Sure!"}])
report = harness.run_evaluations()
assert report["passed_tests"] == 0
assert len(report["results"][0]["failures"]) == 2
def test_empty_test_suite_handling():
harness = PromptEvaluationHarness(lambda x: "")
report = harness.run_evaluations()
assert report["total_tests"] == 0
assert report["pass_rate"] == 0.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 252 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_a_tested_prompt_library_lib.py (1265 bytes)
import pytest
from examples.a_tested_prompt_library_lib import PromptEvaluationHarness
def test_evaluation_harness_pass_rate():
def mock_runner(inputs):
return '{"status": "OK", "id": 100}'
harness = PromptEvaluationHarness(mock_runner)
harness.add_test_case("test_1", {"query": "q1"}, [{"type": "is_json"}, {"type": "contains", "value": "OK"}])
harness.add_test_case("test_2", {"query": "q2"}, [{"type": "contains_no", "value": "INVALID"}])
report = harness.run_evaluations()
assert report["total_tests"] == 2
assert report["passed_tests"] == 2
assert report["pass_rate"] == 1.0
def test_assertion_failure_tracking():
def mock_runner(inputs):
return "Sure! Here is your invalid text output."
harness = PromptEvaluationHarness(mock_runner)
harness.add_test_case("test_fail", {"query": "fail"}, [{"type": "is_json"}, {"type": "contains_no", "value": "Sure!"}])
report = harness.run_evaluations()
assert report["passed_tests"] == 0
assert len(report["results"][0]["failures"]) == 2
def test_empty_test_suite_handling():
harness = PromptEvaluationHarness(lambda x: "")
report = harness.run_evaluations()
assert report["total_tests"] == 0
assert report["pass_rate"] == 0.0
Troubleshooting
Troubleshooting: Day 252 - A Tested Prompt Library
Common Issues
- Assertion failure in test harness:
- Cause: Output string does not match assertion condition.
- Fix: Check failure reasons array in report output.
Security notes
Security & Privacy: Day 252 - A Tested Prompt Library
Security Guidance
- Automated regression testing prevents prompt injection bypasses from entering production.