Capstone › Capstone Build I: Foundation › Day 357
Hands-on lab — Day 357: Milestone Review and Course Correction
- ← Back to the Day 357 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/capstone/day-357-milestone-review-and-course-correction/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/audit.py Test
bash tests/run_tests.sh File tree
examples/audit.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/audit.py tests/run_tests.sh tests/test_audit.py troubleshooting.md
Lab README
Day 357 Lab: Milestone Review and Course Correction
Lesson
- Lesson title: Milestone Review and Course Correction
- Day number: 357 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-357-milestone-review-and-course-correction
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-357-milestone-review-and-course-correctionwhen the site is running.
Purpose
Build a Capstone Milestone 1 Audit Engine in Python that profiles component latency budgets, verifies evaluation metrics, and certifies the working vertical slice for production shipping.
Learning objectives
- Profile end-to-end vertical slice execution latencies.
- Break down latency budgets by architectural component.
- Audit evaluation accuracy metrics against quality thresholds.
- Generate automated Milestone 1 approval reports.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/audit.py: Starter implementation skeletonexamples/audit.py: Verified reference implementationtests/test_audit.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/audit.py
What the commands do
- Profiles pipeline latency and verifies milestone quality criteria.
Expected output
{'milestone': 'CAPSTONE_MILESTONE_1', 'overall_status': 'APPROVED', 'checks': {'latency_budget': 'PASS', 'faithfulness_accuracy': 'PASS', 'schema_integrity': 'PASS'}, 'metrics': {'measured_latency_ms': 0.0, 'faithfulness_score': 0.95}}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Vertical slice latency profiling
- Latency budget compliance (<1,500ms)
- Accuracy metric threshold audit
- Schema validity checks
- Milestone approval vs rejection logic
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure pipeline return values match expected dictionary formats.
Security notes
Runs locally with zero external network calls.
Extension exercises
Add memory allocation profiling via tracemalloc.
Navigation
Day number: 357 of 365
Expected output
FIELDS.md
# Measured Values
- `latency_budget_compliance`: 100.0
- `milestone_status`: APPROVED
examples-run.txt
{'milestone': 'CAPSTONE_MILESTONE_1', 'overall_status': 'APPROVED', 'checks': {'latency_budget': 'PASS', 'faithfulness_accuracy': 'PASS', 'schema_integrity': 'PASS'}, 'metrics': {'measured_latency_ms': 0.0, 'faithfulness_score': 0.95}}
measured-values.txt
latency_budget_compliance=100.0
milestone_status=APPROVED
run-output.txt
{'milestone': 'CAPSTONE_MILESTONE_1', 'overall_status': 'APPROVED', 'checks': {'latency_budget': 'PASS', 'faithfulness_accuracy': 'PASS', 'schema_integrity': 'PASS'}, 'metrics': {'measured_latency_ms': 0.0, 'faithfulness_score': 0.95}}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.audit.Milestone1AuditSuite object at 0x10888a9e0>
pipeline_fn = <function test_milestone_rejection_on_latency_exceeded.<locals>.slow_pipe at 0x1088f7270>
eval_metrics = {'faithfulness': 0.95}
def audit_milestone(self, pipeline_fn, eval_metrics: Dict[str, float]) -> Dict[str, Any]:
> raise NotImplementedError('TASK 2: implement audit_milestone.')
E NotImplementedError: TASK 2: implement audit_milestone.
examples/audit.py:22: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_audit.py::test_latency_profiling_compliance - NotImplemente...
FAILED tests/test_audit.py::test_milestone_approval_on_healthy_system - NotIm...
FAILED tests/test_audit.py::test_milestone_rejection_on_low_accuracy - NotImp...
FAILED tests/test_audit.py::test_milestone_rejection_on_schema_invalid - NotI...
FAILED tests/test_audit.py::test_milestone_rejection_on_latency_exceeded - No...
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/capstone/day-357-milestone-review-and-course-correction
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_audit.py::test_latency_profiling_compliance PASSED [ 20%]
tests/test_audit.py::test_milestone_approval_on_healthy_system PASSED [ 40%]
tests/test_audit.py::test_milestone_rejection_on_low_accuracy PASSED [ 60%]
tests/test_audit.py::test_milestone_rejection_on_schema_invalid PASSED [ 80%]
tests/test_audit.py::test_milestone_rejection_on_latency_exceeded PASSED [100%]
============================== 5 passed in 0.03s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/capstone/day-357-milestone-review-and-course-correction
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_audit.py::test_latency_profiling_compliance PASSED [ 20%]
tests/test_audit.py::test_milestone_approval_on_healthy_system PASSED [ 40%]
tests/test_audit.py::test_milestone_rejection_on_low_accuracy PASSED [ 60%]
tests/test_audit.py::test_milestone_rejection_on_schema_invalid PASSED [ 80%]
tests/test_audit.py::test_milestone_rejection_on_latency_exceeded PASSED [100%]
============================== 5 passed in 0.03s ===============================
Source files
examples/audit.py (2111 bytes)
import time
import json
from typing import Dict, Any, List, Tuple
class Milestone1AuditSuite:
def __init__(self, max_latency_ms: float = 1500.0, min_faithfulness: float = 0.90):
self.max_latency_ms = max_latency_ms
self.min_faithfulness = min_faithfulness
def profile_vertical_slice(self, mock_pipeline_fn) -> Dict[str, Any]:
t0 = time.perf_counter()
breakdown, result = mock_pipeline_fn()
total_time_ms = round((time.perf_counter() - t0) * 1000.0, 2)
return {
"total_latency_ms": total_time_ms,
"latency_budget_ms": self.max_latency_ms,
"latency_compliant": total_time_ms <= self.max_latency_ms,
"component_breakdown_ms": breakdown,
"output_result": result
}
def audit_milestone(self, pipeline_fn, eval_metrics: Dict[str, float]) -> Dict[str, Any]:
profile = self.profile_vertical_slice(pipeline_fn)
faithfulness_pass = eval_metrics.get("faithfulness", 0.0) >= self.min_faithfulness
latency_pass = profile["latency_compliant"]
schema_pass = profile["output_result"].get("schema_valid", False)
all_passed = faithfulness_pass and latency_pass and schema_pass
return {
"milestone": "CAPSTONE_MILESTONE_1",
"overall_status": "APPROVED" if all_passed else "REJECTED",
"checks": {
"latency_budget": "PASS" if latency_pass else "FAIL",
"faithfulness_accuracy": "PASS" if faithfulness_pass else "FAIL",
"schema_integrity": "PASS" if schema_pass else "FAIL"
},
"metrics": {
"measured_latency_ms": profile["total_latency_ms"],
"faithfulness_score": eval_metrics.get("faithfulness", 0.0)
}
}
if __name__ == "__main__":
def sample_pipeline():
return {"ingress": 1.0, "retrieval": 5.0, "reasoning": 5.0}, {"schema_valid": True, "answer": "Done"}
auditor = Milestone1AuditSuite()
print(auditor.audit_milestone(sample_pipeline, {"faithfulness": 0.95}))
metadata.yml (392 bytes)
lesson_id: D357
day: 357
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/audit.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/audit.py (1111 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import time
import json
from typing import Dict, Any, List, Tuple
class Milestone1AuditSuite:
def __init__(self, max_latency_ms: float=1500.0, min_faithfulness: float=0.9):
self.max_latency_ms = max_latency_ms
self.min_faithfulness = min_faithfulness
def profile_vertical_slice(self, mock_pipeline_fn) -> Dict[str, Any]:
raise NotImplementedError('TASK 1: implement profile_vertical_slice.')
def audit_milestone(self, pipeline_fn, eval_metrics: Dict[str, float]) -> Dict[str, Any]:
raise NotImplementedError('TASK 2: implement audit_milestone.')
if __name__ == '__main__':
def sample_pipeline():
raise NotImplementedError('TASK 3: implement sample_pipeline.')
auditor = Milestone1AuditSuite()
print(auditor.audit_milestone(sample_pipeline, {'faithfulness': 0.95}))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_audit.py (2139 bytes)
import pytest
import time
from examples.audit import Milestone1AuditSuite
def test_latency_profiling_compliance():
auditor = Milestone1AuditSuite(max_latency_ms=1500.0)
def fast_pipeline():
return {"retrieval": 10.0, "llm": 20.0}, {"schema_valid": True}
profile = auditor.profile_vertical_slice(fast_pipeline)
assert profile["latency_compliant"] is True
assert profile["total_latency_ms"] < 1500.0
assert "retrieval" in profile["component_breakdown_ms"]
def test_milestone_approval_on_healthy_system():
auditor = Milestone1AuditSuite()
def healthy_pipe():
return {"all": 15.0}, {"schema_valid": True, "answer": "OK"}
report = auditor.audit_milestone(healthy_pipe, {"faithfulness": 0.95, "recall": 0.90})
assert report["overall_status"] == "APPROVED"
assert report["checks"]["latency_budget"] == "PASS"
assert report["checks"]["faithfulness_accuracy"] == "PASS"
assert report["checks"]["schema_integrity"] == "PASS"
def test_milestone_rejection_on_low_accuracy():
auditor = Milestone1AuditSuite(min_faithfulness=0.90)
def healthy_pipe():
return {"all": 15.0}, {"schema_valid": True}
report = auditor.audit_milestone(healthy_pipe, {"faithfulness": 0.75})
assert report["overall_status"] == "REJECTED"
assert report["checks"]["faithfulness_accuracy"] == "FAIL"
def test_milestone_rejection_on_schema_invalid():
auditor = Milestone1AuditSuite()
def broken_schema_pipe():
return {"all": 10.0}, {"schema_valid": False}
report = auditor.audit_milestone(broken_schema_pipe, {"faithfulness": 0.95})
assert report["overall_status"] == "REJECTED"
assert report["checks"]["schema_integrity"] == "FAIL"
def test_milestone_rejection_on_latency_exceeded():
auditor = Milestone1AuditSuite(max_latency_ms=1.0)
def slow_pipe():
time.sleep(0.01) # 10ms > 1.0ms budget
return {}, {"schema_valid": True}
report = auditor.audit_milestone(slow_pipe, {"faithfulness": 0.95})
assert report["checks"]["latency_budget"] == "FAIL"
assert report["overall_status"] == "REJECTED"