Capstone › Capstone Build I: Foundation › Day 357

Hands-on lab — Day 357: Milestone Review and Course Correction

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/audit.py

Test

bash tests/run_tests.sh

File tree

examples/audit.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/audit.py
tests/run_tests.sh
tests/test_audit.py
troubleshooting.md

Lab README

Day 357 Lab: Milestone Review and Course Correction

Lesson

Purpose

Build a Capstone Milestone 1 Audit Engine in Python that profiles component latency budgets, verifies evaluation metrics, and certifies the working vertical slice for production shipping.

Learning objectives

  • Profile end-to-end vertical slice execution latencies.
  • Break down latency budgets by architectural component.
  • Audit evaluation accuracy metrics against quality thresholds.
  • Generate automated Milestone 1 approval reports.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/audit.py: Starter implementation skeleton
  • examples/audit.py: Verified reference implementation
  • tests/test_audit.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/audit.py

What the commands do

  • Profiles pipeline latency and verifies milestone quality criteria.

Expected output

{'milestone': 'CAPSTONE_MILESTONE_1', 'overall_status': 'APPROVED', 'checks': {'latency_budget': 'PASS', 'faithfulness_accuracy': 'PASS', 'schema_integrity': 'PASS'}, 'metrics': {'measured_latency_ms': 0.0, 'faithfulness_score': 0.95}}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Vertical slice latency profiling
  • Latency budget compliance (<1,500ms)
  • Accuracy metric threshold audit
  • Schema validity checks
  • Milestone approval vs rejection logic

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure pipeline return values match expected dictionary formats.

Security notes

Runs locally with zero external network calls.

Extension exercises

Add memory allocation profiling via tracemalloc.

Day number: 357 of 365

Expected output

FIELDS.md

# Measured Values
- `latency_budget_compliance`: 100.0
- `milestone_status`: APPROVED

examples-run.txt

{'milestone': 'CAPSTONE_MILESTONE_1', 'overall_status': 'APPROVED', 'checks': {'latency_budget': 'PASS', 'faithfulness_accuracy': 'PASS', 'schema_integrity': 'PASS'}, 'metrics': {'measured_latency_ms': 0.0, 'faithfulness_score': 0.95}}

measured-values.txt

latency_budget_compliance=100.0
milestone_status=APPROVED

run-output.txt

{'milestone': 'CAPSTONE_MILESTONE_1', 'overall_status': 'APPROVED', 'checks': {'latency_budget': 'PASS', 'faithfulness_accuracy': 'PASS', 'schema_integrity': 'PASS'}, 'metrics': {'measured_latency_ms': 0.0, 'faithfulness_score': 0.95}}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

self = <examples.audit.Milestone1AuditSuite object at 0x10888a9e0>
pipeline_fn = <function test_milestone_rejection_on_latency_exceeded.<locals>.slow_pipe at 0x1088f7270>
eval_metrics = {'faithfulness': 0.95}

    def audit_milestone(self, pipeline_fn, eval_metrics: Dict[str, float]) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 2: implement audit_milestone.')
E       NotImplementedError: TASK 2: implement audit_milestone.

examples/audit.py:22: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_audit.py::test_latency_profiling_compliance - NotImplemente...
FAILED tests/test_audit.py::test_milestone_approval_on_healthy_system - NotIm...
FAILED tests/test_audit.py::test_milestone_rejection_on_low_accuracy - NotImp...
FAILED tests/test_audit.py::test_milestone_rejection_on_schema_invalid - NotI...
FAILED tests/test_audit.py::test_milestone_rejection_on_latency_exceeded - No...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/capstone/day-357-milestone-review-and-course-correction
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_audit.py::test_latency_profiling_compliance PASSED            [ 20%]
tests/test_audit.py::test_milestone_approval_on_healthy_system PASSED    [ 40%]
tests/test_audit.py::test_milestone_rejection_on_low_accuracy PASSED     [ 60%]
tests/test_audit.py::test_milestone_rejection_on_schema_invalid PASSED   [ 80%]
tests/test_audit.py::test_milestone_rejection_on_latency_exceeded PASSED [100%]

============================== 5 passed in 0.03s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/capstone/day-357-milestone-review-and-course-correction
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_audit.py::test_latency_profiling_compliance PASSED            [ 20%]
tests/test_audit.py::test_milestone_approval_on_healthy_system PASSED    [ 40%]
tests/test_audit.py::test_milestone_rejection_on_low_accuracy PASSED     [ 60%]
tests/test_audit.py::test_milestone_rejection_on_schema_invalid PASSED   [ 80%]
tests/test_audit.py::test_milestone_rejection_on_latency_exceeded PASSED [100%]

============================== 5 passed in 0.03s ===============================

Source files

examples/audit.py (2111 bytes)
import time
import json
from typing import Dict, Any, List, Tuple

class Milestone1AuditSuite:
    def __init__(self, max_latency_ms: float = 1500.0, min_faithfulness: float = 0.90):
        self.max_latency_ms = max_latency_ms
        self.min_faithfulness = min_faithfulness

    def profile_vertical_slice(self, mock_pipeline_fn) -> Dict[str, Any]:
        t0 = time.perf_counter()
        breakdown, result = mock_pipeline_fn()
        total_time_ms = round((time.perf_counter() - t0) * 1000.0, 2)

        return {
            "total_latency_ms": total_time_ms,
            "latency_budget_ms": self.max_latency_ms,
            "latency_compliant": total_time_ms <= self.max_latency_ms,
            "component_breakdown_ms": breakdown,
            "output_result": result
        }

    def audit_milestone(self, pipeline_fn, eval_metrics: Dict[str, float]) -> Dict[str, Any]:
        profile = self.profile_vertical_slice(pipeline_fn)
        
        faithfulness_pass = eval_metrics.get("faithfulness", 0.0) >= self.min_faithfulness
        latency_pass = profile["latency_compliant"]
        schema_pass = profile["output_result"].get("schema_valid", False)

        all_passed = faithfulness_pass and latency_pass and schema_pass

        return {
            "milestone": "CAPSTONE_MILESTONE_1",
            "overall_status": "APPROVED" if all_passed else "REJECTED",
            "checks": {
                "latency_budget": "PASS" if latency_pass else "FAIL",
                "faithfulness_accuracy": "PASS" if faithfulness_pass else "FAIL",
                "schema_integrity": "PASS" if schema_pass else "FAIL"
            },
            "metrics": {
                "measured_latency_ms": profile["total_latency_ms"],
                "faithfulness_score": eval_metrics.get("faithfulness", 0.0)
            }
        }

if __name__ == "__main__":
    def sample_pipeline():
        return {"ingress": 1.0, "retrieval": 5.0, "reasoning": 5.0}, {"schema_valid": True, "answer": "Done"}
    auditor = Milestone1AuditSuite()
    print(auditor.audit_milestone(sample_pipeline, {"faithfulness": 0.95}))
metadata.yml (392 bytes)
lesson_id: D357
day: 357
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/audit.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/audit.py (1111 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import time
import json
from typing import Dict, Any, List, Tuple

class Milestone1AuditSuite:

    def __init__(self, max_latency_ms: float=1500.0, min_faithfulness: float=0.9):
        self.max_latency_ms = max_latency_ms
        self.min_faithfulness = min_faithfulness

    def profile_vertical_slice(self, mock_pipeline_fn) -> Dict[str, Any]:
        raise NotImplementedError('TASK 1: implement profile_vertical_slice.')

    def audit_milestone(self, pipeline_fn, eval_metrics: Dict[str, float]) -> Dict[str, Any]:
        raise NotImplementedError('TASK 2: implement audit_milestone.')
if __name__ == '__main__':

    def sample_pipeline():
        raise NotImplementedError('TASK 3: implement sample_pipeline.')
    auditor = Milestone1AuditSuite()
    print(auditor.audit_milestone(sample_pipeline, {'faithfulness': 0.95}))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_audit.py (2139 bytes)
import pytest
import time
from examples.audit import Milestone1AuditSuite

def test_latency_profiling_compliance():
    auditor = Milestone1AuditSuite(max_latency_ms=1500.0)
    def fast_pipeline():
        return {"retrieval": 10.0, "llm": 20.0}, {"schema_valid": True}
    
    profile = auditor.profile_vertical_slice(fast_pipeline)
    assert profile["latency_compliant"] is True
    assert profile["total_latency_ms"] < 1500.0
    assert "retrieval" in profile["component_breakdown_ms"]

def test_milestone_approval_on_healthy_system():
    auditor = Milestone1AuditSuite()
    def healthy_pipe():
        return {"all": 15.0}, {"schema_valid": True, "answer": "OK"}
    
    report = auditor.audit_milestone(healthy_pipe, {"faithfulness": 0.95, "recall": 0.90})
    assert report["overall_status"] == "APPROVED"
    assert report["checks"]["latency_budget"] == "PASS"
    assert report["checks"]["faithfulness_accuracy"] == "PASS"
    assert report["checks"]["schema_integrity"] == "PASS"

def test_milestone_rejection_on_low_accuracy():
    auditor = Milestone1AuditSuite(min_faithfulness=0.90)
    def healthy_pipe():
        return {"all": 15.0}, {"schema_valid": True}
    
    report = auditor.audit_milestone(healthy_pipe, {"faithfulness": 0.75})
    assert report["overall_status"] == "REJECTED"
    assert report["checks"]["faithfulness_accuracy"] == "FAIL"

def test_milestone_rejection_on_schema_invalid():
    auditor = Milestone1AuditSuite()
    def broken_schema_pipe():
        return {"all": 10.0}, {"schema_valid": False}
    
    report = auditor.audit_milestone(broken_schema_pipe, {"faithfulness": 0.95})
    assert report["overall_status"] == "REJECTED"
    assert report["checks"]["schema_integrity"] == "FAIL"

def test_milestone_rejection_on_latency_exceeded():
    auditor = Milestone1AuditSuite(max_latency_ms=1.0)
    def slow_pipe():
        time.sleep(0.01) # 10ms > 1.0ms budget
        return {}, {"schema_valid": True}
    
    report = auditor.audit_milestone(slow_pipe, {"faithfulness": 0.95})
    assert report["checks"]["latency_budget"] == "FAIL"
    assert report["overall_status"] == "REJECTED"

Troubleshooting

Troubleshooting - Day 357\nEnsure pipeline profiling functions return a tuple of (breakdown, result).\n

Security notes

Security - Day 357\nMilestone audit engine verifies zero secrets and validates sandboxed execution limits.\n