Deployment, MLOps, and Security › Operating AI in Production › Day 343
Hands-on lab — Day 343: A Monitored Production Deployment
- ← Back to the Day 343 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deployment-mlops-and-security/day-343-a-monitored-production-deployment/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/production_deployment.py Test
bash tests/run_tests.sh File tree
examples/production_deployment.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/production_deployment.py tests/run_tests.sh tests/test_production_deployment.py troubleshooting.md
Lab README
Day 343 Lab: A Monitored Production Deployment
Lesson
- Lesson title: A Monitored Production Deployment
- Day number: 343 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-343-a-monitored-production-deployment
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-343-a-monitored-production-deploymentwhen the site is running.
Purpose
Build a Unified Monitored Production AI Deployment Platform in Python integrating in-memory PII sanitization, deterministic feature flag routing, latency percentile tracking, multi-tenant cost attribution, and automated circuit breaker rollbacks.
Learning objectives
- Scrub PII (SSNs, emails) from user prompts.
- Route user traffic deterministically to canary or baseline variants.
- Compute rolling percentile latencies (P50, P95).
- Attribute financial costs by tenant ID.
- Execute automated fast rollbacks when error rates spike.
Prerequisites
- Python 3.10+ installed
- pytest, numpy installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest, numpy
Free and open-source options
- Python Standard Library, Pytest, NumPy
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/production_deployment.py: Starter implementation skeletonexamples/production_deployment.py: Verified reference implementationtests/test_production_deployment.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/production_deployment.py
What the commands do
- Executes unified inference, computes observability metrics, and verifies automated rollback.
Expected output
{'trace_id': 'a0de7e3ad0354054b18897f6b5d63174', 'tenant_id': 't1', 'user_id': 'u1', 'variant': 'BASELINE_V1', 'prompt_sanitized': 'test', 'tokens': 150, 'cost_usd': 0.0005, 'is_error': False, 'circuit_tripped': False, 'timestamp': 1788138002.306768}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- PII scrubbing and trace ID generation
- Multi-tenant cost attribution ledger
- Observability report percentile calculation
- Automated circuit breaker rollback on error rate spike
- Empty platform report handling
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure numpy is installed for percentile calculation.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement a real-time gross margin calculator.
Navigation
Day number: 343 of 365
Expected output
FIELDS.md
# Measured Values
- `platform_requests`: 18
- `circuit_tripped`: True
examples-run.txt
# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
{'trace_id': 'a0de7e3ad0354054b18897f6b5d63174', 'tenant_id': 't1', 'user_id': 'u1', 'variant': 'BASELINE_V1', 'prompt_sanitized': 'test', 'tokens': 150, 'cost_usd': 0.0005, 'is_error': False, 'circuit_tripped': False, 'timestamp': 1788138002.306768}
measured-values.txt
platform_requests=18
circuit_tripped=True
run-output.txt
# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
{'trace_id': 'a0de7e3ad0354054b18897f6b5d63174', 'tenant_id': 't1', 'user_id': 'u1', 'variant': 'BASELINE_V1', 'prompt_sanitized': 'test', 'tokens': 150, 'cost_usd': 0.0005, 'is_error': False, 'circuit_tripped': False, 'timestamp': 1788138002.306768}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <examples.production_deployment.UnifiedProductionAIPlatform object at 0x109b6e8b0>
def get_observability_report(self) -> Dict[str, Any]:
> raise NotImplementedError('TASK 4: implement get_observability_report.')
E NotImplementedError: TASK 4: implement get_observability_report.
examples/production_deployment.py:40: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_production_deployment.py::test_pii_sanitization_and_trace_injection
FAILED tests/test_production_deployment.py::test_cost_attribution_ledger - No...
FAILED tests/test_production_deployment.py::test_observability_report_percentiles
FAILED tests/test_production_deployment.py::test_automated_circuit_breaker_rollback_on_error_spike
FAILED tests/test_production_deployment.py::test_empty_platform_report - NotI...
============================== 5 failed in 0.07s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-343-a-monitored-production-deployment
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_production_deployment.py::test_pii_sanitization_and_trace_injection PASSED [ 20%]
tests/test_production_deployment.py::test_cost_attribution_ledger PASSED [ 40%]
tests/test_production_deployment.py::test_observability_report_percentiles PASSED [ 60%]
tests/test_production_deployment.py::test_automated_circuit_breaker_rollback_on_error_spike PASSED [ 80%]
tests/test_production_deployment.py::test_empty_platform_report PASSED [100%]
============================== 5 passed in 0.05s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-343-a-monitored-production-deployment
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_production_deployment.py::test_pii_sanitization_and_trace_injection PASSED [ 20%]
tests/test_production_deployment.py::test_cost_attribution_ledger PASSED [ 40%]
tests/test_production_deployment.py::test_observability_report_percentiles PASSED [ 60%]
tests/test_production_deployment.py::test_automated_circuit_breaker_rollback_on_error_spike PASSED [ 80%]
tests/test_production_deployment.py::test_empty_platform_report PASSED [100%]
============================== 5 passed in 0.05s ===============================
Source files
examples/production_deployment.py (3612 bytes)
import re
import uuid
import time
import hashlib
import numpy as np
from typing import Dict, Any, List, Optional
class UnifiedProductionAIPlatform:
def __init__(self, canary_pct: int = 20, error_threshold_pct: float = 5.0, min_eval_requests: int = 10):
self.canary_pct = canary_pct
self.error_threshold = error_threshold_pct
self.min_eval = min_eval_requests
self.circuit_tripped = False
self.baseline_variant = "BASELINE_V1"
self.candidate_variant = "CANDIDATE_V2"
self.latencies_ms: List[float] = []
self.total_requests = 0
self.total_errors = 0
self.tenant_ledger: Dict[str, float] = {}
self.emitted_logs: List[Dict[str, Any]] = []
def sanitize_pii(self, text: str) -> str:
text = re.sub(r'\b\d{3}-\d{2}-\d{4}\b', '[REDACTED_SSN]', text)
text = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[REDACTED_EMAIL]', text)
return text
def _get_variant(self, user_id: str) -> str:
if self.circuit_tripped:
return self.baseline_variant
h = int(hashlib.md5(f"flag:{user_id}".encode("utf-8")).hexdigest()[:8], 16) % 100
return self.candidate_variant if h < self.canary_pct else self.baseline_variant
def execute_inference(self, tenant_id: str, user_id: str, prompt: str,
prompt_tokens: int = 100, completion_tokens: int = 50,
simulate_error: bool = False, latency_ms: float = 120.0) -> Dict[str, Any]:
trace_id = uuid.uuid4().hex
clean_prompt = self.sanitize_pii(prompt)
variant = self._get_variant(user_id)
self.total_requests += 1
if simulate_error:
self.total_errors += 1
else:
self.latencies_ms.append(float(latency_ms))
if not self.circuit_tripped and self.total_requests >= self.min_eval:
err_rate = (self.total_errors / self.total_requests) * 100.0
if err_rate > self.error_threshold:
self.circuit_tripped = True
cost = (prompt_tokens * 0.000002) + (completion_tokens * 0.000006)
self.tenant_ledger[tenant_id] = round(self.tenant_ledger.get(tenant_id, 0.0) + cost, 6)
log_rec = {
"trace_id": trace_id,
"tenant_id": tenant_id,
"user_id": user_id,
"variant": variant,
"prompt_sanitized": clean_prompt,
"tokens": prompt_tokens + completion_tokens,
"cost_usd": round(cost, 6),
"is_error": simulate_error,
"circuit_tripped": self.circuit_tripped,
"timestamp": time.time()
}
self.emitted_logs.append(log_rec)
return log_rec
def get_observability_report(self) -> Dict[str, Any]:
p50 = float(np.percentile(self.latencies_ms, 50)) if self.latencies_ms else 0.0
p95 = float(np.percentile(self.latencies_ms, 95)) if self.latencies_ms else 0.0
err_rate = (self.total_errors / self.total_requests * 100.0) if self.total_requests > 0 else 0.0
return {
"total_requests": self.total_requests,
"total_errors": self.total_errors,
"error_rate_pct": round(err_rate, 2),
"p50_latency_ms": round(p50, 2),
"p95_latency_ms": round(p95, 2),
"circuit_tripped": self.circuit_tripped,
"total_tenants": len(self.tenant_ledger)
}
if __name__ == "__main__":
p = UnifiedProductionAIPlatform()
print(p.execute_inference("t1", "u1", "test"))
metadata.yml (408 bytes)
lesson_id: D343
day: 343
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/production_deployment.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
pytest>=7.0.0
numpy>=1.20.0
starter/production_deployment.py (1764 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import re
import uuid
import time
import hashlib
import numpy as np
from typing import Dict, Any, List, Optional
class UnifiedProductionAIPlatform:
def __init__(self, canary_pct: int=20, error_threshold_pct: float=5.0, min_eval_requests: int=10):
self.canary_pct = canary_pct
self.error_threshold = error_threshold_pct
self.min_eval = min_eval_requests
self.circuit_tripped = False
self.baseline_variant = 'BASELINE_V1'
self.candidate_variant = 'CANDIDATE_V2'
self.latencies_ms: List[float] = []
self.total_requests = 0
self.total_errors = 0
self.tenant_ledger: Dict[str, float] = {}
self.emitted_logs: List[Dict[str, Any]] = []
def sanitize_pii(self, text: str) -> str:
raise NotImplementedError('TASK 1: implement sanitize_pii.')
def _get_variant(self, user_id: str) -> str:
raise NotImplementedError('TASK 2: implement _get_variant.')
def execute_inference(self, tenant_id: str, user_id: str, prompt: str, prompt_tokens: int=100, completion_tokens: int=50, simulate_error: bool=False, latency_ms: float=120.0) -> Dict[str, Any]:
raise NotImplementedError('TASK 3: implement execute_inference.')
def get_observability_report(self) -> Dict[str, Any]:
raise NotImplementedError('TASK 4: implement get_observability_report.')
if __name__ == '__main__':
p = UnifiedProductionAIPlatform()
print(p.execute_inference('t1', 'u1', 'test'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_production_deployment.py (2260 bytes)
import pytest
from examples.production_deployment import UnifiedProductionAIPlatform
def test_pii_sanitization_and_trace_injection():
platform = UnifiedProductionAIPlatform()
res = platform.execute_inference("tenant_1", "user_1", "User email is alice@corp.com and SSN is 111-22-3333.")
assert "[REDACTED_EMAIL]" in res["prompt_sanitized"]
assert "[REDACTED_SSN]" in res["prompt_sanitized"]
assert len(res["trace_id"]) == 32
def test_cost_attribution_ledger():
platform = UnifiedProductionAIPlatform()
# 1000 prompt * 0.000002 = 0.002, 500 comp * 0.000006 = 0.003 -> Total 0.005
platform.execute_inference("tenant_fin", "u1", "p1", prompt_tokens=1000, completion_tokens=500)
platform.execute_inference("tenant_fin", "u2", "p2", prompt_tokens=1000, completion_tokens=500)
assert platform.tenant_ledger["tenant_fin"] == 0.010
def test_observability_report_percentiles():
platform = UnifiedProductionAIPlatform()
for lat in [100.0, 120.0, 140.0, 160.0, 500.0]:
platform.execute_inference("t1", "u1", "p", latency_ms=lat)
rep = platform.get_observability_report()
assert rep["total_requests"] == 5
assert rep["p50_latency_ms"] == 140.0
assert rep["p95_latency_ms"] == 432.0
def test_automated_circuit_breaker_rollback_on_error_spike():
platform = UnifiedProductionAIPlatform(canary_pct=100, error_threshold_pct=10.0, min_eval_requests=10)
# 8 success, 4 errors = 33.3% > 10%
for _ in range(8):
platform.execute_inference("t1", "u1", "good", simulate_error=False)
for _ in range(4):
platform.execute_inference("t1", "u1", "bad", simulate_error=True)
rep = platform.get_observability_report()
assert rep["circuit_tripped"] is True
assert platform.circuit_tripped is True
# Next inference should automatically receive BASELINE_V1 due to tripped circuit
rec = platform.execute_inference("t1", "u1", "next query")
assert rec["variant"] == "BASELINE_V1"
def test_empty_platform_report():
platform = UnifiedProductionAIPlatform()
rep = platform.get_observability_report()
assert rep["total_requests"] == 0
assert rep["p50_latency_ms"] == 0.0
assert rep["circuit_tripped"] is False