Deployment, MLOps, and Security › Operating AI in Production › Day 343

Hands-on lab — Day 343: A Monitored Production Deployment

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/production_deployment.py

Test

bash tests/run_tests.sh

File tree

examples/production_deployment.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/production_deployment.py
tests/run_tests.sh
tests/test_production_deployment.py
troubleshooting.md

Lab README

Day 343 Lab: A Monitored Production Deployment

Lesson

Purpose

Build a Unified Monitored Production AI Deployment Platform in Python integrating in-memory PII sanitization, deterministic feature flag routing, latency percentile tracking, multi-tenant cost attribution, and automated circuit breaker rollbacks.

Learning objectives

  • Scrub PII (SSNs, emails) from user prompts.
  • Route user traffic deterministically to canary or baseline variants.
  • Compute rolling percentile latencies (P50, P95).
  • Attribute financial costs by tenant ID.
  • Execute automated fast rollbacks when error rates spike.

Prerequisites

  • Python 3.10+ installed
  • pytest, numpy installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest, numpy

Free and open-source options

  • Python Standard Library, Pytest, NumPy

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/production_deployment.py: Starter implementation skeleton
  • examples/production_deployment.py: Verified reference implementation
  • tests/test_production_deployment.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/production_deployment.py

What the commands do

  • Executes unified inference, computes observability metrics, and verifies automated rollback.

Expected output

{'trace_id': 'a0de7e3ad0354054b18897f6b5d63174', 'tenant_id': 't1', 'user_id': 'u1', 'variant': 'BASELINE_V1', 'prompt_sanitized': 'test', 'tokens': 150, 'cost_usd': 0.0005, 'is_error': False, 'circuit_tripped': False, 'timestamp': 1788138002.306768}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • PII scrubbing and trace ID generation
  • Multi-tenant cost attribution ledger
  • Observability report percentile calculation
  • Automated circuit breaker rollback on error rate spike
  • Empty platform report handling

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure numpy is installed for percentile calculation.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement a real-time gross margin calculator.

Day number: 343 of 365

Expected output

FIELDS.md

# Measured Values
- `platform_requests`: 18
- `circuit_tripped`: True

examples-run.txt

# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
{'trace_id': 'a0de7e3ad0354054b18897f6b5d63174', 'tenant_id': 't1', 'user_id': 'u1', 'variant': 'BASELINE_V1', 'prompt_sanitized': 'test', 'tokens': 150, 'cost_usd': 0.0005, 'is_error': False, 'circuit_tripped': False, 'timestamp': 1788138002.306768}

measured-values.txt

platform_requests=18
circuit_tripped=True

run-output.txt

# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
{'trace_id': 'a0de7e3ad0354054b18897f6b5d63174', 'tenant_id': 't1', 'user_id': 'u1', 'variant': 'BASELINE_V1', 'prompt_sanitized': 'test', 'tokens': 150, 'cost_usd': 0.0005, 'is_error': False, 'circuit_tripped': False, 'timestamp': 1788138002.306768}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <examples.production_deployment.UnifiedProductionAIPlatform object at 0x109b6e8b0>

    def get_observability_report(self) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 4: implement get_observability_report.')
E       NotImplementedError: TASK 4: implement get_observability_report.

examples/production_deployment.py:40: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_production_deployment.py::test_pii_sanitization_and_trace_injection
FAILED tests/test_production_deployment.py::test_cost_attribution_ledger - No...
FAILED tests/test_production_deployment.py::test_observability_report_percentiles
FAILED tests/test_production_deployment.py::test_automated_circuit_breaker_rollback_on_error_spike
FAILED tests/test_production_deployment.py::test_empty_platform_report - NotI...
============================== 5 failed in 0.07s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-343-a-monitored-production-deployment
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_production_deployment.py::test_pii_sanitization_and_trace_injection PASSED [ 20%]
tests/test_production_deployment.py::test_cost_attribution_ledger PASSED [ 40%]
tests/test_production_deployment.py::test_observability_report_percentiles PASSED [ 60%]
tests/test_production_deployment.py::test_automated_circuit_breaker_rollback_on_error_spike PASSED [ 80%]
tests/test_production_deployment.py::test_empty_platform_report PASSED   [100%]

============================== 5 passed in 0.05s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-343-a-monitored-production-deployment
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_production_deployment.py::test_pii_sanitization_and_trace_injection PASSED [ 20%]
tests/test_production_deployment.py::test_cost_attribution_ledger PASSED [ 40%]
tests/test_production_deployment.py::test_observability_report_percentiles PASSED [ 60%]
tests/test_production_deployment.py::test_automated_circuit_breaker_rollback_on_error_spike PASSED [ 80%]
tests/test_production_deployment.py::test_empty_platform_report PASSED   [100%]

============================== 5 passed in 0.05s ===============================

Source files

examples/production_deployment.py (3612 bytes)
import re
import uuid
import time
import hashlib
import numpy as np
from typing import Dict, Any, List, Optional

class UnifiedProductionAIPlatform:
    def __init__(self, canary_pct: int = 20, error_threshold_pct: float = 5.0, min_eval_requests: int = 10):
        self.canary_pct = canary_pct
        self.error_threshold = error_threshold_pct
        self.min_eval = min_eval_requests
        
        self.circuit_tripped = False
        self.baseline_variant = "BASELINE_V1"
        self.candidate_variant = "CANDIDATE_V2"
        
        self.latencies_ms: List[float] = []
        self.total_requests = 0
        self.total_errors = 0
        self.tenant_ledger: Dict[str, float] = {}
        self.emitted_logs: List[Dict[str, Any]] = []

    def sanitize_pii(self, text: str) -> str:
        text = re.sub(r'\b\d{3}-\d{2}-\d{4}\b', '[REDACTED_SSN]', text)
        text = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[REDACTED_EMAIL]', text)
        return text

    def _get_variant(self, user_id: str) -> str:
        if self.circuit_tripped:
            return self.baseline_variant
        h = int(hashlib.md5(f"flag:{user_id}".encode("utf-8")).hexdigest()[:8], 16) % 100
        return self.candidate_variant if h < self.canary_pct else self.baseline_variant

    def execute_inference(self, tenant_id: str, user_id: str, prompt: str, 
                          prompt_tokens: int = 100, completion_tokens: int = 50, 
                          simulate_error: bool = False, latency_ms: float = 120.0) -> Dict[str, Any]:
        trace_id = uuid.uuid4().hex
        clean_prompt = self.sanitize_pii(prompt)
        variant = self._get_variant(user_id)
        
        self.total_requests += 1
        if simulate_error:
            self.total_errors += 1
        else:
            self.latencies_ms.append(float(latency_ms))

        if not self.circuit_tripped and self.total_requests >= self.min_eval:
            err_rate = (self.total_errors / self.total_requests) * 100.0
            if err_rate > self.error_threshold:
                self.circuit_tripped = True

        cost = (prompt_tokens * 0.000002) + (completion_tokens * 0.000006)
        self.tenant_ledger[tenant_id] = round(self.tenant_ledger.get(tenant_id, 0.0) + cost, 6)

        log_rec = {
            "trace_id": trace_id,
            "tenant_id": tenant_id,
            "user_id": user_id,
            "variant": variant,
            "prompt_sanitized": clean_prompt,
            "tokens": prompt_tokens + completion_tokens,
            "cost_usd": round(cost, 6),
            "is_error": simulate_error,
            "circuit_tripped": self.circuit_tripped,
            "timestamp": time.time()
        }
        self.emitted_logs.append(log_rec)
        return log_rec

    def get_observability_report(self) -> Dict[str, Any]:
        p50 = float(np.percentile(self.latencies_ms, 50)) if self.latencies_ms else 0.0
        p95 = float(np.percentile(self.latencies_ms, 95)) if self.latencies_ms else 0.0
        err_rate = (self.total_errors / self.total_requests * 100.0) if self.total_requests > 0 else 0.0
        
        return {
            "total_requests": self.total_requests,
            "total_errors": self.total_errors,
            "error_rate_pct": round(err_rate, 2),
            "p50_latency_ms": round(p50, 2),
            "p95_latency_ms": round(p95, 2),
            "circuit_tripped": self.circuit_tripped,
            "total_tenants": len(self.tenant_ledger)
        }

if __name__ == "__main__":
    p = UnifiedProductionAIPlatform()
    print(p.execute_inference("t1", "u1", "test"))
metadata.yml (408 bytes)
lesson_id: D343
day: 343
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/production_deployment.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
pytest>=7.0.0
numpy>=1.20.0
starter/production_deployment.py (1764 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import re
import uuid
import time
import hashlib
import numpy as np
from typing import Dict, Any, List, Optional

class UnifiedProductionAIPlatform:

    def __init__(self, canary_pct: int=20, error_threshold_pct: float=5.0, min_eval_requests: int=10):
        self.canary_pct = canary_pct
        self.error_threshold = error_threshold_pct
        self.min_eval = min_eval_requests
        self.circuit_tripped = False
        self.baseline_variant = 'BASELINE_V1'
        self.candidate_variant = 'CANDIDATE_V2'
        self.latencies_ms: List[float] = []
        self.total_requests = 0
        self.total_errors = 0
        self.tenant_ledger: Dict[str, float] = {}
        self.emitted_logs: List[Dict[str, Any]] = []

    def sanitize_pii(self, text: str) -> str:
        raise NotImplementedError('TASK 1: implement sanitize_pii.')

    def _get_variant(self, user_id: str) -> str:
        raise NotImplementedError('TASK 2: implement _get_variant.')

    def execute_inference(self, tenant_id: str, user_id: str, prompt: str, prompt_tokens: int=100, completion_tokens: int=50, simulate_error: bool=False, latency_ms: float=120.0) -> Dict[str, Any]:
        raise NotImplementedError('TASK 3: implement execute_inference.')

    def get_observability_report(self) -> Dict[str, Any]:
        raise NotImplementedError('TASK 4: implement get_observability_report.')
if __name__ == '__main__':
    p = UnifiedProductionAIPlatform()
    print(p.execute_inference('t1', 'u1', 'test'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_production_deployment.py (2260 bytes)
import pytest
from examples.production_deployment import UnifiedProductionAIPlatform

def test_pii_sanitization_and_trace_injection():
    platform = UnifiedProductionAIPlatform()
    res = platform.execute_inference("tenant_1", "user_1", "User email is alice@corp.com and SSN is 111-22-3333.")
    
    assert "[REDACTED_EMAIL]" in res["prompt_sanitized"]
    assert "[REDACTED_SSN]" in res["prompt_sanitized"]
    assert len(res["trace_id"]) == 32

def test_cost_attribution_ledger():
    platform = UnifiedProductionAIPlatform()
    # 1000 prompt * 0.000002 = 0.002, 500 comp * 0.000006 = 0.003 -> Total 0.005
    platform.execute_inference("tenant_fin", "u1", "p1", prompt_tokens=1000, completion_tokens=500)
    platform.execute_inference("tenant_fin", "u2", "p2", prompt_tokens=1000, completion_tokens=500)
    
    assert platform.tenant_ledger["tenant_fin"] == 0.010

def test_observability_report_percentiles():
    platform = UnifiedProductionAIPlatform()
    for lat in [100.0, 120.0, 140.0, 160.0, 500.0]:
        platform.execute_inference("t1", "u1", "p", latency_ms=lat)
        
    rep = platform.get_observability_report()
    assert rep["total_requests"] == 5
    assert rep["p50_latency_ms"] == 140.0
    assert rep["p95_latency_ms"] == 432.0

def test_automated_circuit_breaker_rollback_on_error_spike():
    platform = UnifiedProductionAIPlatform(canary_pct=100, error_threshold_pct=10.0, min_eval_requests=10)
    
    # 8 success, 4 errors = 33.3% > 10%
    for _ in range(8):
        platform.execute_inference("t1", "u1", "good", simulate_error=False)
    for _ in range(4):
        platform.execute_inference("t1", "u1", "bad", simulate_error=True)
        
    rep = platform.get_observability_report()
    assert rep["circuit_tripped"] is True
    assert platform.circuit_tripped is True
    
    # Next inference should automatically receive BASELINE_V1 due to tripped circuit
    rec = platform.execute_inference("t1", "u1", "next query")
    assert rec["variant"] == "BASELINE_V1"

def test_empty_platform_report():
    platform = UnifiedProductionAIPlatform()
    rep = platform.get_observability_report()
    assert rep["total_requests"] == 0
    assert rep["p50_latency_ms"] == 0.0
    assert rep["circuit_tripped"] is False

Troubleshooting

Troubleshooting - Day 343\nEnsure latency percentiles are computed on non-error samples.\n

Security notes

Security - Day 343\nUnified platform enforces in-memory PII sanitization and automated circuit breaker protection.\n