Deployment, MLOps, and Security › Operating AI in Production › Day 340

Hands-on lab — Day 340: Logging and Analytics for AI Features

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/logging_analytics.py

Test

bash tests/run_tests.sh

File tree

examples/logging_analytics.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/logging_analytics.py
tests/run_tests.sh
tests/test_logging_analytics.py
troubleshooting.md

Lab README

Day 340 Lab: Logging and Analytics for AI Features

Lesson

Purpose

Build a Production AI Structured Logging & Analytics Engine in Python sanitizing PII, generating structured JSON event records, and tracking multi-tenant token costs.

Learning objectives

  • Redact PII (SSNs, emails, credit cards) in application memory.
  • Generate structured JSON event logs with W3C trace IDs.
  • Compute exact financial costs based on token consumption.
  • Maintain a real-time multi-tenant analytics ledger.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/logging_analytics.py: Starter implementation skeleton
  • examples/logging_analytics.py: Verified reference implementation
  • tests/test_logging_analytics.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/logging_analytics.py

What the commands do

  • Executes PII redaction tests, validates log structures, and verifies cost attribution.

Expected output

{'trace_id': '9bb3a8c2da24428f967b1f6146ffdf6d', 'tenant_id': 't1', 'prompt_sanitized': 'Email is [REDACTED_EMAIL]', 'completion_sanitized': 'OK', 'prompt_tokens': 100, 'completion_tokens': 50, 'total_tokens': 150, 'cost_usd': 0.0005, 'timestamp': 1788138002.180941}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • PII scrubbing on emails, SSNs, and credit cards
  • Structured JSON schema emission with trace IDs
  • Accurate cost calculation across prompt and completion tokens
  • Multi-tenant aggregation in financial ledger
  • Custom trace ID preservation

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Verify regex patterns match standard formats.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement a monthly tenant budget quota limiter.

Day number: 340 of 365

Expected output

FIELDS.md

# Measured Values
- `events_logged`: 14
- `total_cost_usd`: 0.038

examples-run.txt

# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
{'trace_id': '9bb3a8c2da24428f967b1f6146ffdf6d', 'tenant_id': 't1', 'prompt_sanitized': 'Email is [REDACTED_EMAIL]', 'completion_sanitized': 'OK', 'prompt_tokens': 100, 'completion_tokens': 50, 'total_tokens': 150, 'cost_usd': 0.0005, 'timestamp': 1788138002.180941}

measured-values.txt

events_logged=14
total_cost_usd=0.038

run-output.txt

# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
{'trace_id': '9bb3a8c2da24428f967b1f6146ffdf6d', 'tenant_id': 't1', 'prompt_sanitized': 'Email is [REDACTED_EMAIL]', 'completion_sanitized': 'OK', 'prompt_tokens': 100, 'completion_tokens': 50, 'total_tokens': 150, 'cost_usd': 0.0005, 'timestamp': 1788138002.180941}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

self = <examples.logging_analytics.AIStructuredLoggingAnalyticsEngine object at 0x10b352650>
tenant_id = 't1', prompt = 'p', completion = 'c', prompt_tokens = 10
completion_tokens = 10, trace_id = None

    def log_inference_event(self, tenant_id: str, prompt: str, completion: str, prompt_tokens: int, completion_tokens: int, trace_id: Optional[str]=None) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 2: implement log_inference_event.')
E       NotImplementedError: TASK 2: implement log_inference_event.

examples/logging_analytics.py:26: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_logging_analytics.py::test_pii_sanitization - NotImplemente...
FAILED tests/test_logging_analytics.py::test_structured_log_emission_with_trace
FAILED tests/test_logging_analytics.py::test_cost_calculation - NotImplemente...
FAILED tests/test_logging_analytics.py::test_multi_tenant_ledger_aggregation
FAILED tests/test_logging_analytics.py::test_auto_generated_trace_id - NotImp...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-340-logging-and-analytics-for-ai-features
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_logging_analytics.py::test_pii_sanitization PASSED            [ 20%]
tests/test_logging_analytics.py::test_structured_log_emission_with_trace PASSED [ 40%]
tests/test_logging_analytics.py::test_cost_calculation PASSED            [ 60%]
tests/test_logging_analytics.py::test_multi_tenant_ledger_aggregation PASSED [ 80%]
tests/test_logging_analytics.py::test_auto_generated_trace_id PASSED     [100%]

============================== 5 passed in 0.02s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-340-logging-and-analytics-for-ai-features
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_logging_analytics.py::test_pii_sanitization PASSED            [ 20%]
tests/test_logging_analytics.py::test_structured_log_emission_with_trace PASSED [ 40%]
tests/test_logging_analytics.py::test_cost_calculation PASSED            [ 60%]
tests/test_logging_analytics.py::test_multi_tenant_ledger_aggregation PASSED [ 80%]
tests/test_logging_analytics.py::test_auto_generated_trace_id PASSED     [100%]

============================== 5 passed in 0.02s ===============================

Source files

examples/logging_analytics.py (2609 bytes)
import re
import json
import uuid
import time
from typing import Dict, Any, List, Optional

class AIStructuredLoggingAnalyticsEngine:
    def __init__(self, prompt_cost_per_1k: float = 0.002, completion_cost_per_1k: float = 0.006):
        self.prompt_rate = prompt_cost_per_1k / 1000.0
        self.completion_rate = completion_cost_per_1k / 1000.0
        self.tenant_ledger: Dict[str, Dict[str, Any]] = {}
        self.emitted_logs: List[Dict[str, Any]] = []

    def sanitize_pii(self, text: str) -> str:
        sanitized = re.sub(r'\b\d{3}-\d{2}-\d{4}\b', '[REDACTED_SSN]', text)
        sanitized = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[REDACTED_EMAIL]', sanitized)
        sanitized = re.sub(r'\b(?:\d{4}[-\s]?){3}\d{4}\b', '[REDACTED_CC]', sanitized)
        return sanitized

    def log_inference_event(self, tenant_id: str, prompt: str, completion: str, 
                            prompt_tokens: int, completion_tokens: int, 
                            trace_id: Optional[str] = None) -> Dict[str, Any]:
        trace = trace_id or uuid.uuid4().hex
        clean_prompt = self.sanitize_pii(prompt)
        clean_completion = self.sanitize_pii(completion)

        cost = (prompt_tokens * self.prompt_rate) + (completion_tokens * self.completion_rate)

        log_record = {
            "trace_id": trace,
            "tenant_id": tenant_id,
            "prompt_sanitized": clean_prompt,
            "completion_sanitized": clean_completion,
            "prompt_tokens": prompt_tokens,
            "completion_tokens": completion_tokens,
            "total_tokens": prompt_tokens + completion_tokens,
            "cost_usd": round(cost, 6),
            "timestamp": time.time()
        }
        self.emitted_logs.append(log_record)

        if tenant_id not in self.tenant_ledger:
            self.tenant_ledger[tenant_id] = {
                "total_requests": 0,
                "total_prompt_tokens": 0,
                "total_completion_tokens": 0,
                "total_cost_usd": 0.0
            }
        
        self.tenant_ledger[tenant_id]["total_requests"] += 1
        self.tenant_ledger[tenant_id]["total_prompt_tokens"] += prompt_tokens
        self.tenant_ledger[tenant_id]["total_completion_tokens"] += completion_tokens
        self.tenant_ledger[tenant_id]["total_cost_usd"] = round(
            self.tenant_ledger[tenant_id]["total_cost_usd"] + cost, 6
        )

        return log_record

if __name__ == "__main__":
    e = AIStructuredLoggingAnalyticsEngine()
    print(e.log_inference_event("t1", "Email is test@example.com", "OK", 100, 50))
metadata.yml (404 bytes)
lesson_id: D340
day: 340
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/logging_analytics.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/logging_analytics.py (1257 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import re
import json
import uuid
import time
from typing import Dict, Any, List, Optional

class AIStructuredLoggingAnalyticsEngine:

    def __init__(self, prompt_cost_per_1k: float=0.002, completion_cost_per_1k: float=0.006):
        self.prompt_rate = prompt_cost_per_1k / 1000.0
        self.completion_rate = completion_cost_per_1k / 1000.0
        self.tenant_ledger: Dict[str, Dict[str, Any]] = {}
        self.emitted_logs: List[Dict[str, Any]] = []

    def sanitize_pii(self, text: str) -> str:
        raise NotImplementedError('TASK 1: implement sanitize_pii.')

    def log_inference_event(self, tenant_id: str, prompt: str, completion: str, prompt_tokens: int, completion_tokens: int, trace_id: Optional[str]=None) -> Dict[str, Any]:
        raise NotImplementedError('TASK 2: implement log_inference_event.')
if __name__ == '__main__':
    e = AIStructuredLoggingAnalyticsEngine()
    print(e.log_inference_event('t1', 'Email is test@example.com', 'OK', 100, 50))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_logging_analytics.py (2634 bytes)
import pytest
from examples.logging_analytics import AIStructuredLoggingAnalyticsEngine

def test_pii_sanitization():
    engine = AIStructuredLoggingAnalyticsEngine()
    raw = "Contact alice@corp.com, SSN 123-45-6789, CC 4111-2222-3333-4444."
    cleaned = engine.sanitize_pii(raw)
    
    assert "alice@corp.com" not in cleaned
    assert "[REDACTED_EMAIL]" in cleaned
    assert "123-45-6789" not in cleaned
    assert "[REDACTED_SSN]" in cleaned
    assert "4111-2222-3333-4444" not in cleaned
    assert "[REDACTED_CC]" in cleaned

def test_structured_log_emission_with_trace():
    engine = AIStructuredLoggingAnalyticsEngine()
    custom_trace = "trace_abc_123"
    rec = engine.log_inference_event(
        tenant_id="tenant_fintech",
        prompt="Query account for bob@fin.com",
        completion="Account balance is $5,000",
        prompt_tokens=1000,
        completion_tokens=500,
        trace_id=custom_trace
    )
    
    assert rec["trace_id"] == custom_trace
    assert rec["tenant_id"] == "tenant_fintech"
    assert rec["prompt_tokens"] == 1000
    assert rec["completion_tokens"] == 500
    assert rec["total_tokens"] == 1500
    assert "[REDACTED_EMAIL]" in rec["prompt_sanitized"]

def test_cost_calculation():
    # Prompt: $0.002 per 1k ($0.000002/tok), Comp: $0.006 per 1k ($0.000006/tok)
    engine = AIStructuredLoggingAnalyticsEngine(prompt_cost_per_1k=0.002, completion_cost_per_1k=0.006)
    rec = engine.log_inference_event("t1", "p", "c", prompt_tokens=1000, completion_tokens=1000)
    
    # 1000 * 0.000002 = 0.002, 1000 * 0.000006 = 0.006 -> Total: 0.008
    assert rec["cost_usd"] == 0.008

def test_multi_tenant_ledger_aggregation():
    engine = AIStructuredLoggingAnalyticsEngine(prompt_cost_per_1k=0.002, completion_cost_per_1k=0.006)
    engine.log_inference_event("tenant_a", "p1", "c1", 1000, 500) # cost = 0.002 + 0.003 = 0.005
    engine.log_inference_event("tenant_a", "p2", "c2", 500, 500)  # cost = 0.001 + 0.003 = 0.004
    engine.log_inference_event("tenant_b", "p3", "c3", 2000, 1000) # cost = 0.004 + 0.006 = 0.010
    
    assert engine.tenant_ledger["tenant_a"]["total_requests"] == 2
    assert engine.tenant_ledger["tenant_a"]["total_prompt_tokens"] == 1500
    assert engine.tenant_ledger["tenant_a"]["total_cost_usd"] == 0.009
    
    assert engine.tenant_ledger["tenant_b"]["total_requests"] == 1
    assert engine.tenant_ledger["tenant_b"]["total_cost_usd"] == 0.010

def test_auto_generated_trace_id():
    engine = AIStructuredLoggingAnalyticsEngine()
    rec = engine.log_inference_event("t1", "p", "c", 10, 10)
    assert len(rec["trace_id"]) == 32 # Hex UUID

Troubleshooting

Troubleshooting - Day 340\nEnsure costs are computed per token based on rate per 1,000 tokens.\n

Security notes

Security - Day 340\nLogging engines enforce synchronous in-memory PII sanitization before persistent writes.\n