Deployment, MLOps, and Security › Operating AI in Production › Day 340
Hands-on lab — Day 340: Logging and Analytics for AI Features
- ← Back to the Day 340 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deployment-mlops-and-security/day-340-logging-and-analytics-for-ai-features/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/logging_analytics.py Test
bash tests/run_tests.sh File tree
examples/logging_analytics.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/logging_analytics.py tests/run_tests.sh tests/test_logging_analytics.py troubleshooting.md
Lab README
Day 340 Lab: Logging and Analytics for AI Features
Lesson
- Lesson title: Logging and Analytics for AI Features
- Day number: 340 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-340-logging-and-analytics-for-ai-features
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-340-logging-and-analytics-for-ai-featureswhen the site is running.
Purpose
Build a Production AI Structured Logging & Analytics Engine in Python sanitizing PII, generating structured JSON event records, and tracking multi-tenant token costs.
Learning objectives
- Redact PII (SSNs, emails, credit cards) in application memory.
- Generate structured JSON event logs with W3C trace IDs.
- Compute exact financial costs based on token consumption.
- Maintain a real-time multi-tenant analytics ledger.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/logging_analytics.py: Starter implementation skeletonexamples/logging_analytics.py: Verified reference implementationtests/test_logging_analytics.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/logging_analytics.py
What the commands do
- Executes PII redaction tests, validates log structures, and verifies cost attribution.
Expected output
{'trace_id': '9bb3a8c2da24428f967b1f6146ffdf6d', 'tenant_id': 't1', 'prompt_sanitized': 'Email is [REDACTED_EMAIL]', 'completion_sanitized': 'OK', 'prompt_tokens': 100, 'completion_tokens': 50, 'total_tokens': 150, 'cost_usd': 0.0005, 'timestamp': 1788138002.180941}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- PII scrubbing on emails, SSNs, and credit cards
- Structured JSON schema emission with trace IDs
- Accurate cost calculation across prompt and completion tokens
- Multi-tenant aggregation in financial ledger
- Custom trace ID preservation
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Verify regex patterns match standard formats.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement a monthly tenant budget quota limiter.
Navigation
Day number: 340 of 365
Expected output
FIELDS.md
# Measured Values
- `events_logged`: 14
- `total_cost_usd`: 0.038
examples-run.txt
# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
{'trace_id': '9bb3a8c2da24428f967b1f6146ffdf6d', 'tenant_id': 't1', 'prompt_sanitized': 'Email is [REDACTED_EMAIL]', 'completion_sanitized': 'OK', 'prompt_tokens': 100, 'completion_tokens': 50, 'total_tokens': 150, 'cost_usd': 0.0005, 'timestamp': 1788138002.180941}
measured-values.txt
events_logged=14
total_cost_usd=0.038
run-output.txt
# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
{'trace_id': '9bb3a8c2da24428f967b1f6146ffdf6d', 'tenant_id': 't1', 'prompt_sanitized': 'Email is [REDACTED_EMAIL]', 'completion_sanitized': 'OK', 'prompt_tokens': 100, 'completion_tokens': 50, 'total_tokens': 150, 'cost_usd': 0.0005, 'timestamp': 1788138002.180941}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.logging_analytics.AIStructuredLoggingAnalyticsEngine object at 0x10b352650>
tenant_id = 't1', prompt = 'p', completion = 'c', prompt_tokens = 10
completion_tokens = 10, trace_id = None
def log_inference_event(self, tenant_id: str, prompt: str, completion: str, prompt_tokens: int, completion_tokens: int, trace_id: Optional[str]=None) -> Dict[str, Any]:
> raise NotImplementedError('TASK 2: implement log_inference_event.')
E NotImplementedError: TASK 2: implement log_inference_event.
examples/logging_analytics.py:26: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_logging_analytics.py::test_pii_sanitization - NotImplemente...
FAILED tests/test_logging_analytics.py::test_structured_log_emission_with_trace
FAILED tests/test_logging_analytics.py::test_cost_calculation - NotImplemente...
FAILED tests/test_logging_analytics.py::test_multi_tenant_ledger_aggregation
FAILED tests/test_logging_analytics.py::test_auto_generated_trace_id - NotImp...
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-340-logging-and-analytics-for-ai-features
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_logging_analytics.py::test_pii_sanitization PASSED [ 20%]
tests/test_logging_analytics.py::test_structured_log_emission_with_trace PASSED [ 40%]
tests/test_logging_analytics.py::test_cost_calculation PASSED [ 60%]
tests/test_logging_analytics.py::test_multi_tenant_ledger_aggregation PASSED [ 80%]
tests/test_logging_analytics.py::test_auto_generated_trace_id PASSED [100%]
============================== 5 passed in 0.02s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-340-logging-and-analytics-for-ai-features
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_logging_analytics.py::test_pii_sanitization PASSED [ 20%]
tests/test_logging_analytics.py::test_structured_log_emission_with_trace PASSED [ 40%]
tests/test_logging_analytics.py::test_cost_calculation PASSED [ 60%]
tests/test_logging_analytics.py::test_multi_tenant_ledger_aggregation PASSED [ 80%]
tests/test_logging_analytics.py::test_auto_generated_trace_id PASSED [100%]
============================== 5 passed in 0.02s ===============================
Source files
examples/logging_analytics.py (2609 bytes)
import re
import json
import uuid
import time
from typing import Dict, Any, List, Optional
class AIStructuredLoggingAnalyticsEngine:
def __init__(self, prompt_cost_per_1k: float = 0.002, completion_cost_per_1k: float = 0.006):
self.prompt_rate = prompt_cost_per_1k / 1000.0
self.completion_rate = completion_cost_per_1k / 1000.0
self.tenant_ledger: Dict[str, Dict[str, Any]] = {}
self.emitted_logs: List[Dict[str, Any]] = []
def sanitize_pii(self, text: str) -> str:
sanitized = re.sub(r'\b\d{3}-\d{2}-\d{4}\b', '[REDACTED_SSN]', text)
sanitized = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[REDACTED_EMAIL]', sanitized)
sanitized = re.sub(r'\b(?:\d{4}[-\s]?){3}\d{4}\b', '[REDACTED_CC]', sanitized)
return sanitized
def log_inference_event(self, tenant_id: str, prompt: str, completion: str,
prompt_tokens: int, completion_tokens: int,
trace_id: Optional[str] = None) -> Dict[str, Any]:
trace = trace_id or uuid.uuid4().hex
clean_prompt = self.sanitize_pii(prompt)
clean_completion = self.sanitize_pii(completion)
cost = (prompt_tokens * self.prompt_rate) + (completion_tokens * self.completion_rate)
log_record = {
"trace_id": trace,
"tenant_id": tenant_id,
"prompt_sanitized": clean_prompt,
"completion_sanitized": clean_completion,
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"total_tokens": prompt_tokens + completion_tokens,
"cost_usd": round(cost, 6),
"timestamp": time.time()
}
self.emitted_logs.append(log_record)
if tenant_id not in self.tenant_ledger:
self.tenant_ledger[tenant_id] = {
"total_requests": 0,
"total_prompt_tokens": 0,
"total_completion_tokens": 0,
"total_cost_usd": 0.0
}
self.tenant_ledger[tenant_id]["total_requests"] += 1
self.tenant_ledger[tenant_id]["total_prompt_tokens"] += prompt_tokens
self.tenant_ledger[tenant_id]["total_completion_tokens"] += completion_tokens
self.tenant_ledger[tenant_id]["total_cost_usd"] = round(
self.tenant_ledger[tenant_id]["total_cost_usd"] + cost, 6
)
return log_record
if __name__ == "__main__":
e = AIStructuredLoggingAnalyticsEngine()
print(e.log_inference_event("t1", "Email is test@example.com", "OK", 100, 50))
metadata.yml (404 bytes)
lesson_id: D340
day: 340
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/logging_analytics.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/logging_analytics.py (1257 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import re
import json
import uuid
import time
from typing import Dict, Any, List, Optional
class AIStructuredLoggingAnalyticsEngine:
def __init__(self, prompt_cost_per_1k: float=0.002, completion_cost_per_1k: float=0.006):
self.prompt_rate = prompt_cost_per_1k / 1000.0
self.completion_rate = completion_cost_per_1k / 1000.0
self.tenant_ledger: Dict[str, Dict[str, Any]] = {}
self.emitted_logs: List[Dict[str, Any]] = []
def sanitize_pii(self, text: str) -> str:
raise NotImplementedError('TASK 1: implement sanitize_pii.')
def log_inference_event(self, tenant_id: str, prompt: str, completion: str, prompt_tokens: int, completion_tokens: int, trace_id: Optional[str]=None) -> Dict[str, Any]:
raise NotImplementedError('TASK 2: implement log_inference_event.')
if __name__ == '__main__':
e = AIStructuredLoggingAnalyticsEngine()
print(e.log_inference_event('t1', 'Email is test@example.com', 'OK', 100, 50))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_logging_analytics.py (2634 bytes)
import pytest
from examples.logging_analytics import AIStructuredLoggingAnalyticsEngine
def test_pii_sanitization():
engine = AIStructuredLoggingAnalyticsEngine()
raw = "Contact alice@corp.com, SSN 123-45-6789, CC 4111-2222-3333-4444."
cleaned = engine.sanitize_pii(raw)
assert "alice@corp.com" not in cleaned
assert "[REDACTED_EMAIL]" in cleaned
assert "123-45-6789" not in cleaned
assert "[REDACTED_SSN]" in cleaned
assert "4111-2222-3333-4444" not in cleaned
assert "[REDACTED_CC]" in cleaned
def test_structured_log_emission_with_trace():
engine = AIStructuredLoggingAnalyticsEngine()
custom_trace = "trace_abc_123"
rec = engine.log_inference_event(
tenant_id="tenant_fintech",
prompt="Query account for bob@fin.com",
completion="Account balance is $5,000",
prompt_tokens=1000,
completion_tokens=500,
trace_id=custom_trace
)
assert rec["trace_id"] == custom_trace
assert rec["tenant_id"] == "tenant_fintech"
assert rec["prompt_tokens"] == 1000
assert rec["completion_tokens"] == 500
assert rec["total_tokens"] == 1500
assert "[REDACTED_EMAIL]" in rec["prompt_sanitized"]
def test_cost_calculation():
# Prompt: $0.002 per 1k ($0.000002/tok), Comp: $0.006 per 1k ($0.000006/tok)
engine = AIStructuredLoggingAnalyticsEngine(prompt_cost_per_1k=0.002, completion_cost_per_1k=0.006)
rec = engine.log_inference_event("t1", "p", "c", prompt_tokens=1000, completion_tokens=1000)
# 1000 * 0.000002 = 0.002, 1000 * 0.000006 = 0.006 -> Total: 0.008
assert rec["cost_usd"] == 0.008
def test_multi_tenant_ledger_aggregation():
engine = AIStructuredLoggingAnalyticsEngine(prompt_cost_per_1k=0.002, completion_cost_per_1k=0.006)
engine.log_inference_event("tenant_a", "p1", "c1", 1000, 500) # cost = 0.002 + 0.003 = 0.005
engine.log_inference_event("tenant_a", "p2", "c2", 500, 500) # cost = 0.001 + 0.003 = 0.004
engine.log_inference_event("tenant_b", "p3", "c3", 2000, 1000) # cost = 0.004 + 0.006 = 0.010
assert engine.tenant_ledger["tenant_a"]["total_requests"] == 2
assert engine.tenant_ledger["tenant_a"]["total_prompt_tokens"] == 1500
assert engine.tenant_ledger["tenant_a"]["total_cost_usd"] == 0.009
assert engine.tenant_ledger["tenant_b"]["total_requests"] == 1
assert engine.tenant_ledger["tenant_b"]["total_cost_usd"] == 0.010
def test_auto_generated_trace_id():
engine = AIStructuredLoggingAnalyticsEngine()
rec = engine.log_inference_event("t1", "p", "c", 10, 10)
assert len(rec["trace_id"]) == 32 # Hex UUID