Deployment, MLOps, and Security › Operating AI in Production › Day 339
Hands-on lab — Day 339: Monitoring and Alerting
- ← Back to the Day 339 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deployment-mlops-and-security/day-339-monitoring-and-alerting/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/monitoring_alerting.py Test
bash tests/run_tests.sh File tree
examples/monitoring_alerting.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/monitoring_alerting.py tests/run_tests.sh tests/test_monitoring_alerting.py troubleshooting.md
Lab README
Day 339 Lab: Monitoring and Alerting
Lesson
- Lesson title: Monitoring and Alerting
- Day number: 339 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-339-monitoring-and-alerting
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-339-monitoring-and-alertingwhen the site is running.
Purpose
Build an AI Observability and Alerting Engine in Python calculating rolling percentile latencies, evaluating SRE Golden Signals, and triggering actionable alerts.
Learning objectives
- Record inference latency samples and error outcomes.
- Compute rolling percentiles (P50, P95, P99).
- Evaluate error rate percentages against SLA thresholds.
- Trigger structured critical alerts on SLA breaches.
Prerequisites
- Python 3.10+ installed
- pytest, numpy installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest, numpy
Free and open-source options
- Python Standard Library, Pytest, NumPy
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/monitoring_alerting.py: Starter implementation skeletonexamples/monitoring_alerting.py: Verified reference implementationtests/test_monitoring_alerting.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/monitoring_alerting.py
What the commands do
- Evaluates percentile latency distributions and verifies alert trigger thresholds.
Expected output
{'total_requests': 1, 'p50_ttft_ms': 100.0, 'p95_ttft_ms': 100.0, 'p99_ttft_ms': 100.0, 'error_rate_pct': 0.0, 'active_alert_count': 0}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Correct P50, P95, and P99 calculation on uniform and heavy-tailed samples
- Triggering HighTailLatencyP95 alert when threshold is exceeded
- Triggering HighInferenceErrorRate alert when error rate spikes
- Zero alert activation on healthy baseline traffic
- Graceful empty metrics handling
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure numpy is imported for percentile calculation.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement alert resolution cooldown hysteresis.
Navigation
Day number: 339 of 365
Expected output
FIELDS.md
# Measured Values
- `metrics_evaluated`: 12
- `alerts_fired`: 2
examples-run.txt
{'total_requests': 1, 'p50_ttft_ms': 100.0, 'p95_ttft_ms': 100.0, 'p99_ttft_ms': 100.0, 'error_rate_pct': 0.0, 'active_alert_count': 0}
measured-values.txt
metrics_evaluated=12
alerts_fired=2
run-output.txt
{'total_requests': 1, 'p50_ttft_ms': 100.0, 'p95_ttft_ms': 100.0, 'p99_ttft_ms': 100.0, 'error_rate_pct': 0.0, 'active_alert_count': 0}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <examples.monitoring_alerting.AIObservabilityAlertEngine object at 0x10719e650>
def evaluate_metrics(self) -> Dict[str, Any]:
> raise NotImplementedError('TASK 2: implement evaluate_metrics.')
E NotImplementedError: TASK 2: implement evaluate_metrics.
examples/monitoring_alerting.py:26: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_monitoring_alerting.py::test_percentile_calculation - NotIm...
FAILED tests/test_monitoring_alerting.py::test_high_tail_latency_alert_trigger
FAILED tests/test_monitoring_alerting.py::test_error_rate_alert_trigger - Not...
FAILED tests/test_monitoring_alerting.py::test_healthy_traffic_zero_alerts - ...
FAILED tests/test_monitoring_alerting.py::test_empty_metrics_handling - NotIm...
============================== 5 failed in 0.06s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-339-monitoring-and-alerting
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_monitoring_alerting.py::test_percentile_calculation PASSED [ 20%]
tests/test_monitoring_alerting.py::test_high_tail_latency_alert_trigger PASSED [ 40%]
tests/test_monitoring_alerting.py::test_error_rate_alert_trigger PASSED [ 60%]
tests/test_monitoring_alerting.py::test_healthy_traffic_zero_alerts PASSED [ 80%]
tests/test_monitoring_alerting.py::test_empty_metrics_handling PASSED [100%]
============================== 5 passed in 0.04s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-339-monitoring-and-alerting
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_monitoring_alerting.py::test_percentile_calculation PASSED [ 20%]
tests/test_monitoring_alerting.py::test_high_tail_latency_alert_trigger PASSED [ 40%]
tests/test_monitoring_alerting.py::test_error_rate_alert_trigger PASSED [ 60%]
tests/test_monitoring_alerting.py::test_healthy_traffic_zero_alerts PASSED [ 80%]
tests/test_monitoring_alerting.py::test_empty_metrics_handling PASSED [100%]
============================== 5 passed in 0.04s ===============================
Source files
examples/monitoring_alerting.py (2323 bytes)
import numpy as np
import time
from typing import Dict, Any, List, Optional
class AIObservabilityAlertEngine:
def __init__(self, p95_ttft_threshold_ms: float = 300.0, error_rate_threshold_pct: float = 1.0):
self.ttft_threshold = float(p95_ttft_threshold_ms)
self.error_threshold = float(error_rate_threshold_pct)
self.ttft_samples: List[float] = []
self.total_requests = 0
self.total_errors = 0
self.active_alerts: List[Dict[str, Any]] = []
def record_inference_event(self, ttft_ms: float, is_error: bool = False):
self.total_requests += 1
if is_error:
self.total_errors += 1
else:
self.ttft_samples.append(float(ttft_ms))
def evaluate_metrics(self) -> Dict[str, Any]:
if not self.ttft_samples:
p50 = p95 = p99 = 0.0
else:
p50 = float(np.percentile(self.ttft_samples, 50))
p95 = float(np.percentile(self.ttft_samples, 95))
p99 = float(np.percentile(self.ttft_samples, 99))
error_rate = (self.total_errors / self.total_requests * 100.0) if self.total_requests > 0 else 0.0
self.active_alerts.clear()
if p95 > self.ttft_threshold:
self.active_alerts.append({
"severity": "CRITICAL",
"alert_name": "HighTailLatencyP95",
"message": f"P95 TTFT ({round(p95, 1)}ms) exceeded threshold ({self.ttft_threshold}ms)",
"timestamp": time.time()
})
if error_rate > self.error_threshold:
self.active_alerts.append({
"severity": "CRITICAL",
"alert_name": "HighInferenceErrorRate",
"message": f"Error rate ({round(error_rate, 2)}%) exceeded threshold ({self.error_threshold}%)",
"timestamp": time.time()
})
return {
"total_requests": self.total_requests,
"p50_ttft_ms": round(p50, 2),
"p95_ttft_ms": round(p95, 2),
"p99_ttft_ms": round(p99, 2),
"error_rate_pct": round(error_rate, 2),
"active_alert_count": len(self.active_alerts)
}
if __name__ == "__main__":
e = AIObservabilityAlertEngine()
e.record_inference_event(100.0)
print(e.evaluate_metrics())
metadata.yml (406 bytes)
lesson_id: D339
day: 339
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/monitoring_alerting.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
pytest>=7.0.0
numpy>=1.20.0
starter/monitoring_alerting.py (1180 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import numpy as np
import time
from typing import Dict, Any, List, Optional
class AIObservabilityAlertEngine:
def __init__(self, p95_ttft_threshold_ms: float=300.0, error_rate_threshold_pct: float=1.0):
self.ttft_threshold = float(p95_ttft_threshold_ms)
self.error_threshold = float(error_rate_threshold_pct)
self.ttft_samples: List[float] = []
self.total_requests = 0
self.total_errors = 0
self.active_alerts: List[Dict[str, Any]] = []
def record_inference_event(self, ttft_ms: float, is_error: bool=False):
raise NotImplementedError('TASK 1: implement record_inference_event.')
def evaluate_metrics(self) -> Dict[str, Any]:
raise NotImplementedError('TASK 2: implement evaluate_metrics.')
if __name__ == '__main__':
e = AIObservabilityAlertEngine()
e.record_inference_event(100.0)
print(e.evaluate_metrics())
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_monitoring_alerting.py (2230 bytes)
import pytest
from examples.monitoring_alerting import AIObservabilityAlertEngine
def test_percentile_calculation():
engine = AIObservabilityAlertEngine(p95_ttft_threshold_ms=300.0)
# 100 uniform samples from 1 to 100
for val in range(1, 101):
engine.record_inference_event(float(val))
metrics = engine.evaluate_metrics()
assert metrics["p50_ttft_ms"] == 50.5
assert metrics["p95_ttft_ms"] == 95.05
assert metrics["active_alert_count"] == 0
def test_high_tail_latency_alert_trigger():
engine = AIObservabilityAlertEngine(p95_ttft_threshold_ms=200.0)
# 90 normal requests at 50ms, 10 tail requests at 500ms
for _ in range(90):
engine.record_inference_event(50.0)
for _ in range(10):
engine.record_inference_event(500.0)
metrics = engine.evaluate_metrics()
assert metrics["p95_ttft_ms"] > 200.0
assert metrics["active_alert_count"] == 1
assert engine.active_alerts[0]["alert_name"] == "HighTailLatencyP95"
assert engine.active_alerts[0]["severity"] == "CRITICAL"
def test_error_rate_alert_trigger():
engine = AIObservabilityAlertEngine(error_rate_threshold_pct=2.0)
# 95 successful, 5 errors = 5% error rate > 2% threshold
for _ in range(95):
engine.record_inference_event(100.0, is_error=False)
for _ in range(5):
engine.record_inference_event(0.0, is_error=True)
metrics = engine.evaluate_metrics()
assert metrics["error_rate_pct"] == 5.0
assert metrics["active_alert_count"] == 1
assert engine.active_alerts[0]["alert_name"] == "HighInferenceErrorRate"
def test_healthy_traffic_zero_alerts():
engine = AIObservabilityAlertEngine(p95_ttft_threshold_ms=300.0, error_rate_threshold_pct=1.0)
for _ in range(100):
engine.record_inference_event(120.0, is_error=False)
metrics = engine.evaluate_metrics()
assert metrics["active_alert_count"] == 0
assert len(engine.active_alerts) == 0
def test_empty_metrics_handling():
engine = AIObservabilityAlertEngine()
metrics = engine.evaluate_metrics()
assert metrics["total_requests"] == 0
assert metrics["p50_ttft_ms"] == 0.0
assert metrics["active_alert_count"] == 0