Deployment, MLOps, and Security › Operating AI in Production › Day 339

Hands-on lab — Day 339: Monitoring and Alerting

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/monitoring_alerting.py

Test

bash tests/run_tests.sh

File tree

examples/monitoring_alerting.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/monitoring_alerting.py
tests/run_tests.sh
tests/test_monitoring_alerting.py
troubleshooting.md

Lab README

Day 339 Lab: Monitoring and Alerting

Lesson

  • Lesson title: Monitoring and Alerting
  • Day number: 339 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-339-monitoring-and-alerting
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-339-monitoring-and-alerting when the site is running.

Purpose

Build an AI Observability and Alerting Engine in Python calculating rolling percentile latencies, evaluating SRE Golden Signals, and triggering actionable alerts.

Learning objectives

  • Record inference latency samples and error outcomes.
  • Compute rolling percentiles (P50, P95, P99).
  • Evaluate error rate percentages against SLA thresholds.
  • Trigger structured critical alerts on SLA breaches.

Prerequisites

  • Python 3.10+ installed
  • pytest, numpy installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest, numpy

Free and open-source options

  • Python Standard Library, Pytest, NumPy

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/monitoring_alerting.py: Starter implementation skeleton
  • examples/monitoring_alerting.py: Verified reference implementation
  • tests/test_monitoring_alerting.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/monitoring_alerting.py

What the commands do

  • Evaluates percentile latency distributions and verifies alert trigger thresholds.

Expected output

{'total_requests': 1, 'p50_ttft_ms': 100.0, 'p95_ttft_ms': 100.0, 'p99_ttft_ms': 100.0, 'error_rate_pct': 0.0, 'active_alert_count': 0}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Correct P50, P95, and P99 calculation on uniform and heavy-tailed samples
  • Triggering HighTailLatencyP95 alert when threshold is exceeded
  • Triggering HighInferenceErrorRate alert when error rate spikes
  • Zero alert activation on healthy baseline traffic
  • Graceful empty metrics handling

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure numpy is imported for percentile calculation.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement alert resolution cooldown hysteresis.

Day number: 339 of 365

Expected output

FIELDS.md

# Measured Values
- `metrics_evaluated`: 12
- `alerts_fired`: 2

examples-run.txt

{'total_requests': 1, 'p50_ttft_ms': 100.0, 'p95_ttft_ms': 100.0, 'p99_ttft_ms': 100.0, 'error_rate_pct': 0.0, 'active_alert_count': 0}

measured-values.txt

metrics_evaluated=12
alerts_fired=2

run-output.txt

{'total_requests': 1, 'p50_ttft_ms': 100.0, 'p95_ttft_ms': 100.0, 'p99_ttft_ms': 100.0, 'error_rate_pct': 0.0, 'active_alert_count': 0}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <examples.monitoring_alerting.AIObservabilityAlertEngine object at 0x10719e650>

    def evaluate_metrics(self) -> Dict[str, Any]:
>       raise NotImplementedError('TASK 2: implement evaluate_metrics.')
E       NotImplementedError: TASK 2: implement evaluate_metrics.

examples/monitoring_alerting.py:26: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_monitoring_alerting.py::test_percentile_calculation - NotIm...
FAILED tests/test_monitoring_alerting.py::test_high_tail_latency_alert_trigger
FAILED tests/test_monitoring_alerting.py::test_error_rate_alert_trigger - Not...
FAILED tests/test_monitoring_alerting.py::test_healthy_traffic_zero_alerts - ...
FAILED tests/test_monitoring_alerting.py::test_empty_metrics_handling - NotIm...
============================== 5 failed in 0.06s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-339-monitoring-and-alerting
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_monitoring_alerting.py::test_percentile_calculation PASSED    [ 20%]
tests/test_monitoring_alerting.py::test_high_tail_latency_alert_trigger PASSED [ 40%]
tests/test_monitoring_alerting.py::test_error_rate_alert_trigger PASSED  [ 60%]
tests/test_monitoring_alerting.py::test_healthy_traffic_zero_alerts PASSED [ 80%]
tests/test_monitoring_alerting.py::test_empty_metrics_handling PASSED    [100%]

============================== 5 passed in 0.04s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-339-monitoring-and-alerting
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_monitoring_alerting.py::test_percentile_calculation PASSED    [ 20%]
tests/test_monitoring_alerting.py::test_high_tail_latency_alert_trigger PASSED [ 40%]
tests/test_monitoring_alerting.py::test_error_rate_alert_trigger PASSED  [ 60%]
tests/test_monitoring_alerting.py::test_healthy_traffic_zero_alerts PASSED [ 80%]
tests/test_monitoring_alerting.py::test_empty_metrics_handling PASSED    [100%]

============================== 5 passed in 0.04s ===============================

Source files

examples/monitoring_alerting.py (2323 bytes)
import numpy as np
import time
from typing import Dict, Any, List, Optional

class AIObservabilityAlertEngine:
    def __init__(self, p95_ttft_threshold_ms: float = 300.0, error_rate_threshold_pct: float = 1.0):
        self.ttft_threshold = float(p95_ttft_threshold_ms)
        self.error_threshold = float(error_rate_threshold_pct)
        self.ttft_samples: List[float] = []
        self.total_requests = 0
        self.total_errors = 0
        self.active_alerts: List[Dict[str, Any]] = []

    def record_inference_event(self, ttft_ms: float, is_error: bool = False):
        self.total_requests += 1
        if is_error:
            self.total_errors += 1
        else:
            self.ttft_samples.append(float(ttft_ms))

    def evaluate_metrics(self) -> Dict[str, Any]:
        if not self.ttft_samples:
            p50 = p95 = p99 = 0.0
        else:
            p50 = float(np.percentile(self.ttft_samples, 50))
            p95 = float(np.percentile(self.ttft_samples, 95))
            p99 = float(np.percentile(self.ttft_samples, 99))

        error_rate = (self.total_errors / self.total_requests * 100.0) if self.total_requests > 0 else 0.0

        self.active_alerts.clear()
        if p95 > self.ttft_threshold:
            self.active_alerts.append({
                "severity": "CRITICAL",
                "alert_name": "HighTailLatencyP95",
                "message": f"P95 TTFT ({round(p95, 1)}ms) exceeded threshold ({self.ttft_threshold}ms)",
                "timestamp": time.time()
            })

        if error_rate > self.error_threshold:
            self.active_alerts.append({
                "severity": "CRITICAL",
                "alert_name": "HighInferenceErrorRate",
                "message": f"Error rate ({round(error_rate, 2)}%) exceeded threshold ({self.error_threshold}%)",
                "timestamp": time.time()
            })

        return {
            "total_requests": self.total_requests,
            "p50_ttft_ms": round(p50, 2),
            "p95_ttft_ms": round(p95, 2),
            "p99_ttft_ms": round(p99, 2),
            "error_rate_pct": round(error_rate, 2),
            "active_alert_count": len(self.active_alerts)
        }

if __name__ == "__main__":
    e = AIObservabilityAlertEngine()
    e.record_inference_event(100.0)
    print(e.evaluate_metrics())
metadata.yml (406 bytes)
lesson_id: D339
day: 339
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/monitoring_alerting.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
pytest>=7.0.0
numpy>=1.20.0
starter/monitoring_alerting.py (1180 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import numpy as np
import time
from typing import Dict, Any, List, Optional

class AIObservabilityAlertEngine:

    def __init__(self, p95_ttft_threshold_ms: float=300.0, error_rate_threshold_pct: float=1.0):
        self.ttft_threshold = float(p95_ttft_threshold_ms)
        self.error_threshold = float(error_rate_threshold_pct)
        self.ttft_samples: List[float] = []
        self.total_requests = 0
        self.total_errors = 0
        self.active_alerts: List[Dict[str, Any]] = []

    def record_inference_event(self, ttft_ms: float, is_error: bool=False):
        raise NotImplementedError('TASK 1: implement record_inference_event.')

    def evaluate_metrics(self) -> Dict[str, Any]:
        raise NotImplementedError('TASK 2: implement evaluate_metrics.')
if __name__ == '__main__':
    e = AIObservabilityAlertEngine()
    e.record_inference_event(100.0)
    print(e.evaluate_metrics())
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_monitoring_alerting.py (2230 bytes)
import pytest
from examples.monitoring_alerting import AIObservabilityAlertEngine

def test_percentile_calculation():
    engine = AIObservabilityAlertEngine(p95_ttft_threshold_ms=300.0)
    # 100 uniform samples from 1 to 100
    for val in range(1, 101):
        engine.record_inference_event(float(val))
        
    metrics = engine.evaluate_metrics()
    assert metrics["p50_ttft_ms"] == 50.5
    assert metrics["p95_ttft_ms"] == 95.05
    assert metrics["active_alert_count"] == 0

def test_high_tail_latency_alert_trigger():
    engine = AIObservabilityAlertEngine(p95_ttft_threshold_ms=200.0)
    # 90 normal requests at 50ms, 10 tail requests at 500ms
    for _ in range(90):
        engine.record_inference_event(50.0)
    for _ in range(10):
        engine.record_inference_event(500.0)
        
    metrics = engine.evaluate_metrics()
    assert metrics["p95_ttft_ms"] > 200.0
    assert metrics["active_alert_count"] == 1
    assert engine.active_alerts[0]["alert_name"] == "HighTailLatencyP95"
    assert engine.active_alerts[0]["severity"] == "CRITICAL"

def test_error_rate_alert_trigger():
    engine = AIObservabilityAlertEngine(error_rate_threshold_pct=2.0)
    # 95 successful, 5 errors = 5% error rate > 2% threshold
    for _ in range(95):
        engine.record_inference_event(100.0, is_error=False)
    for _ in range(5):
        engine.record_inference_event(0.0, is_error=True)
        
    metrics = engine.evaluate_metrics()
    assert metrics["error_rate_pct"] == 5.0
    assert metrics["active_alert_count"] == 1
    assert engine.active_alerts[0]["alert_name"] == "HighInferenceErrorRate"

def test_healthy_traffic_zero_alerts():
    engine = AIObservabilityAlertEngine(p95_ttft_threshold_ms=300.0, error_rate_threshold_pct=1.0)
    for _ in range(100):
        engine.record_inference_event(120.0, is_error=False)
        
    metrics = engine.evaluate_metrics()
    assert metrics["active_alert_count"] == 0
    assert len(engine.active_alerts) == 0

def test_empty_metrics_handling():
    engine = AIObservabilityAlertEngine()
    metrics = engine.evaluate_metrics()
    assert metrics["total_requests"] == 0
    assert metrics["p50_ttft_ms"] == 0.0
    assert metrics["active_alert_count"] == 0

Troubleshooting

Troubleshooting - Day 339\nEnsure latency percentiles are computed over non-error samples.\n

Security notes

Security - Day 339\nMetrics streams enforce low-cardinality tags and strictly scrub PII from metric labels.\n