Capstone › Capstone Build II: Ship It › Day 360

Hands-on lab — Day 360: Monitoring and Cost Controls

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/monitoring_demo.py

Test

bash tests/run_tests.sh

File tree

examples/monitoring_demo.py
examples/monitoring.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/monitoring.py
tests/run_tests.sh
tests/test_monitoring.py
troubleshooting.md

Lab README

Lab — Day 360: Monitoring and Cost Controls

Lesson

Purpose

Build windowed metrics, alert rules and cost anomaly detection, then run them over traffic with three problems hidden in it — none of which raises a single error.

Requests carry a logical timestamp and the traffic is seeded, so every percentile and every alert is exactly reproducible.

Learning objectives

  • Compute nearest-rank percentiles and explain why the mean hides a long tail.
  • Measure a quality signal alongside availability, and see the failure only it catches.
  • Compare spend against a rolling median baseline rather than a fixed cap.
  • Refuse to judge a window too small to be evidence.
  • Distinguish symptom alerts from cause alerts.

Prerequisites

  • Day 359, "Deploying Your Capstone" — monitoring is the other half of that job.
  • Day 327, "Cost Engineering for AI Systems", for the spend concepts.
  • Comfortable with Python dataclasses, enums and list comprehensions.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.

Required software

  • Python 3.10 or newer.
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network. The lesson discusses Prometheus, Grafana, OpenTelemetry, Langfuse and Arize Phoenix as the real options — all of which compute the same four signals this lab computes with a dictionary and a sort.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/monitoring.py         your work: percentile, windows, evaluate,
                              rolling_baseline
examples/monitoring.py        reference implementation
examples/monitoring_demo.py   25 minutes of traffic with three problems in it
tests/test_monitoring.py      grouped by signal
tests/run_tests.sh            suite entry point
expected-output/              real captured output and measured values
requirements/                 pinned dependency

How to run

python3 examples/monitoring_demo.py   # five windows, three problems
bash tests/run_tests.sh               # run the suite

To work on the exercise, edit starter/monitoring.py, then copy it over the reference:

cp starter/monitoring.py examples/monitoring.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/monitoring_demo.py generates twenty-five logical minutes of seeded traffic containing a latency problem, a quality problem and a cost problem, buckets it into five-minute windows, and evaluates the alert rules against each — carrying a rolling baseline forward so the cost rule has something to compare against.

bash tests/run_tests.sh runs pytest over twenty-two tests grouped by signal: percentiles, windows, small samples, latency, errors, quality, cost and severity.

Expected output

[  0-5  ) n=20  p50=678   p95=1253   err=0% ungrounded=0% spend=$0.0300
  OK    healthy: all signals within budget
[  5-10 ) n=20  p50=906   p95=1434   err=0% ungrounded=0% spend=$0.0300
  OK    healthy: all signals within budget
[ 10-15 ) n=20  p50=5470  p95=8573   err=0% ungrounded=0% spend=$0.0300
  PAGE  latency_slo: p95 8573ms exceeds 4000ms
[ 15-20 ) n=20  p50=738   p95=1311   err=0% ungrounded=40% spend=$0.0300
  WARN  ungrounded_answers: 40% of answers cited nothing
[ 20-25 ) n=20  p50=968   p95=1365   err=0% ungrounded=0% spend=$0.4000
  PAGE  cost_anomaly: spend $0.4000 is 13.3x the $0.0300 baseline

Read the err column: zero in every window, including all three with real problems. expected-output/FIELDS.md explains each field.

Validation steps

  1. bash tests/run_tests.sh reports 22 passed.
  2. Confirm the fourth window warns on groundedness while its error rate is zero — that is the failure conventional monitoring cannot see.
  3. A window of two failed requests must report insufficient_sample, not a page. A 100 percent error rate over two requests is not evidence.
  4. A spend spike with no baseline must produce no cost alert. A first window has nothing to compare against.

Tests

Twenty-two tests in eight groups:

  • percentiles — nearest rank rather than banker's rounding; empty input; and the case showing p95 over twenty samples cannot see a single outlier while p99 can.
  • windows — bucketing by logical minute, empty traffic, and an empty window reporting zeros rather than raising.
  • small samples — a tiny window is not judged; the same rate alerts once the sample is large enough.
  • latency, errors, quality, cost — one group per rule, including that ungrounded answers warn while the error rate is zero.
  • severity — the worst severity wins, and the worst of nothing is OK.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md.

Security notes

See security.md. In short: no network, no credentials — and the lesson covers why a cost anomaly is often the first sign of abuse, and why traces are the highest-risk store in an AI system.

Extension exercises

  1. Burn-rate alerting. Alert on the rate of error-budget consumption rather than a single window, so a fast severe spike pages and a slow mild elevation warns.
  2. Cardinality protection. Allow labels but bound the number of distinct combinations, collapsing the rest into other — then demonstrate the explosion you are preventing.
  3. A quality baseline. Compare groundedness against its own rolling baseline rather than a fixed budget, so a system that is normally 85 percent grounded alerts on a drop rather than on an absolute value.
  • Lesson
  • Previous: Day 359 — Deploying Your Capstone
  • Next: Day 361 — Security Review of Your Capstone

Expected output

FIELDS.md

# Reading the monitoring output

## A window line

    [ 15-20 ) n=20  p50=738   p95=1311   err=0% ungrounded=40% spend=$0.0300

| Field | Meaning |
| --- | --- |
| `[a-b)` | The window, half-open on the right. |
| `n` | Requests in it. Below the minimum sample, the window is not judged at all. |
| `p50` / `p95` | Latency percentiles in milliseconds. Never a mean — a long tail has an unremarkable average. |
| `err` | Share of requests that failed outright. |
| `ungrounded` | Share of answers that cited nothing. **These requests succeeded.** |
| `spend` | Cost over the window, compared against a rolling median of recent windows. |

## An alert line

      PAGE  cost_anomaly: spend $0.4000 is 13.3x the $0.0300 baseline

| Part | Meaning |
| --- | --- |
| severity | `OK`, `WARN` or `PAGE`. Page for what a user feels now; warn for what needs attention today. |
| name | The rule that fired. |
| detail | The measured values, so the alert is actionable without opening a dashboard. |

## The five rules

| Rule | Severity | Catches |
| --- | --- | --- |
| `insufficient_sample` | OK | Nothing — it *prevents* judging a window too small to be evidence. |
| `latency_slo` | PAGE | p95 above the objective. |
| `error_budget` | PAGE | Failures above the agreed allowance. |
| `ungrounded_answers` | WARN | Answers citing nothing. The AI-specific failure. |
| `cost_anomaly` | PAGE | Spend far above the recent norm. Needs a baseline; a first window has none. |

## What to notice

Run the demo and read the `err` column: **it is 0% in every window**, including the three with real problems. That is the argument for measuring more than availability, made with data rather than assertion.

examples-run.txt

[  0-5  ) n=20  p50=678   p95=1253   err=0% ungrounded=0% spend=$0.0300
  OK    healthy: all signals within budget
[  5-10 ) n=20  p50=906   p95=1434   err=0% ungrounded=0% spend=$0.0300
  OK    healthy: all signals within budget
[ 10-15 ) n=20  p50=5470  p95=8573   err=0% ungrounded=0% spend=$0.0300
  PAGE  latency_slo: p95 8573ms exceeds 4000ms
[ 15-20 ) n=20  p50=738   p95=1311   err=0% ungrounded=40% spend=$0.0300
  WARN  ungrounded_answers: 40% of answers cited nothing
[ 20-25 ) n=20  p50=968   p95=1365   err=0% ungrounded=0% spend=$0.4000
  PAGE  cost_anomaly: spend $0.4000 is 13.3x the $0.0300 baseline

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/monitoring_demo.py -> 10 lines, exit 0
  bash tests/run_tests.sh             -> 22 passed in 0.02s

Traffic is generated from a seeded RNG, so every percentile below is exactly
reproducible.

  window   n   p50    p95    err  ungrounded  spend     alert
  0-5      20   678   1253    0%      0%      $0.0300   healthy
  5-10     20   906   1434    0%      0%      $0.0300   healthy
  10-15    20  5470   8573    0%      0%      $0.0300   PAGE latency_slo
  15-20    20   738   1311    0%     40%      $0.0300   WARN ungrounded
  20-25    20   968   1365    0%      0%      $0.4000   PAGE cost_anomaly

THE ERROR RATE IS ZERO IN EVERY WINDOW.

Three real problems, none of them visible to availability monitoring. Window
15-20 is the one to sit with: latency fine, errors zero, spend normal, and 40%
of answers cited nothing at all. Every conventional dashboard shows green while
the system returns confident invented text to nearly half its users.

Window 20-25 shows why the baseline matters. $0.40 in a window is not obviously
wrong in isolation. It is obviously wrong at 13.3x the recent median.

run-output.txt

[  0-5  ) n=20  p50=678   p95=1253   err=0% ungrounded=0% spend=$0.0300
  OK    healthy: all signals within budget
[  5-10 ) n=20  p50=906   p95=1434   err=0% ungrounded=0% spend=$0.0300
  OK    healthy: all signals within budget
[ 10-15 ) n=20  p50=5470  p95=8573   err=0% ungrounded=0% spend=$0.0300
  PAGE  latency_slo: p95 8573ms exceeds 4000ms
[ 15-20 ) n=20  p50=738   p95=1311   err=0% ungrounded=40% spend=$0.0300
  WARN  ungrounded_answers: 40% of answers cited nothing
[ 20-25 ) n=20  p50=968   p95=1365   err=0% ungrounded=0% spend=$0.4000
  PAGE  cost_anomaly: spend $0.4000 is 13.3x the $0.0300 baseline

starter-run.txt

An untouched starter fails nearly the whole suite:

    21 failed, 1 passed in 0.07s

The one that passes is test_worst_of_nothing_is_ok, which exercises worst() on
an empty list -- worst() is not stubbed. Everything else depends on percentile,
windows, evaluate or rolling_baseline.

test-output.txt

......................                                                   [100%]
22 passed in 0.01s

test-run.txt

......................                                                   [100%]
22 passed in 0.01s

Source files

examples/monitoring_demo.py (1762 bytes)
#!/usr/bin/env python3
"""Twenty-five minutes of traffic with three problems buried in it."""

from __future__ import annotations

import os
import random
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from monitoring import Request, evaluate, rolling_baseline, windows


def build_traffic() -> list[Request]:
    """Five windows: two healthy, then latency, quality and cost problems."""
    rng = random.Random(11)
    out: list[Request] = []

    def batch(at_from: int, n: int, *, slow=False, errors=0.0, ungrounded=0.0, pricey=False):
        for i in range(n):
            at = at_from + i % 5
            latency = rng.randint(3800, 9000) if slow else rng.randint(300, 1500)
            out.append(
                Request(
                    at=at,
                    latency_ms=latency,
                    ok=rng.random() >= errors,
                    cost=(0.02 if pricey else 0.0015),
                    tokens=rng.randint(400, 900),
                    grounded=rng.random() >= ungrounded,
                )
            )

    batch(0, 20)                      # healthy
    batch(5, 20)                      # healthy
    batch(10, 20, slow=True)          # latency problem
    batch(15, 20, ungrounded=0.45)    # quality problem, all 200s
    batch(20, 20, pricey=True)        # cost problem
    return out


def main() -> int:
    traffic = build_traffic()
    history = []
    for window in windows(traffic, size=5):
        baseline = rolling_baseline(history)
        alerts = evaluate(window, baseline_spend=baseline)
        print(window.line())
        for alert in alerts:
            print(alert.line())
        history.append(window)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/monitoring.py (6587 bytes)
"""Windowed metrics, alert rules and cost anomaly detection for an AI service.

Offline and standard-library only. Requests are records with a logical
timestamp, so every percentile and every alert is exactly reproducible.

The shape of the problem is different from ordinary web monitoring in three
ways, and each drives something here:

  cost is per request      so spend is a first-class metric, not a monthly bill
  failure is often partial a request can succeed and still be useless
  latency is long-tailed   so the mean is actively misleading

Two principles run through the module:

  * Alert on SYMPTOMS a user would notice, not on causes. "p95 latency is
    above 4s" is actionable; "cache hit rate fell" may be entirely fine.
  * Compare against a BASELINE, not a fixed threshold, wherever normal varies.
    A fixed spend limit is either always breached or never useful.
"""

from __future__ import annotations

import math
from dataclasses import dataclass, field
from enum import Enum


class Severity(str, Enum):
    OK = "ok"
    WARN = "warn"
    PAGE = "page"


@dataclass(frozen=True)
class Request:
    """One served request."""

    at: int  # logical minute
    latency_ms: int
    ok: bool
    cost: float
    tokens: int
    grounded: bool = True  # did the answer cite retrieved context?


@dataclass
class Window:
    """Metrics over one slice of time."""

    start: int
    end: int
    requests: list[Request] = field(default_factory=list)

    @property
    def count(self) -> int:
        return len(self.requests)

    @property
    def error_rate(self) -> float:
        if not self.requests:
            return 0.0
        return sum(1 for r in self.requests if not r.ok) / len(self.requests)

    @property
    def ungrounded_rate(self) -> float:
        """Answers that cited nothing.

        A quality signal rather than an availability one. These requests
        succeeded -- they returned 200 and a fluent answer -- and are still a
        failure of the thing the system exists to do.
        """
        if not self.requests:
            return 0.0
        return sum(1 for r in self.requests if not r.grounded) / len(self.requests)

    @property
    def spend(self) -> float:
        return sum(r.cost for r in self.requests)

    def latency(self, p: float) -> int:
        return percentile([r.latency_ms for r in self.requests], p)

    def line(self) -> str:
        return (
            f"[{self.start:>3}-{self.end:<3}) n={self.count:<3} "
            f"p50={self.latency(50):<5} p95={self.latency(95):<6} "
            f"err={self.error_rate:.0%} ungrounded={self.ungrounded_rate:.0%} "
            f"spend=${self.spend:.4f}"
        )


def percentile(values: list[int], p: float) -> int:
    """Nearest-rank percentile. Zero for an empty input.

    `math.ceil`, not `round`: Python rounds halves to even, which puts the p50
    of a five-element list on the wrong observation.
    """
    if not values:
        return 0
    ordered = sorted(values)
    rank = max(1, min(len(ordered), math.ceil(p / 100 * len(ordered))))
    return ordered[rank - 1]


def windows(requests: list[Request], *, size: int = 5) -> list[Window]:
    """Bucket requests into fixed windows by their logical minute."""
    if not requests:
        return []
    lo = min(r.at for r in requests)
    hi = max(r.at for r in requests)
    out: list[Window] = []
    for start in range(lo, hi + 1, size):
        end = start + size
        out.append(
            Window(start, end, [r for r in requests if start <= r.at < end])
        )
    return out


@dataclass
class Alert:
    name: str
    severity: Severity
    detail: str

    def line(self) -> str:
        return f"  {self.severity.value.upper():<5} {self.name}: {self.detail}"


def evaluate(
    window: Window,
    *,
    baseline_spend: float | None = None,
    latency_slo_ms: int = 4000,
    error_budget: float = 0.02,
    ungrounded_budget: float = 0.10,
    spend_multiple: float = 3.0,
    min_sample: int = 10,
) -> list[Alert]:
    """Apply the alert rules to one window.

    `min_sample` matters more than it looks. On a small window a single failed
    request is a 20% error rate, and paging on that trains people to ignore
    pages -- which is worse than not alerting at all.
    """
    alerts: list[Alert] = []

    if window.count < min_sample:
        return [
            Alert(
                "insufficient_sample",
                Severity.OK,
                f"{window.count} requests is below the {min_sample} needed to judge",
            )
        ]

    p95 = window.latency(95)
    if p95 > latency_slo_ms:
        alerts.append(
            Alert("latency_slo", Severity.PAGE, f"p95 {p95}ms exceeds {latency_slo_ms}ms")
        )

    if window.error_rate > error_budget:
        alerts.append(
            Alert(
                "error_budget",
                Severity.PAGE,
                f"{window.error_rate:.0%} errors exceeds {error_budget:.0%}",
            )
        )

    # A quality symptom, not an availability one. These requests SUCCEEDED.
    if window.ungrounded_rate > ungrounded_budget:
        alerts.append(
            Alert(
                "ungrounded_answers",
                Severity.WARN,
                f"{window.ungrounded_rate:.0%} of answers cited nothing",
            )
        )

    # Spend is compared against what normal looked like, not a fixed cap.
    if baseline_spend is not None and baseline_spend > 0:
        ratio = window.spend / baseline_spend
        if ratio > spend_multiple:
            alerts.append(
                Alert(
                    "cost_anomaly",
                    Severity.PAGE,
                    f"spend ${window.spend:.4f} is {ratio:.1f}x the ${baseline_spend:.4f} baseline",
                )
            )

    return alerts or [Alert("healthy", Severity.OK, "all signals within budget")]


def worst(alerts: list[Alert]) -> Severity:
    order = {Severity.OK: 0, Severity.WARN: 1, Severity.PAGE: 2}
    return max((a.severity for a in alerts), key=lambda s: order[s], default=Severity.OK)


def rolling_baseline(history: list[Window], *, keep: int = 3) -> float | None:
    """Median spend of the last `keep` windows.

    Median rather than mean: one anomalous window should not raise the baseline
    enough to hide the next one, which is exactly how a spend alert stops
    firing after the first incident.
    """
    recent = [w.spend for w in history[-keep:] if w.count]
    if not recent:
        return None
    ordered = sorted(recent)
    return ordered[len(ordered) // 2]
metadata.yml (875 bytes)
lesson_id: D360
day: 360
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/monitoring_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3
  examples/monitoring_demo.py reproduced the documented five windows exactly (traffic is seeded, so the
  percentiles are deterministic); bash tests/run_tests.sh -> 22 passed in 0.02s. The untouched starter
  fails 21 of 22; the 1 that passes exercises worst() on an empty list, which is not stubbed.
  requires_network is false: requests are synthetic records with a logical timestamp and no test opens a
  socket.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/monitoring.py (6096 bytes)
"""Windowed metrics, alert rules and cost anomaly detection for an AI service.

Offline and standard-library only. Requests are records with a logical
timestamp, so every percentile and every alert is exactly reproducible.

The shape of the problem is different from ordinary web monitoring in three
ways, and each drives something here:

  cost is per request      so spend is a first-class metric, not a monthly bill
  failure is often partial a request can succeed and still be useless
  latency is long-tailed   so the mean is actively misleading

Two principles run through the module:

  * Alert on SYMPTOMS a user would notice, not on causes. "p95 latency is
    above 4s" is actionable; "cache hit rate fell" may be entirely fine.
  * Compare against a BASELINE, not a fixed threshold, wherever normal varies.
    A fixed spend limit is either always breached or never useful.
"""

from __future__ import annotations

import math
from dataclasses import dataclass, field
from enum import Enum


class Severity(str, Enum):
    OK = "ok"
    WARN = "warn"
    PAGE = "page"


@dataclass(frozen=True)
class Request:
    """One served request."""

    at: int  # logical minute
    latency_ms: int
    ok: bool
    cost: float
    tokens: int
    grounded: bool = True  # did the answer cite retrieved context?


@dataclass
class Window:
    """Metrics over one slice of time."""

    start: int
    end: int
    requests: list[Request] = field(default_factory=list)

    @property
    def count(self) -> int:
        return len(self.requests)

    @property
    def error_rate(self) -> float:
        if not self.requests:
            return 0.0
        return sum(1 for r in self.requests if not r.ok) / len(self.requests)

    @property
    def ungrounded_rate(self) -> float:
        """Answers that cited nothing.

        A quality signal rather than an availability one. These requests
        succeeded -- they returned 200 and a fluent answer -- and are still a
        failure of the thing the system exists to do.
        """
        if not self.requests:
            return 0.0
        return sum(1 for r in self.requests if not r.grounded) / len(self.requests)

    @property
    def spend(self) -> float:
        return sum(r.cost for r in self.requests)

    def latency(self, p: float) -> int:
        return percentile([r.latency_ms for r in self.requests], p)

    def line(self) -> str:
        return (
            f"[{self.start:>3}-{self.end:<3}) n={self.count:<3} "
            f"p50={self.latency(50):<5} p95={self.latency(95):<6} "
            f"err={self.error_rate:.0%} ungrounded={self.ungrounded_rate:.0%} "
            f"spend=${self.spend:.4f}"
        )


def percentile(values: list[int], p: float) -> int:
    """Nearest-rank percentile. Zero for an empty input.

    `math.ceil`, not `round`: Python rounds halves to even, which puts the p50
    of a five-element list on the wrong observation.
    """
    # TASK 1: nearest-rank percentile; 0 for an empty list.
    # rank = ceil(p/100 * n), clamped to [1, n], then ordered[rank-1].
    # math.ceil, NOT round -- Python rounds halves to even, which puts the p50
    # of a five-element list on the wrong observation.
    raise NotImplementedError("implement percentile")

def windows(requests: list[Request], *, size: int = 5) -> list[Window]:
    """Bucket requests into fixed windows by their logical minute."""
    # TASK 2: bucket requests into fixed windows of `size` logical minutes,
    # from the earliest to the latest timestamp. Empty input gives no windows;
    # a window with no requests in it is still a window.
    raise NotImplementedError("implement windows")

@dataclass
class Alert:
    name: str
    severity: Severity
    detail: str

    def line(self) -> str:
        return f"  {self.severity.value.upper():<5} {self.name}: {self.detail}"


def evaluate(
    window: Window,
    *,
    baseline_spend: float | None = None,
    latency_slo_ms: int = 4000,
    error_budget: float = 0.02,
    ungrounded_budget: float = 0.10,
    spend_multiple: float = 3.0,
    min_sample: int = 10,
) -> list[Alert]:
    """Apply the alert rules to one window.

    `min_sample` matters more than it looks. On a small window a single failed
    request is a 20% error rate, and paging on that trains people to ignore
    pages -- which is worse than not alerting at all.
    """
    # TASK 3: apply the alert rules to one window.
    #   - FIRST: if window.count < min_sample, return a single OK
    #     "insufficient_sample" alert and nothing else. One failure in three
    #     requests is a 33% error rate, and paging on that trains people to
    #     ignore the channel.
    #   - p95 latency over latency_slo_ms      -> PAGE "latency_slo"
    #   - error_rate over error_budget         -> PAGE "error_budget"
    #   - ungrounded_rate over its budget      -> WARN "ungrounded_answers"
    #     (note these requests SUCCEEDED -- that is the whole point)
    #   - spend over baseline_spend * spend_multiple -> PAGE "cost_anomaly",
    #     but only when a baseline exists. Compare against normal, never a
    #     fixed cap.
    #   - if nothing fired, return one OK "healthy" alert.
    raise NotImplementedError("implement evaluate")

def worst(alerts: list[Alert]) -> Severity:
    order = {Severity.OK: 0, Severity.WARN: 1, Severity.PAGE: 2}
    return max((a.severity for a in alerts), key=lambda s: order[s], default=Severity.OK)


def rolling_baseline(history: list[Window], *, keep: int = 3) -> float | None:
    """Median spend of the last `keep` windows.

    Median rather than mean: one anomalous window should not raise the baseline
    enough to hide the next one, which is exactly how a spend alert stops
    firing after the first incident.
    """
    # TASK 4: median spend of the last `keep` non-empty windows, or None.
    # MEDIAN, not mean: one anomalous window must not raise the baseline enough
    # to hide the next one, which is exactly how a spend alert stops firing
    # after the first incident.
    raise NotImplementedError("implement rolling_baseline")
tests/run_tests.sh (200 bytes)
#!/usr/bin/env bash
# Run the day 360 monitoring test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_monitoring.py (6236 bytes)
"""Grouped by signal, so a failure names which alert stopped working.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from monitoring import (  # noqa: E402
    Request,
    Severity,
    Window,
    evaluate,
    percentile,
    rolling_baseline,
    windows,
    worst,
)


def reqs(n, *, latency=500, ok=True, cost=0.001, grounded=True, at=0):
    return [Request(at=at, latency_ms=latency, ok=ok, cost=cost, tokens=500,
                    grounded=grounded) for _ in range(n)]


def names(alerts):
    return {a.name for a in alerts}


# --------------------------------------------------------------- percentiles


def test_percentile_uses_nearest_rank_not_banker_rounding():
    # round(2.5) is 2 in Python, which would pick the wrong observation.
    assert percentile([1, 2, 3, 4, 100], 50) == 3
    assert percentile([1, 2, 3, 4, 100], 95) == 100


def test_percentile_of_nothing_is_zero():
    assert percentile([], 95) == 0


def test_p95_is_not_the_mean():
    # A tenth of requests are slow. The mean stays comfortable; p95 does not.
    values = [200] * 18 + [9000] * 2
    assert sum(values) / len(values) < 1100
    assert percentile(values, 95) == 9000


def test_p95_cannot_see_a_single_outlier_in_twenty_samples():
    # Worth knowing rather than discovering during an incident. Nearest rank
    # for p95 over 20 samples is the 19th value, so ONE slow request sits above
    # the percentile entirely and is invisible to it.
    values = [200] * 19 + [9000]
    assert percentile(values, 95) == 200
    # p99 over the same data is the 20th, which does see it.
    assert percentile(values, 99) == 9000
    # The general rule: a percentile can only see events more frequent than
    # 1 - p. Choose the percentile to match what you need to catch, and make
    # sure the window is large enough to contain them.


# ------------------------------------------------------------------ windows


def test_requests_are_bucketed_by_logical_minute():
    traffic = reqs(3, at=0) + reqs(3, at=6) + reqs(3, at=11)
    got = windows(traffic, size=5)
    assert [w.count for w in got] == [3, 3, 3]
    assert [(w.start, w.end) for w in got] == [(0, 5), (5, 10), (10, 15)]


def test_empty_traffic_produces_no_windows():
    assert windows([]) == []


def test_a_window_with_no_requests_reports_zero_not_an_error():
    w = Window(0, 5, [])
    assert w.error_rate == 0.0 and w.spend == 0.0 and w.latency(95) == 0


# ------------------------------------------------------------ small samples


def test_a_small_window_is_not_judged():
    # One failure in three requests is a 33% error rate. Paging on that trains
    # people to ignore pages, which is worse than not alerting.
    w = Window(0, 5, reqs(2, ok=False))
    alerts = evaluate(w)
    assert names(alerts) == {"insufficient_sample"}
    assert worst(alerts) is Severity.OK


def test_the_same_rate_alerts_once_the_sample_is_large_enough():
    w = Window(0, 5, reqs(10, ok=False))
    assert "error_budget" in names(evaluate(w))


# ------------------------------------------------------------------ latency


def test_latency_over_the_slo_pages():
    w = Window(0, 5, reqs(20, latency=9000))
    alerts = evaluate(w)
    assert "latency_slo" in names(alerts)
    assert worst(alerts) is Severity.PAGE


def test_latency_inside_the_slo_is_healthy():
    w = Window(0, 5, reqs(20, latency=900))
    assert names(evaluate(w)) == {"healthy"}


# ------------------------------------------------------------------- errors


def test_error_rate_over_budget_pages():
    w = Window(0, 5, reqs(10, ok=False) + reqs(10))
    assert "error_budget" in names(evaluate(w))


def test_error_rate_inside_budget_is_healthy():
    w = Window(0, 5, reqs(100))
    assert names(evaluate(w)) == {"healthy"}


# ------------------------------------------------------------------ quality


def test_ungrounded_answers_warn_even_though_every_request_succeeded():
    # The distinguishing case for an AI service: 100% availability, and the
    # answers cite nothing. Conventional monitoring reports this as healthy.
    w = Window(0, 5, reqs(10, grounded=False) + reqs(10))
    alerts = evaluate(w)
    assert w.error_rate == 0.0
    assert "ungrounded_answers" in names(alerts)
    assert worst(alerts) is Severity.WARN


def test_grounded_answers_do_not_warn():
    w = Window(0, 5, reqs(20, grounded=True))
    assert "ungrounded_answers" not in names(evaluate(w))


# --------------------------------------------------------------------- cost


def test_a_spend_spike_against_the_baseline_pages():
    w = Window(0, 5, reqs(20, cost=0.02))
    alerts = evaluate(w, baseline_spend=0.03)
    assert "cost_anomaly" in names(alerts)
    assert "13.3x" in next(a for a in alerts if a.name == "cost_anomaly").detail


def test_normal_spend_against_the_baseline_is_healthy():
    w = Window(0, 5, reqs(20, cost=0.0015))
    assert names(evaluate(w, baseline_spend=0.03)) == {"healthy"}


def test_no_baseline_means_no_cost_alert():
    # A first window has nothing to compare against, and inventing a fixed
    # threshold would be either always breached or never useful.
    w = Window(0, 5, reqs(20, cost=0.02))
    assert "cost_anomaly" not in names(evaluate(w, baseline_spend=None))


def test_baseline_is_a_median_so_one_spike_does_not_hide_the_next():
    normal = Window(0, 5, reqs(20, cost=0.0015))
    spike = Window(5, 10, reqs(20, cost=0.02))
    # Mean of these would be pulled up by the spike; the median is not.
    assert rolling_baseline([normal, normal, spike]) == normal.spend


def test_rolling_baseline_ignores_empty_windows():
    assert rolling_baseline([Window(0, 5, [])]) is None


# ------------------------------------------------------------------ severity


def test_worst_severity_wins():
    w = Window(0, 5, reqs(10, latency=9000, grounded=False) + reqs(10, latency=9000))
    alerts = evaluate(w)
    assert names(alerts) >= {"latency_slo", "ungrounded_answers"}
    assert worst(alerts) is Severity.PAGE


def test_worst_of_nothing_is_ok():
    assert worst([]) is Severity.OK

Troubleshooting

Troubleshooting — Day 360

Every small window pages

You are evaluating the rules before checking the sample size. The min_sample check must come first and return immediately — one failure in three requests is a 33 percent error rate, and paging on it is how a channel gets muted.

The p50 of [1, 2, 3, 4, 100] comes out as 2

You used round for the nearest-rank calculation. Python rounds halves to even, so round(2.5) is 2 and you land on the wrong observation. Use math.ceil.

test_p95_cannot_see_a_single_outlier_in_twenty_samples fails

Read the test before changing the code — it asserts that p95 over twenty samples is 200, not 9000, and that is correct. Nearest rank puts p95 at the nineteenth value, so a single outlier sits above it. If your implementation returns 9000 you are computing a maximum rather than a percentile.

The cost alert stops firing after the first spike

Your baseline is a mean, and the spike raised it enough to make the next one look normal. Use a median, which is unmoved by a single outlier. This failure is self-concealing, which is why it has its own test.

A spend spike alerts on the very first window

There is no baseline yet, so there is nothing to compare against. rolling_baseline should return None for an empty history, and evaluate should skip the cost rule when the baseline is None. Inventing a fixed threshold instead would either never fire or always fire.

ZeroDivisionError on an empty window

A window with no requests is a normal state, particularly at the edges of a traffic period. Guard every denominator and return zeros.

Ungrounded answers never warn

Check that your generator is actually producing them, and that the rule compares against ungrounded_budget rather than the error budget. The point of this rule is that it fires while error_rate is zero — if both move together, you are measuring failures rather than quality.

The demo numbers do not match

Traffic is generated from a seeded RNG, so a correct implementation reproduces the table exactly. Differences usually come from the percentile rank calculation or from bucketing boundaries — windows are half-open, start <= at < end.

NotImplementedError on nearly every test

Expected. The starter stubs four functions — see expected-output/starter-run.txt, which also names the one test that passes without them.

Security notes

Security notes — Day 360

What this lab does and does not touch

  • No network. Requests are synthetic records with a logical timestamp. No test opens a socket.
  • No credentials. No API key, token or account.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/.
  • No sudo, no global installs.
  • No real user data. The request records carry latency, cost and boolean flags — nothing identifying.

A cost alert is a security control

The cost_anomaly rule looks like budgeting. In production it is frequently the first detection of an attack, and often the only one, because the traffic is otherwise well-formed.

Three cases it catches that nothing else does:

  • A leaked key. Someone else is using your quota. Requests succeed, latency is fine, errors are zero — spend is the only signal that moves.
  • A scripted client. A denial-of-wallet attempt, deliberately staying under rate limits so the requests look legitimate individually.
  • Prompt injection inducing recursion. Text inside a retrieved document makes an agent call itself. Every individual call is valid.

This is why the rule pages rather than warns, and why it compares against a baseline: an attacker operating just above normal is exactly what a fixed threshold set generously will miss.

Traces are the highest-risk store you own

Worth stating plainly, because it is easy to miss while thinking about metrics.

Metrics are aggregates — counts, percentiles, sums. They are safe to retain, cheap to store, and contain no personal data.

Traces are not. A trace captures the full prompt and the full completion, which means it contains whatever the user typed and whatever the system said back. In most AI systems traces are:

  • the store with the most sensitive content,
  • retained the longest, often by default rather than by decision,
  • examined the least, so nobody notices what accumulated,
  • and the one most often forgotten in a deletion path.

Everything from Day 328 applies to them. Set a retention period deliberately, include traces in the erasure path, and redact before writing rather than after.

Cardinality is an availability problem

A label per user or per document identifier produces millions of distinct series. The consequence is not a bad dashboard — it is a metrics backend that becomes slow, then expensive, then unavailable, usually during the incident when you need it. Keep labels to a small bounded set, and cap them (extension exercise 2).

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.