Capstone › Capstone Build II: Ship It › Day 360
Hands-on lab — Day 360: Monitoring and Cost Controls
- ← Back to the Day 360 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/capstone/day-360-monitoring-and-cost-controls/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/monitoring_demo.py Test
bash tests/run_tests.sh File tree
examples/monitoring_demo.py examples/monitoring.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/monitoring.py tests/run_tests.sh tests/test_monitoring.py troubleshooting.md
Lab README
Lab — Day 360: Monitoring and Cost Controls
Lesson
- Lesson title: Monitoring and Cost Controls
- Day number: 360 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-360-monitoring-and-cost-controls
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-360-monitoring-and-cost-controlswhen the site is running.
Purpose
Build windowed metrics, alert rules and cost anomaly detection, then run them over traffic with three problems hidden in it — none of which raises a single error.
Requests carry a logical timestamp and the traffic is seeded, so every percentile and every alert is exactly reproducible.
Learning objectives
- Compute nearest-rank percentiles and explain why the mean hides a long tail.
- Measure a quality signal alongside availability, and see the failure only it catches.
- Compare spend against a rolling median baseline rather than a fixed cap.
- Refuse to judge a window too small to be evidence.
- Distinguish symptom alerts from cause alerts.
Prerequisites
- Day 359, "Deploying Your Capstone" — monitoring is the other half of that job.
- Day 327, "Cost Engineering for AI Systems", for the spend concepts.
- Comfortable with Python dataclasses, enums and list comprehensions.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.
Hardware requirements
Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.
Required software
- Python 3.10 or newer.
pytest7 or newer.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network. The lesson discusses Prometheus, Grafana, OpenTelemetry, Langfuse and Arize Phoenix as the real options — all of which compute the same four signals this lab computes with a dictionary and a sort.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/monitoring.py your work: percentile, windows, evaluate,
rolling_baseline
examples/monitoring.py reference implementation
examples/monitoring_demo.py 25 minutes of traffic with three problems in it
tests/test_monitoring.py grouped by signal
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/monitoring_demo.py # five windows, three problems
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/monitoring.py, then copy it over the reference:
cp starter/monitoring.py examples/monitoring.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/monitoring_demo.py generates twenty-five logical minutes of seeded traffic containing a latency problem, a quality problem and a cost problem, buckets it into five-minute windows, and evaluates the alert rules against each — carrying a rolling baseline forward so the cost rule has something to compare against.
bash tests/run_tests.sh runs pytest over twenty-two tests grouped by signal: percentiles, windows, small samples, latency, errors, quality, cost and severity.
Expected output
[ 0-5 ) n=20 p50=678 p95=1253 err=0% ungrounded=0% spend=$0.0300
OK healthy: all signals within budget
[ 5-10 ) n=20 p50=906 p95=1434 err=0% ungrounded=0% spend=$0.0300
OK healthy: all signals within budget
[ 10-15 ) n=20 p50=5470 p95=8573 err=0% ungrounded=0% spend=$0.0300
PAGE latency_slo: p95 8573ms exceeds 4000ms
[ 15-20 ) n=20 p50=738 p95=1311 err=0% ungrounded=40% spend=$0.0300
WARN ungrounded_answers: 40% of answers cited nothing
[ 20-25 ) n=20 p50=968 p95=1365 err=0% ungrounded=0% spend=$0.4000
PAGE cost_anomaly: spend $0.4000 is 13.3x the $0.0300 baseline
Read the err column: zero in every window, including all three with real problems. expected-output/FIELDS.md explains each field.
Validation steps
bash tests/run_tests.shreports22 passed.- Confirm the fourth window warns on groundedness while its error rate is zero — that is the failure conventional monitoring cannot see.
- A window of two failed requests must report
insufficient_sample, not a page. A 100 percent error rate over two requests is not evidence. - A spend spike with no baseline must produce no cost alert. A first window has nothing to compare against.
Tests
Twenty-two tests in eight groups:
- percentiles — nearest rank rather than banker's rounding; empty input; and the case showing p95 over twenty samples cannot see a single outlier while p99 can.
- windows — bucketing by logical minute, empty traffic, and an empty window reporting zeros rather than raising.
- small samples — a tiny window is not judged; the same rate alerts once the sample is large enough.
- latency, errors, quality, cost — one group per rule, including that ungrounded answers warn while the error rate is zero.
- severity — the worst severity wins, and the worst of nothing is OK.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md.
Security notes
See security.md. In short: no network, no credentials — and the lesson covers why a cost anomaly is often the first sign of abuse, and why traces are the highest-risk store in an AI system.
Extension exercises
- Burn-rate alerting. Alert on the rate of error-budget consumption rather than a single window, so a fast severe spike pages and a slow mild elevation warns.
- Cardinality protection. Allow labels but bound the number of distinct combinations, collapsing the rest into
other— then demonstrate the explosion you are preventing. - A quality baseline. Compare groundedness against its own rolling baseline rather than a fixed budget, so a system that is normally 85 percent grounded alerts on a drop rather than on an absolute value.
Navigation
- Lesson
- Previous: Day 359 — Deploying Your Capstone
- Next: Day 361 — Security Review of Your Capstone
Expected output
FIELDS.md
# Reading the monitoring output
## A window line
[ 15-20 ) n=20 p50=738 p95=1311 err=0% ungrounded=40% spend=$0.0300
| Field | Meaning |
| --- | --- |
| `[a-b)` | The window, half-open on the right. |
| `n` | Requests in it. Below the minimum sample, the window is not judged at all. |
| `p50` / `p95` | Latency percentiles in milliseconds. Never a mean — a long tail has an unremarkable average. |
| `err` | Share of requests that failed outright. |
| `ungrounded` | Share of answers that cited nothing. **These requests succeeded.** |
| `spend` | Cost over the window, compared against a rolling median of recent windows. |
## An alert line
PAGE cost_anomaly: spend $0.4000 is 13.3x the $0.0300 baseline
| Part | Meaning |
| --- | --- |
| severity | `OK`, `WARN` or `PAGE`. Page for what a user feels now; warn for what needs attention today. |
| name | The rule that fired. |
| detail | The measured values, so the alert is actionable without opening a dashboard. |
## The five rules
| Rule | Severity | Catches |
| --- | --- | --- |
| `insufficient_sample` | OK | Nothing — it *prevents* judging a window too small to be evidence. |
| `latency_slo` | PAGE | p95 above the objective. |
| `error_budget` | PAGE | Failures above the agreed allowance. |
| `ungrounded_answers` | WARN | Answers citing nothing. The AI-specific failure. |
| `cost_anomaly` | PAGE | Spend far above the recent norm. Needs a baseline; a first window has none. |
## What to notice
Run the demo and read the `err` column: **it is 0% in every window**, including the three with real problems. That is the argument for measuring more than availability, made with data rather than assertion.
examples-run.txt
[ 0-5 ) n=20 p50=678 p95=1253 err=0% ungrounded=0% spend=$0.0300
OK healthy: all signals within budget
[ 5-10 ) n=20 p50=906 p95=1434 err=0% ungrounded=0% spend=$0.0300
OK healthy: all signals within budget
[ 10-15 ) n=20 p50=5470 p95=8573 err=0% ungrounded=0% spend=$0.0300
PAGE latency_slo: p95 8573ms exceeds 4000ms
[ 15-20 ) n=20 p50=738 p95=1311 err=0% ungrounded=40% spend=$0.0300
WARN ungrounded_answers: 40% of answers cited nothing
[ 20-25 ) n=20 p50=968 p95=1365 err=0% ungrounded=0% spend=$0.4000
PAGE cost_anomaly: spend $0.4000 is 13.3x the $0.0300 baseline
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/monitoring_demo.py -> 10 lines, exit 0
bash tests/run_tests.sh -> 22 passed in 0.02s
Traffic is generated from a seeded RNG, so every percentile below is exactly
reproducible.
window n p50 p95 err ungrounded spend alert
0-5 20 678 1253 0% 0% $0.0300 healthy
5-10 20 906 1434 0% 0% $0.0300 healthy
10-15 20 5470 8573 0% 0% $0.0300 PAGE latency_slo
15-20 20 738 1311 0% 40% $0.0300 WARN ungrounded
20-25 20 968 1365 0% 0% $0.4000 PAGE cost_anomaly
THE ERROR RATE IS ZERO IN EVERY WINDOW.
Three real problems, none of them visible to availability monitoring. Window
15-20 is the one to sit with: latency fine, errors zero, spend normal, and 40%
of answers cited nothing at all. Every conventional dashboard shows green while
the system returns confident invented text to nearly half its users.
Window 20-25 shows why the baseline matters. $0.40 in a window is not obviously
wrong in isolation. It is obviously wrong at 13.3x the recent median.
run-output.txt
[ 0-5 ) n=20 p50=678 p95=1253 err=0% ungrounded=0% spend=$0.0300
OK healthy: all signals within budget
[ 5-10 ) n=20 p50=906 p95=1434 err=0% ungrounded=0% spend=$0.0300
OK healthy: all signals within budget
[ 10-15 ) n=20 p50=5470 p95=8573 err=0% ungrounded=0% spend=$0.0300
PAGE latency_slo: p95 8573ms exceeds 4000ms
[ 15-20 ) n=20 p50=738 p95=1311 err=0% ungrounded=40% spend=$0.0300
WARN ungrounded_answers: 40% of answers cited nothing
[ 20-25 ) n=20 p50=968 p95=1365 err=0% ungrounded=0% spend=$0.4000
PAGE cost_anomaly: spend $0.4000 is 13.3x the $0.0300 baseline
starter-run.txt
An untouched starter fails nearly the whole suite:
21 failed, 1 passed in 0.07s
The one that passes is test_worst_of_nothing_is_ok, which exercises worst() on
an empty list -- worst() is not stubbed. Everything else depends on percentile,
windows, evaluate or rolling_baseline.
test-output.txt
...................... [100%]
22 passed in 0.01s
test-run.txt
...................... [100%]
22 passed in 0.01s
Source files
examples/monitoring_demo.py (1762 bytes)
#!/usr/bin/env python3
"""Twenty-five minutes of traffic with three problems buried in it."""
from __future__ import annotations
import os
import random
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from monitoring import Request, evaluate, rolling_baseline, windows
def build_traffic() -> list[Request]:
"""Five windows: two healthy, then latency, quality and cost problems."""
rng = random.Random(11)
out: list[Request] = []
def batch(at_from: int, n: int, *, slow=False, errors=0.0, ungrounded=0.0, pricey=False):
for i in range(n):
at = at_from + i % 5
latency = rng.randint(3800, 9000) if slow else rng.randint(300, 1500)
out.append(
Request(
at=at,
latency_ms=latency,
ok=rng.random() >= errors,
cost=(0.02 if pricey else 0.0015),
tokens=rng.randint(400, 900),
grounded=rng.random() >= ungrounded,
)
)
batch(0, 20) # healthy
batch(5, 20) # healthy
batch(10, 20, slow=True) # latency problem
batch(15, 20, ungrounded=0.45) # quality problem, all 200s
batch(20, 20, pricey=True) # cost problem
return out
def main() -> int:
traffic = build_traffic()
history = []
for window in windows(traffic, size=5):
baseline = rolling_baseline(history)
alerts = evaluate(window, baseline_spend=baseline)
print(window.line())
for alert in alerts:
print(alert.line())
history.append(window)
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/monitoring.py (6587 bytes)
"""Windowed metrics, alert rules and cost anomaly detection for an AI service.
Offline and standard-library only. Requests are records with a logical
timestamp, so every percentile and every alert is exactly reproducible.
The shape of the problem is different from ordinary web monitoring in three
ways, and each drives something here:
cost is per request so spend is a first-class metric, not a monthly bill
failure is often partial a request can succeed and still be useless
latency is long-tailed so the mean is actively misleading
Two principles run through the module:
* Alert on SYMPTOMS a user would notice, not on causes. "p95 latency is
above 4s" is actionable; "cache hit rate fell" may be entirely fine.
* Compare against a BASELINE, not a fixed threshold, wherever normal varies.
A fixed spend limit is either always breached or never useful.
"""
from __future__ import annotations
import math
from dataclasses import dataclass, field
from enum import Enum
class Severity(str, Enum):
OK = "ok"
WARN = "warn"
PAGE = "page"
@dataclass(frozen=True)
class Request:
"""One served request."""
at: int # logical minute
latency_ms: int
ok: bool
cost: float
tokens: int
grounded: bool = True # did the answer cite retrieved context?
@dataclass
class Window:
"""Metrics over one slice of time."""
start: int
end: int
requests: list[Request] = field(default_factory=list)
@property
def count(self) -> int:
return len(self.requests)
@property
def error_rate(self) -> float:
if not self.requests:
return 0.0
return sum(1 for r in self.requests if not r.ok) / len(self.requests)
@property
def ungrounded_rate(self) -> float:
"""Answers that cited nothing.
A quality signal rather than an availability one. These requests
succeeded -- they returned 200 and a fluent answer -- and are still a
failure of the thing the system exists to do.
"""
if not self.requests:
return 0.0
return sum(1 for r in self.requests if not r.grounded) / len(self.requests)
@property
def spend(self) -> float:
return sum(r.cost for r in self.requests)
def latency(self, p: float) -> int:
return percentile([r.latency_ms for r in self.requests], p)
def line(self) -> str:
return (
f"[{self.start:>3}-{self.end:<3}) n={self.count:<3} "
f"p50={self.latency(50):<5} p95={self.latency(95):<6} "
f"err={self.error_rate:.0%} ungrounded={self.ungrounded_rate:.0%} "
f"spend=${self.spend:.4f}"
)
def percentile(values: list[int], p: float) -> int:
"""Nearest-rank percentile. Zero for an empty input.
`math.ceil`, not `round`: Python rounds halves to even, which puts the p50
of a five-element list on the wrong observation.
"""
if not values:
return 0
ordered = sorted(values)
rank = max(1, min(len(ordered), math.ceil(p / 100 * len(ordered))))
return ordered[rank - 1]
def windows(requests: list[Request], *, size: int = 5) -> list[Window]:
"""Bucket requests into fixed windows by their logical minute."""
if not requests:
return []
lo = min(r.at for r in requests)
hi = max(r.at for r in requests)
out: list[Window] = []
for start in range(lo, hi + 1, size):
end = start + size
out.append(
Window(start, end, [r for r in requests if start <= r.at < end])
)
return out
@dataclass
class Alert:
name: str
severity: Severity
detail: str
def line(self) -> str:
return f" {self.severity.value.upper():<5} {self.name}: {self.detail}"
def evaluate(
window: Window,
*,
baseline_spend: float | None = None,
latency_slo_ms: int = 4000,
error_budget: float = 0.02,
ungrounded_budget: float = 0.10,
spend_multiple: float = 3.0,
min_sample: int = 10,
) -> list[Alert]:
"""Apply the alert rules to one window.
`min_sample` matters more than it looks. On a small window a single failed
request is a 20% error rate, and paging on that trains people to ignore
pages -- which is worse than not alerting at all.
"""
alerts: list[Alert] = []
if window.count < min_sample:
return [
Alert(
"insufficient_sample",
Severity.OK,
f"{window.count} requests is below the {min_sample} needed to judge",
)
]
p95 = window.latency(95)
if p95 > latency_slo_ms:
alerts.append(
Alert("latency_slo", Severity.PAGE, f"p95 {p95}ms exceeds {latency_slo_ms}ms")
)
if window.error_rate > error_budget:
alerts.append(
Alert(
"error_budget",
Severity.PAGE,
f"{window.error_rate:.0%} errors exceeds {error_budget:.0%}",
)
)
# A quality symptom, not an availability one. These requests SUCCEEDED.
if window.ungrounded_rate > ungrounded_budget:
alerts.append(
Alert(
"ungrounded_answers",
Severity.WARN,
f"{window.ungrounded_rate:.0%} of answers cited nothing",
)
)
# Spend is compared against what normal looked like, not a fixed cap.
if baseline_spend is not None and baseline_spend > 0:
ratio = window.spend / baseline_spend
if ratio > spend_multiple:
alerts.append(
Alert(
"cost_anomaly",
Severity.PAGE,
f"spend ${window.spend:.4f} is {ratio:.1f}x the ${baseline_spend:.4f} baseline",
)
)
return alerts or [Alert("healthy", Severity.OK, "all signals within budget")]
def worst(alerts: list[Alert]) -> Severity:
order = {Severity.OK: 0, Severity.WARN: 1, Severity.PAGE: 2}
return max((a.severity for a in alerts), key=lambda s: order[s], default=Severity.OK)
def rolling_baseline(history: list[Window], *, keep: int = 3) -> float | None:
"""Median spend of the last `keep` windows.
Median rather than mean: one anomalous window should not raise the baseline
enough to hide the next one, which is exactly how a spend alert stops
firing after the first incident.
"""
recent = [w.spend for w in history[-keep:] if w.count]
if not recent:
return None
ordered = sorted(recent)
return ordered[len(ordered) // 2]
metadata.yml (875 bytes)
lesson_id: D360
day: 360
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/monitoring_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3
examples/monitoring_demo.py reproduced the documented five windows exactly (traffic is seeded, so the
percentiles are deterministic); bash tests/run_tests.sh -> 22 passed in 0.02s. The untouched starter
fails 21 of 22; the 1 that passes exercises worst() on an empty list, which is not stubbed.
requires_network is false: requests are synthetic records with a logical timestamp and no test opens a
socket.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/monitoring.py (6096 bytes)
"""Windowed metrics, alert rules and cost anomaly detection for an AI service.
Offline and standard-library only. Requests are records with a logical
timestamp, so every percentile and every alert is exactly reproducible.
The shape of the problem is different from ordinary web monitoring in three
ways, and each drives something here:
cost is per request so spend is a first-class metric, not a monthly bill
failure is often partial a request can succeed and still be useless
latency is long-tailed so the mean is actively misleading
Two principles run through the module:
* Alert on SYMPTOMS a user would notice, not on causes. "p95 latency is
above 4s" is actionable; "cache hit rate fell" may be entirely fine.
* Compare against a BASELINE, not a fixed threshold, wherever normal varies.
A fixed spend limit is either always breached or never useful.
"""
from __future__ import annotations
import math
from dataclasses import dataclass, field
from enum import Enum
class Severity(str, Enum):
OK = "ok"
WARN = "warn"
PAGE = "page"
@dataclass(frozen=True)
class Request:
"""One served request."""
at: int # logical minute
latency_ms: int
ok: bool
cost: float
tokens: int
grounded: bool = True # did the answer cite retrieved context?
@dataclass
class Window:
"""Metrics over one slice of time."""
start: int
end: int
requests: list[Request] = field(default_factory=list)
@property
def count(self) -> int:
return len(self.requests)
@property
def error_rate(self) -> float:
if not self.requests:
return 0.0
return sum(1 for r in self.requests if not r.ok) / len(self.requests)
@property
def ungrounded_rate(self) -> float:
"""Answers that cited nothing.
A quality signal rather than an availability one. These requests
succeeded -- they returned 200 and a fluent answer -- and are still a
failure of the thing the system exists to do.
"""
if not self.requests:
return 0.0
return sum(1 for r in self.requests if not r.grounded) / len(self.requests)
@property
def spend(self) -> float:
return sum(r.cost for r in self.requests)
def latency(self, p: float) -> int:
return percentile([r.latency_ms for r in self.requests], p)
def line(self) -> str:
return (
f"[{self.start:>3}-{self.end:<3}) n={self.count:<3} "
f"p50={self.latency(50):<5} p95={self.latency(95):<6} "
f"err={self.error_rate:.0%} ungrounded={self.ungrounded_rate:.0%} "
f"spend=${self.spend:.4f}"
)
def percentile(values: list[int], p: float) -> int:
"""Nearest-rank percentile. Zero for an empty input.
`math.ceil`, not `round`: Python rounds halves to even, which puts the p50
of a five-element list on the wrong observation.
"""
# TASK 1: nearest-rank percentile; 0 for an empty list.
# rank = ceil(p/100 * n), clamped to [1, n], then ordered[rank-1].
# math.ceil, NOT round -- Python rounds halves to even, which puts the p50
# of a five-element list on the wrong observation.
raise NotImplementedError("implement percentile")
def windows(requests: list[Request], *, size: int = 5) -> list[Window]:
"""Bucket requests into fixed windows by their logical minute."""
# TASK 2: bucket requests into fixed windows of `size` logical minutes,
# from the earliest to the latest timestamp. Empty input gives no windows;
# a window with no requests in it is still a window.
raise NotImplementedError("implement windows")
@dataclass
class Alert:
name: str
severity: Severity
detail: str
def line(self) -> str:
return f" {self.severity.value.upper():<5} {self.name}: {self.detail}"
def evaluate(
window: Window,
*,
baseline_spend: float | None = None,
latency_slo_ms: int = 4000,
error_budget: float = 0.02,
ungrounded_budget: float = 0.10,
spend_multiple: float = 3.0,
min_sample: int = 10,
) -> list[Alert]:
"""Apply the alert rules to one window.
`min_sample` matters more than it looks. On a small window a single failed
request is a 20% error rate, and paging on that trains people to ignore
pages -- which is worse than not alerting at all.
"""
# TASK 3: apply the alert rules to one window.
# - FIRST: if window.count < min_sample, return a single OK
# "insufficient_sample" alert and nothing else. One failure in three
# requests is a 33% error rate, and paging on that trains people to
# ignore the channel.
# - p95 latency over latency_slo_ms -> PAGE "latency_slo"
# - error_rate over error_budget -> PAGE "error_budget"
# - ungrounded_rate over its budget -> WARN "ungrounded_answers"
# (note these requests SUCCEEDED -- that is the whole point)
# - spend over baseline_spend * spend_multiple -> PAGE "cost_anomaly",
# but only when a baseline exists. Compare against normal, never a
# fixed cap.
# - if nothing fired, return one OK "healthy" alert.
raise NotImplementedError("implement evaluate")
def worst(alerts: list[Alert]) -> Severity:
order = {Severity.OK: 0, Severity.WARN: 1, Severity.PAGE: 2}
return max((a.severity for a in alerts), key=lambda s: order[s], default=Severity.OK)
def rolling_baseline(history: list[Window], *, keep: int = 3) -> float | None:
"""Median spend of the last `keep` windows.
Median rather than mean: one anomalous window should not raise the baseline
enough to hide the next one, which is exactly how a spend alert stops
firing after the first incident.
"""
# TASK 4: median spend of the last `keep` non-empty windows, or None.
# MEDIAN, not mean: one anomalous window must not raise the baseline enough
# to hide the next one, which is exactly how a spend alert stops firing
# after the first incident.
raise NotImplementedError("implement rolling_baseline")
tests/run_tests.sh (200 bytes)
#!/usr/bin/env bash
# Run the day 360 monitoring test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_monitoring.py (6236 bytes)
"""Grouped by signal, so a failure names which alert stopped working.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from monitoring import ( # noqa: E402
Request,
Severity,
Window,
evaluate,
percentile,
rolling_baseline,
windows,
worst,
)
def reqs(n, *, latency=500, ok=True, cost=0.001, grounded=True, at=0):
return [Request(at=at, latency_ms=latency, ok=ok, cost=cost, tokens=500,
grounded=grounded) for _ in range(n)]
def names(alerts):
return {a.name for a in alerts}
# --------------------------------------------------------------- percentiles
def test_percentile_uses_nearest_rank_not_banker_rounding():
# round(2.5) is 2 in Python, which would pick the wrong observation.
assert percentile([1, 2, 3, 4, 100], 50) == 3
assert percentile([1, 2, 3, 4, 100], 95) == 100
def test_percentile_of_nothing_is_zero():
assert percentile([], 95) == 0
def test_p95_is_not_the_mean():
# A tenth of requests are slow. The mean stays comfortable; p95 does not.
values = [200] * 18 + [9000] * 2
assert sum(values) / len(values) < 1100
assert percentile(values, 95) == 9000
def test_p95_cannot_see_a_single_outlier_in_twenty_samples():
# Worth knowing rather than discovering during an incident. Nearest rank
# for p95 over 20 samples is the 19th value, so ONE slow request sits above
# the percentile entirely and is invisible to it.
values = [200] * 19 + [9000]
assert percentile(values, 95) == 200
# p99 over the same data is the 20th, which does see it.
assert percentile(values, 99) == 9000
# The general rule: a percentile can only see events more frequent than
# 1 - p. Choose the percentile to match what you need to catch, and make
# sure the window is large enough to contain them.
# ------------------------------------------------------------------ windows
def test_requests_are_bucketed_by_logical_minute():
traffic = reqs(3, at=0) + reqs(3, at=6) + reqs(3, at=11)
got = windows(traffic, size=5)
assert [w.count for w in got] == [3, 3, 3]
assert [(w.start, w.end) for w in got] == [(0, 5), (5, 10), (10, 15)]
def test_empty_traffic_produces_no_windows():
assert windows([]) == []
def test_a_window_with_no_requests_reports_zero_not_an_error():
w = Window(0, 5, [])
assert w.error_rate == 0.0 and w.spend == 0.0 and w.latency(95) == 0
# ------------------------------------------------------------ small samples
def test_a_small_window_is_not_judged():
# One failure in three requests is a 33% error rate. Paging on that trains
# people to ignore pages, which is worse than not alerting.
w = Window(0, 5, reqs(2, ok=False))
alerts = evaluate(w)
assert names(alerts) == {"insufficient_sample"}
assert worst(alerts) is Severity.OK
def test_the_same_rate_alerts_once_the_sample_is_large_enough():
w = Window(0, 5, reqs(10, ok=False))
assert "error_budget" in names(evaluate(w))
# ------------------------------------------------------------------ latency
def test_latency_over_the_slo_pages():
w = Window(0, 5, reqs(20, latency=9000))
alerts = evaluate(w)
assert "latency_slo" in names(alerts)
assert worst(alerts) is Severity.PAGE
def test_latency_inside_the_slo_is_healthy():
w = Window(0, 5, reqs(20, latency=900))
assert names(evaluate(w)) == {"healthy"}
# ------------------------------------------------------------------- errors
def test_error_rate_over_budget_pages():
w = Window(0, 5, reqs(10, ok=False) + reqs(10))
assert "error_budget" in names(evaluate(w))
def test_error_rate_inside_budget_is_healthy():
w = Window(0, 5, reqs(100))
assert names(evaluate(w)) == {"healthy"}
# ------------------------------------------------------------------ quality
def test_ungrounded_answers_warn_even_though_every_request_succeeded():
# The distinguishing case for an AI service: 100% availability, and the
# answers cite nothing. Conventional monitoring reports this as healthy.
w = Window(0, 5, reqs(10, grounded=False) + reqs(10))
alerts = evaluate(w)
assert w.error_rate == 0.0
assert "ungrounded_answers" in names(alerts)
assert worst(alerts) is Severity.WARN
def test_grounded_answers_do_not_warn():
w = Window(0, 5, reqs(20, grounded=True))
assert "ungrounded_answers" not in names(evaluate(w))
# --------------------------------------------------------------------- cost
def test_a_spend_spike_against_the_baseline_pages():
w = Window(0, 5, reqs(20, cost=0.02))
alerts = evaluate(w, baseline_spend=0.03)
assert "cost_anomaly" in names(alerts)
assert "13.3x" in next(a for a in alerts if a.name == "cost_anomaly").detail
def test_normal_spend_against_the_baseline_is_healthy():
w = Window(0, 5, reqs(20, cost=0.0015))
assert names(evaluate(w, baseline_spend=0.03)) == {"healthy"}
def test_no_baseline_means_no_cost_alert():
# A first window has nothing to compare against, and inventing a fixed
# threshold would be either always breached or never useful.
w = Window(0, 5, reqs(20, cost=0.02))
assert "cost_anomaly" not in names(evaluate(w, baseline_spend=None))
def test_baseline_is_a_median_so_one_spike_does_not_hide_the_next():
normal = Window(0, 5, reqs(20, cost=0.0015))
spike = Window(5, 10, reqs(20, cost=0.02))
# Mean of these would be pulled up by the spike; the median is not.
assert rolling_baseline([normal, normal, spike]) == normal.spend
def test_rolling_baseline_ignores_empty_windows():
assert rolling_baseline([Window(0, 5, [])]) is None
# ------------------------------------------------------------------ severity
def test_worst_severity_wins():
w = Window(0, 5, reqs(10, latency=9000, grounded=False) + reqs(10, latency=9000))
alerts = evaluate(w)
assert names(alerts) >= {"latency_slo", "ungrounded_answers"}
assert worst(alerts) is Severity.PAGE
def test_worst_of_nothing_is_ok():
assert worst([]) is Severity.OK
Troubleshooting
Troubleshooting — Day 360
Every small window pages
You are evaluating the rules before checking the sample size. The min_sample check must come first and return immediately — one failure in three requests is a 33 percent error rate, and paging on it is how a channel gets muted.
The p50 of [1, 2, 3, 4, 100] comes out as 2
You used round for the nearest-rank calculation. Python rounds halves to even, so round(2.5) is 2 and you land on the wrong observation. Use math.ceil.
test_p95_cannot_see_a_single_outlier_in_twenty_samples fails
Read the test before changing the code — it asserts that p95 over twenty samples is 200, not 9000, and that is correct. Nearest rank puts p95 at the nineteenth value, so a single outlier sits above it. If your implementation returns 9000 you are computing a maximum rather than a percentile.
The cost alert stops firing after the first spike
Your baseline is a mean, and the spike raised it enough to make the next one look normal. Use a median, which is unmoved by a single outlier. This failure is self-concealing, which is why it has its own test.
A spend spike alerts on the very first window
There is no baseline yet, so there is nothing to compare against. rolling_baseline should return None for an empty history, and evaluate should skip the cost rule when the baseline is None. Inventing a fixed threshold instead would either never fire or always fire.
ZeroDivisionError on an empty window
A window with no requests is a normal state, particularly at the edges of a traffic period. Guard every denominator and return zeros.
Ungrounded answers never warn
Check that your generator is actually producing them, and that the rule compares against ungrounded_budget rather than the error budget. The point of this rule is that it fires while error_rate is zero — if both move together, you are measuring failures rather than quality.
The demo numbers do not match
Traffic is generated from a seeded RNG, so a correct implementation reproduces the table exactly. Differences usually come from the percentile rank calculation or from bucketing boundaries — windows are half-open, start <= at < end.
NotImplementedError on nearly every test
Expected. The starter stubs four functions — see expected-output/starter-run.txt, which also names the one test that passes without them.
Security notes
Security notes — Day 360
What this lab does and does not touch
- No network. Requests are synthetic records with a logical timestamp. No test opens a socket.
- No credentials. No API key, token or account.
- No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/. - No
sudo, no global installs. - No real user data. The request records carry latency, cost and boolean flags — nothing identifying.
A cost alert is a security control
The cost_anomaly rule looks like budgeting. In production it is frequently the first detection of an attack, and often the only one, because the traffic is otherwise well-formed.
Three cases it catches that nothing else does:
- A leaked key. Someone else is using your quota. Requests succeed, latency is fine, errors are zero — spend is the only signal that moves.
- A scripted client. A denial-of-wallet attempt, deliberately staying under rate limits so the requests look legitimate individually.
- Prompt injection inducing recursion. Text inside a retrieved document makes an agent call itself. Every individual call is valid.
This is why the rule pages rather than warns, and why it compares against a baseline: an attacker operating just above normal is exactly what a fixed threshold set generously will miss.
Traces are the highest-risk store you own
Worth stating plainly, because it is easy to miss while thinking about metrics.
Metrics are aggregates — counts, percentiles, sums. They are safe to retain, cheap to store, and contain no personal data.
Traces are not. A trace captures the full prompt and the full completion, which means it contains whatever the user typed and whatever the system said back. In most AI systems traces are:
- the store with the most sensitive content,
- retained the longest, often by default rather than by decision,
- examined the least, so nobody notices what accumulated,
- and the one most often forgotten in a deletion path.
Everything from Day 328 applies to them. Set a retention period deliberately, include traces in the erasure path, and redact before writing rather than after.
Cardinality is an availability problem
A label per user or per document identifier produces millions of distinct series. The consequence is not a bad dashboard — it is a metrics backend that becomes slow, then expensive, then unavailable, usually during the incident when you need it. Keep labels to a small bounded set, and cap them (extension exercise 2).
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.