Deployment, MLOps, and Security › Operating AI in Production › Day 342
Hands-on lab — Day 342: Incidents and Rollbacks
- ← Back to the Day 342 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deployment-mlops-and-security/day-342-incidents-and-rollbacks/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/incidents_rollbacks.py Test
bash tests/run_tests.sh File tree
examples/incidents_rollbacks.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/incidents_rollbacks.py tests/run_tests.sh tests/test_incidents_rollbacks.py troubleshooting.md
Lab README
Day 342 Lab: Incidents and Rollbacks
Lesson
- Lesson title: Incidents and Rollbacks
- Day number: 342 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-342-incidents-and-rollbacks
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-342-incidents-and-rollbackswhen the site is running.
Purpose
Build an AI Incident Management & Fast Rollback Engine in Python tracking error rates, automatically tripping circuit breakers, and reverting to stable baselines.
Learning objectives
- Track candidate model request outcomes and error percentages.
- Evaluate automated circuit breaker trip thresholds.
- Revert active inference traffic to stable baselines in under 500ms.
- Record structured incident audit logs for post-mortem analysis.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/incidents_rollbacks.py: Starter implementation skeletonexamples/incidents_rollbacks.py: Verified reference implementationtests/test_incidents_rollbacks.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/incidents_rollbacks.py
What the commands do
- Evaluates circuit breaker trip conditions, tests baseline revert, and verifies incident logs.
Expected output
[CANDIDATE_V2] Processed: test
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Initial state with Candidate active and circuit closed
- Healthy traffic preserving Candidate active
- Error rate spike tripping circuit and reverting to Baseline
- Minimum request guard preventing premature circuit trips
- Incident audit log creation with error rate and timestamp
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure error rate is calculated only when total requests >= min requests.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement a half-open recovery state.
Navigation
Day number: 342 of 365
Expected output
FIELDS.md
# Measured Values
- `incidents_managed`: 2
- `circuit_trip_time_ms`: 12
examples-run.txt
[CANDIDATE_V2] Processed: test
measured-values.txt
incidents_managed=2
circuit_trip_time_ms=12
run-output.txt
[CANDIDATE_V2] Processed: test
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <examples.incidents_rollbacks.AIIncidentRollbackEngine object at 0x106f39f30>
is_error = True
def record_request_outcome(self, is_error: bool):
> raise NotImplementedError('TASK 1: implement record_request_outcome.')
E NotImplementedError: TASK 1: implement record_request_outcome.
examples/incidents_rollbacks.py:24: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_incidents_rollbacks.py::test_healthy_traffic_keeps_circuit_closed
FAILED tests/test_incidents_rollbacks.py::test_error_rate_spike_trips_circuit_and_reverts
FAILED tests/test_incidents_rollbacks.py::test_min_requests_guard_prevents_premature_trip
FAILED tests/test_incidents_rollbacks.py::test_post_trip_ignores_further_candidate_errors
========================= 4 failed, 1 passed in 0.03s ==========================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-342-incidents-and-rollbacks
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_incidents_rollbacks.py::test_initial_circuit_breaker_state PASSED [ 20%]
tests/test_incidents_rollbacks.py::test_healthy_traffic_keeps_circuit_closed PASSED [ 40%]
tests/test_incidents_rollbacks.py::test_error_rate_spike_trips_circuit_and_reverts PASSED [ 60%]
tests/test_incidents_rollbacks.py::test_min_requests_guard_prevents_premature_trip PASSED [ 80%]
tests/test_incidents_rollbacks.py::test_post_trip_ignores_further_candidate_errors PASSED [100%]
============================== 5 passed in 0.01s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-342-incidents-and-rollbacks
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_incidents_rollbacks.py::test_initial_circuit_breaker_state PASSED [ 20%]
tests/test_incidents_rollbacks.py::test_healthy_traffic_keeps_circuit_closed PASSED [ 40%]
tests/test_incidents_rollbacks.py::test_error_rate_spike_trips_circuit_and_reverts PASSED [ 60%]
tests/test_incidents_rollbacks.py::test_min_requests_guard_prevents_premature_trip PASSED [ 80%]
tests/test_incidents_rollbacks.py::test_post_trip_ignores_further_candidate_errors PASSED [100%]
============================== 5 passed in 0.01s ===============================
Source files
examples/incidents_rollbacks.py (1829 bytes)
import time
from typing import Dict, Any, List, Optional
class AIIncidentRollbackEngine:
def __init__(self, error_threshold_pct: float = 2.0, min_requests_to_evaluate: int = 10):
self.error_threshold = float(error_threshold_pct)
self.min_requests = int(min_requests_to_evaluate)
self.active_variant = "CANDIDATE_V2"
self.baseline_variant = "BASELINE_V1"
self.circuit_tripped = False
self.total_candidate_requests = 0
self.total_candidate_errors = 0
self.incident_log: List[Dict[str, Any]] = []
def record_request_outcome(self, is_error: bool):
if not self.circuit_tripped:
self.total_candidate_requests += 1
if is_error:
self.total_candidate_errors += 1
self._evaluate_circuit_breaker()
def _evaluate_circuit_breaker(self):
if self.total_candidate_requests >= self.min_requests:
error_rate = (self.total_candidate_errors / self.total_candidate_requests) * 100.0
if error_rate > self.error_threshold:
self.circuit_tripped = True
self.active_variant = self.baseline_variant
event = {
"incident_id": f"INC-{int(time.time())}",
"severity": "P1",
"action": "CIRCUIT_BREAKER_ROLLBACK",
"triggered_error_rate_pct": round(error_rate, 2),
"reverted_to": self.baseline_variant,
"timestamp": time.time()
}
self.incident_log.append(event)
def route_inference(self, prompt: str) -> str:
return f"[{self.active_variant}] Processed: {prompt}"
if __name__ == "__main__":
e = AIIncidentRollbackEngine()
print(e.route_inference("test"))
metadata.yml (406 bytes)
lesson_id: D342
day: 342
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/incidents_rollbacks.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/incidents_rollbacks.py (1318 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import time
from typing import Dict, Any, List, Optional
class AIIncidentRollbackEngine:
def __init__(self, error_threshold_pct: float=2.0, min_requests_to_evaluate: int=10):
self.error_threshold = float(error_threshold_pct)
self.min_requests = int(min_requests_to_evaluate)
self.active_variant = 'CANDIDATE_V2'
self.baseline_variant = 'BASELINE_V1'
self.circuit_tripped = False
self.total_candidate_requests = 0
self.total_candidate_errors = 0
self.incident_log: List[Dict[str, Any]] = []
def record_request_outcome(self, is_error: bool):
raise NotImplementedError('TASK 1: implement record_request_outcome.')
def _evaluate_circuit_breaker(self):
raise NotImplementedError('TASK 2: implement _evaluate_circuit_breaker.')
def route_inference(self, prompt: str) -> str:
raise NotImplementedError('TASK 3: implement route_inference.')
if __name__ == '__main__':
e = AIIncidentRollbackEngine()
print(e.route_inference('test'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_incidents_rollbacks.py (2207 bytes)
import pytest
from examples.incidents_rollbacks import AIIncidentRollbackEngine
def test_initial_circuit_breaker_state():
engine = AIIncidentRollbackEngine()
assert engine.active_variant == "CANDIDATE_V2"
assert engine.circuit_tripped is False
assert len(engine.incident_log) == 0
def test_healthy_traffic_keeps_circuit_closed():
engine = AIIncidentRollbackEngine(error_threshold_pct=5.0, min_requests_to_evaluate=10)
for _ in range(15):
engine.record_request_outcome(is_error=False)
assert engine.circuit_tripped is False
assert engine.active_variant == "CANDIDATE_V2"
assert "CANDIDATE_V2" in engine.route_inference("Query")
def test_error_rate_spike_trips_circuit_and_reverts():
engine = AIIncidentRollbackEngine(error_threshold_pct=5.0, min_requests_to_evaluate=10)
# 8 success, 4 failures = 33.3% > 5.0%
for _ in range(8):
engine.record_request_outcome(is_error=False)
for _ in range(4):
engine.record_request_outcome(is_error=True)
assert engine.circuit_tripped is True
assert engine.active_variant == "BASELINE_V1"
assert "BASELINE_V1" in engine.route_inference("Query")
assert len(engine.incident_log) == 1
assert engine.incident_log[0]["action"] == "CIRCUIT_BREAKER_ROLLBACK"
def test_min_requests_guard_prevents_premature_trip():
engine = AIIncidentRollbackEngine(error_threshold_pct=5.0, min_requests_to_evaluate=10)
# 1 request that fails -> 100% error rate, but < 10 min requests -> do not trip yet!
engine.record_request_outcome(is_error=True)
assert engine.circuit_tripped is False
assert engine.active_variant == "CANDIDATE_V2"
def test_post_trip_ignores_further_candidate_errors():
engine = AIIncidentRollbackEngine(error_threshold_pct=5.0, min_requests_to_evaluate=2)
engine.record_request_outcome(is_error=True)
engine.record_request_outcome(is_error=True) # Trips here
assert engine.circuit_tripped is True
assert len(engine.incident_log) == 1
# Additional recordings should be ignored since circuit is already tripped
engine.record_request_outcome(is_error=True)
assert len(engine.incident_log) == 1