Deployment, MLOps, and Security › Operating AI in Production › Day 342

Hands-on lab — Day 342: Incidents and Rollbacks

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/incidents_rollbacks.py

Test

bash tests/run_tests.sh

File tree

examples/incidents_rollbacks.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/incidents_rollbacks.py
tests/run_tests.sh
tests/test_incidents_rollbacks.py
troubleshooting.md

Lab README

Day 342 Lab: Incidents and Rollbacks

Lesson

  • Lesson title: Incidents and Rollbacks
  • Day number: 342 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-342-incidents-and-rollbacks
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-342-incidents-and-rollbacks when the site is running.

Purpose

Build an AI Incident Management & Fast Rollback Engine in Python tracking error rates, automatically tripping circuit breakers, and reverting to stable baselines.

Learning objectives

  • Track candidate model request outcomes and error percentages.
  • Evaluate automated circuit breaker trip thresholds.
  • Revert active inference traffic to stable baselines in under 500ms.
  • Record structured incident audit logs for post-mortem analysis.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/incidents_rollbacks.py: Starter implementation skeleton
  • examples/incidents_rollbacks.py: Verified reference implementation
  • tests/test_incidents_rollbacks.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/incidents_rollbacks.py

What the commands do

  • Evaluates circuit breaker trip conditions, tests baseline revert, and verifies incident logs.

Expected output

[CANDIDATE_V2] Processed: test

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Initial state with Candidate active and circuit closed
  • Healthy traffic preserving Candidate active
  • Error rate spike tripping circuit and reverting to Baseline
  • Minimum request guard preventing premature circuit trips
  • Incident audit log creation with error rate and timestamp

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure error rate is calculated only when total requests >= min requests.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement a half-open recovery state.

Day number: 342 of 365

Expected output

FIELDS.md

# Measured Values
- `incidents_managed`: 2
- `circuit_trip_time_ms`: 12

examples-run.txt

[CANDIDATE_V2] Processed: test

measured-values.txt

incidents_managed=2
circuit_trip_time_ms=12

run-output.txt

[CANDIDATE_V2] Processed: test

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <examples.incidents_rollbacks.AIIncidentRollbackEngine object at 0x106f39f30>
is_error = True

    def record_request_outcome(self, is_error: bool):
>       raise NotImplementedError('TASK 1: implement record_request_outcome.')
E       NotImplementedError: TASK 1: implement record_request_outcome.

examples/incidents_rollbacks.py:24: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_incidents_rollbacks.py::test_healthy_traffic_keeps_circuit_closed
FAILED tests/test_incidents_rollbacks.py::test_error_rate_spike_trips_circuit_and_reverts
FAILED tests/test_incidents_rollbacks.py::test_min_requests_guard_prevents_premature_trip
FAILED tests/test_incidents_rollbacks.py::test_post_trip_ignores_further_candidate_errors
========================= 4 failed, 1 passed in 0.03s ==========================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-342-incidents-and-rollbacks
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_incidents_rollbacks.py::test_initial_circuit_breaker_state PASSED [ 20%]
tests/test_incidents_rollbacks.py::test_healthy_traffic_keeps_circuit_closed PASSED [ 40%]
tests/test_incidents_rollbacks.py::test_error_rate_spike_trips_circuit_and_reverts PASSED [ 60%]
tests/test_incidents_rollbacks.py::test_min_requests_guard_prevents_premature_trip PASSED [ 80%]
tests/test_incidents_rollbacks.py::test_post_trip_ignores_further_candidate_errors PASSED [100%]

============================== 5 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-342-incidents-and-rollbacks
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_incidents_rollbacks.py::test_initial_circuit_breaker_state PASSED [ 20%]
tests/test_incidents_rollbacks.py::test_healthy_traffic_keeps_circuit_closed PASSED [ 40%]
tests/test_incidents_rollbacks.py::test_error_rate_spike_trips_circuit_and_reverts PASSED [ 60%]
tests/test_incidents_rollbacks.py::test_min_requests_guard_prevents_premature_trip PASSED [ 80%]
tests/test_incidents_rollbacks.py::test_post_trip_ignores_further_candidate_errors PASSED [100%]

============================== 5 passed in 0.01s ===============================

Source files

examples/incidents_rollbacks.py (1829 bytes)
import time
from typing import Dict, Any, List, Optional

class AIIncidentRollbackEngine:
    def __init__(self, error_threshold_pct: float = 2.0, min_requests_to_evaluate: int = 10):
        self.error_threshold = float(error_threshold_pct)
        self.min_requests = int(min_requests_to_evaluate)
        self.active_variant = "CANDIDATE_V2"
        self.baseline_variant = "BASELINE_V1"
        self.circuit_tripped = False
        self.total_candidate_requests = 0
        self.total_candidate_errors = 0
        self.incident_log: List[Dict[str, Any]] = []

    def record_request_outcome(self, is_error: bool):
        if not self.circuit_tripped:
            self.total_candidate_requests += 1
            if is_error:
                self.total_candidate_errors += 1
            self._evaluate_circuit_breaker()

    def _evaluate_circuit_breaker(self):
        if self.total_candidate_requests >= self.min_requests:
            error_rate = (self.total_candidate_errors / self.total_candidate_requests) * 100.0
            if error_rate > self.error_threshold:
                self.circuit_tripped = True
                self.active_variant = self.baseline_variant
                
                event = {
                    "incident_id": f"INC-{int(time.time())}",
                    "severity": "P1",
                    "action": "CIRCUIT_BREAKER_ROLLBACK",
                    "triggered_error_rate_pct": round(error_rate, 2),
                    "reverted_to": self.baseline_variant,
                    "timestamp": time.time()
                }
                self.incident_log.append(event)

    def route_inference(self, prompt: str) -> str:
        return f"[{self.active_variant}] Processed: {prompt}"

if __name__ == "__main__":
    e = AIIncidentRollbackEngine()
    print(e.route_inference("test"))
metadata.yml (406 bytes)
lesson_id: D342
day: 342
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/incidents_rollbacks.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/incidents_rollbacks.py (1318 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import time
from typing import Dict, Any, List, Optional

class AIIncidentRollbackEngine:

    def __init__(self, error_threshold_pct: float=2.0, min_requests_to_evaluate: int=10):
        self.error_threshold = float(error_threshold_pct)
        self.min_requests = int(min_requests_to_evaluate)
        self.active_variant = 'CANDIDATE_V2'
        self.baseline_variant = 'BASELINE_V1'
        self.circuit_tripped = False
        self.total_candidate_requests = 0
        self.total_candidate_errors = 0
        self.incident_log: List[Dict[str, Any]] = []

    def record_request_outcome(self, is_error: bool):
        raise NotImplementedError('TASK 1: implement record_request_outcome.')

    def _evaluate_circuit_breaker(self):
        raise NotImplementedError('TASK 2: implement _evaluate_circuit_breaker.')

    def route_inference(self, prompt: str) -> str:
        raise NotImplementedError('TASK 3: implement route_inference.')
if __name__ == '__main__':
    e = AIIncidentRollbackEngine()
    print(e.route_inference('test'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_incidents_rollbacks.py (2207 bytes)
import pytest
from examples.incidents_rollbacks import AIIncidentRollbackEngine

def test_initial_circuit_breaker_state():
    engine = AIIncidentRollbackEngine()
    assert engine.active_variant == "CANDIDATE_V2"
    assert engine.circuit_tripped is False
    assert len(engine.incident_log) == 0

def test_healthy_traffic_keeps_circuit_closed():
    engine = AIIncidentRollbackEngine(error_threshold_pct=5.0, min_requests_to_evaluate=10)
    for _ in range(15):
        engine.record_request_outcome(is_error=False)
        
    assert engine.circuit_tripped is False
    assert engine.active_variant == "CANDIDATE_V2"
    assert "CANDIDATE_V2" in engine.route_inference("Query")

def test_error_rate_spike_trips_circuit_and_reverts():
    engine = AIIncidentRollbackEngine(error_threshold_pct=5.0, min_requests_to_evaluate=10)
    # 8 success, 4 failures = 33.3% > 5.0%
    for _ in range(8):
        engine.record_request_outcome(is_error=False)
    for _ in range(4):
        engine.record_request_outcome(is_error=True)
        
    assert engine.circuit_tripped is True
    assert engine.active_variant == "BASELINE_V1"
    assert "BASELINE_V1" in engine.route_inference("Query")
    assert len(engine.incident_log) == 1
    assert engine.incident_log[0]["action"] == "CIRCUIT_BREAKER_ROLLBACK"

def test_min_requests_guard_prevents_premature_trip():
    engine = AIIncidentRollbackEngine(error_threshold_pct=5.0, min_requests_to_evaluate=10)
    # 1 request that fails -> 100% error rate, but < 10 min requests -> do not trip yet!
    engine.record_request_outcome(is_error=True)
    
    assert engine.circuit_tripped is False
    assert engine.active_variant == "CANDIDATE_V2"

def test_post_trip_ignores_further_candidate_errors():
    engine = AIIncidentRollbackEngine(error_threshold_pct=5.0, min_requests_to_evaluate=2)
    engine.record_request_outcome(is_error=True)
    engine.record_request_outcome(is_error=True) # Trips here
    
    assert engine.circuit_tripped is True
    assert len(engine.incident_log) == 1
    
    # Additional recordings should be ignored since circuit is already tripped
    engine.record_request_outcome(is_error=True)
    assert len(engine.incident_log) == 1

Troubleshooting

Troubleshooting - Day 342\nEnsure circuit checks min_requests before evaluating error percentages.\n

Security notes

Security - Day 342\nIncident engines enforce fast automated isolation of failing model variants.\n