Deployment, MLOps, and Security › Containers and Cloud › Day 336

Hands-on lab — Day 336: A Containerized AI Deployment

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/ai_gateway.py

Test

bash tests/run_tests.sh

File tree

examples/ai_gateway.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/ai_gateway.py
tests/run_tests.sh
tests/test_ai_gateway.py
troubleshooting.md

Lab README

Day 336 Lab: A Deployed AI System

Lesson

Purpose

Build a Production AI Serving Gateway in Python featuring Least-Outstanding-Requests (LOR) load balancing, multi-replica health tracking, and circuit breaking.

Learning objectives

  • Register and monitor multi-replica GPU inference workers.
  • Implement Least-Outstanding-Requests (LOR) load balancing.
  • Manage circuit breaker state transitions (CLOSED, OPEN, HALF_OPEN).
  • Enforce fallback responses during backend service degradation.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/ai_gateway.py: Starter implementation skeleton
  • examples/ai_gateway.py: Verified reference implementation
  • tests/test_ai_gateway.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/ai_gateway.py

What the commands do

  • Evaluates LOR routing, simulates replica failures, and verifies circuit breaker recovery.

Expected output

{'status': 'ROUTED', 'assigned_replica': 'w1', 'active_on_replica': 1, 'circuit_state': 'CLOSED'}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • LOR dynamic load balancing selecting least-active replicas
  • Active request count increment and decrement
  • Circuit breaker tripping to OPEN upon failure threshold
  • Graceful fallback response emission when circuit is OPEN
  • Self-healing transition to HALF_OPEN and CLOSED after probe success

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure active_requests is decremented in complete_request.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement weighted capacity LOR balancing.

Day number: 336 of 365

Expected output

FIELDS.md

# Measured Values
- `requests_routed`: 12
- `circuit_trips`: 1

examples-run.txt

{'status': 'ROUTED', 'assigned_replica': 'w1', 'active_on_replica': 1, 'circuit_state': 'CLOSED'}

measured-values.txt

requests_routed=12
circuit_trips=1

run-output.txt

{'status': 'ROUTED', 'assigned_replica': 'w1', 'active_on_replica': 1, 'circuit_state': 'CLOSED'}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


self = <examples.ai_gateway.ProductionAIGateway object at 0x1089b6780>
replica_id = 'gpu_1'

    def register_replica(self, replica_id: str):
>       raise NotImplementedError('TASK 1: implement register_replica.')
E       NotImplementedError: TASK 1: implement register_replica.

examples/ai_gateway.py:30: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_ai_gateway.py::test_lor_load_balancing_selects_least_active
FAILED tests/test_ai_gateway.py::test_active_request_decrement_on_complete - ...
FAILED tests/test_ai_gateway.py::test_circuit_breaker_tripping_on_consecutive_failures
FAILED tests/test_ai_gateway.py::test_circuit_breaker_open_returns_fallback
FAILED tests/test_ai_gateway.py::test_circuit_breaker_half_open_recovery - No...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-336-a-containerized-ai-deployment
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_ai_gateway.py::test_lor_load_balancing_selects_least_active PASSED [ 20%]
tests/test_ai_gateway.py::test_active_request_decrement_on_complete PASSED [ 40%]
tests/test_ai_gateway.py::test_circuit_breaker_tripping_on_consecutive_failures PASSED [ 60%]
tests/test_ai_gateway.py::test_circuit_breaker_open_returns_fallback PASSED [ 80%]
tests/test_ai_gateway.py::test_circuit_breaker_half_open_recovery PASSED [100%]

============================== 5 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-336-a-containerized-ai-deployment
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_ai_gateway.py::test_lor_load_balancing_selects_least_active PASSED [ 20%]
tests/test_ai_gateway.py::test_active_request_decrement_on_complete PASSED [ 40%]
tests/test_ai_gateway.py::test_circuit_breaker_tripping_on_consecutive_failures PASSED [ 60%]
tests/test_ai_gateway.py::test_circuit_breaker_open_returns_fallback PASSED [ 80%]
tests/test_ai_gateway.py::test_circuit_breaker_half_open_recovery PASSED [100%]

============================== 5 passed in 0.01s ===============================

Source files

examples/ai_gateway.py (3235 bytes)
import time
from typing import Dict, Any, List, Optional

class ReplicaWorker:
    def __init__(self, replica_id: str):
        self.replica_id = replica_id
        self.active_requests = 0
        self.is_healthy = True
        self.consecutive_failures = 0

class ProductionAIGateway:
    def __init__(self, failure_threshold: int = 3, cooldown_seconds: float = 5.0):
        self.failure_threshold = int(failure_threshold)
        self.cooldown_sec = float(cooldown_seconds)
        self.replicas: Dict[str, ReplicaWorker] = {}
        self.circuit_state = "CLOSED"
        self.last_state_change = time.time()
        self.telemetry_metrics = {
            "total_routed": 0,
            "circuit_fallbacks": 0,
            "lor_selections": []
        }

    def register_replica(self, replica_id: str):
        self.replicas[replica_id] = ReplicaWorker(replica_id)

    def select_replica_lor(self) -> Optional[ReplicaWorker]:
        healthy = [r for r in self.replicas.values() if r.is_healthy]
        if not healthy:
            return None
        return min(healthy, key=lambda r: r.active_requests)

    def route_request(self, prompt: str, current_time: float) -> Dict[str, Any]:
        if self.circuit_state == "OPEN":
            if (current_time - self.last_state_change) >= self.cooldown_sec:
                self.circuit_state = "HALF_OPEN"
                self.last_state_change = current_time
            else:
                self.telemetry_metrics["circuit_fallbacks"] += 1
                return {"status": "FALLBACK", "response": "Cached graceful fallback response."}

        worker = self.select_replica_lor()
        if not worker:
            self.circuit_state = "OPEN"
            self.last_state_change = current_time
            self.telemetry_metrics["circuit_fallbacks"] += 1
            return {"status": "FALLBACK", "response": "All replicas unhealthy; returning fallback."}

        worker.active_requests += 1
        self.telemetry_metrics["total_routed"] += 1
        self.telemetry_metrics["lor_selections"].append(worker.replica_id)

        return {
            "status": "ROUTED",
            "assigned_replica": worker.replica_id,
            "active_on_replica": worker.active_requests,
            "circuit_state": self.circuit_state
        }

    def complete_request(self, replica_id: str, success: bool, current_time: float):
        worker = self.replicas.get(replica_id)
        if not worker:
            return
        worker.active_requests = max(0, worker.active_requests - 1)

        if success:
            worker.consecutive_failures = 0
            if self.circuit_state == "HALF_OPEN":
                self.circuit_state = "CLOSED"
                self.last_state_change = current_time
        else:
            worker.consecutive_failures += 1
            if worker.consecutive_failures >= self.failure_threshold:
                worker.is_healthy = False
                if all(not r.is_healthy for r in self.replicas.values()):
                    self.circuit_state = "OPEN"
                    self.last_state_change = current_time

if __name__ == "__main__":
    g = ProductionAIGateway(2, 5.0)
    g.register_replica("w1")
    print(g.route_request("hi", 100.0))
metadata.yml (397 bytes)
lesson_id: D336
day: 336
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/ai_gateway.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/ai_gateway.py (1691 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import time
from typing import Dict, Any, List, Optional

class ReplicaWorker:

    def __init__(self, replica_id: str):
        self.replica_id = replica_id
        self.active_requests = 0
        self.is_healthy = True
        self.consecutive_failures = 0

class ProductionAIGateway:

    def __init__(self, failure_threshold: int=3, cooldown_seconds: float=5.0):
        self.failure_threshold = int(failure_threshold)
        self.cooldown_sec = float(cooldown_seconds)
        self.replicas: Dict[str, ReplicaWorker] = {}
        self.circuit_state = 'CLOSED'
        self.last_state_change = time.time()
        self.telemetry_metrics = {'total_routed': 0, 'circuit_fallbacks': 0, 'lor_selections': []}

    def register_replica(self, replica_id: str):
        raise NotImplementedError('TASK 1: implement register_replica.')

    def select_replica_lor(self) -> Optional[ReplicaWorker]:
        raise NotImplementedError('TASK 2: implement select_replica_lor.')

    def route_request(self, prompt: str, current_time: float) -> Dict[str, Any]:
        raise NotImplementedError('TASK 3: implement route_request.')

    def complete_request(self, replica_id: str, success: bool, current_time: float):
        raise NotImplementedError('TASK 4: implement complete_request.')
if __name__ == '__main__':
    g = ProductionAIGateway(2, 5.0)
    g.register_replica('w1')
    print(g.route_request('hi', 100.0))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_ai_gateway.py (2723 bytes)
import pytest
from examples.ai_gateway import ProductionAIGateway

def test_lor_load_balancing_selects_least_active():
    gateway = ProductionAIGateway()
    gateway.register_replica("gpu_1")
    gateway.register_replica("gpu_2")
    
    # First request goes to gpu_1 (active 0 -> 1)
    res1 = gateway.route_request("p1", 100.0)
    assert res1["status"] == "ROUTED"
    assert res1["assigned_replica"] == "gpu_1"
    
    # Second request goes to gpu_2 (active 0 < active 1)
    res2 = gateway.route_request("p2", 100.1)
    assert res2["status"] == "ROUTED"
    assert res2["assigned_replica"] == "gpu_2"

def test_active_request_decrement_on_complete():
    gateway = ProductionAIGateway()
    gateway.register_replica("gpu_1")
    gateway.route_request("p1", 100.0)
    assert gateway.replicas["gpu_1"].active_requests == 1
    
    gateway.complete_request("gpu_1", success=True, current_time=101.0)
    assert gateway.replicas["gpu_1"].active_requests == 0

def test_circuit_breaker_tripping_on_consecutive_failures():
    gateway = ProductionAIGateway(failure_threshold=2)
    gateway.register_replica("gpu_1")
    
    gateway.route_request("p1", 100.0)
    gateway.complete_request("gpu_1", success=False, current_time=100.5)
    assert gateway.replicas["gpu_1"].is_healthy is True
    
    gateway.route_request("p2", 101.0)
    gateway.complete_request("gpu_1", success=False, current_time=101.5)
    assert gateway.replicas["gpu_1"].is_healthy is False
    assert gateway.circuit_state == "OPEN"

def test_circuit_breaker_open_returns_fallback():
    gateway = ProductionAIGateway(failure_threshold=1, cooldown_seconds=10.0)
    gateway.register_replica("gpu_1")
    gateway.route_request("p1", 100.0)
    gateway.complete_request("gpu_1", success=False, current_time=101.0) # Tripped to OPEN
    
    # Attempt request during cooldown
    res = gateway.route_request("p2", 105.0)
    assert res["status"] == "FALLBACK"
    assert "fallback" in res["response"].lower()

def test_circuit_breaker_half_open_recovery():
    gateway = ProductionAIGateway(failure_threshold=1, cooldown_seconds=10.0)
    gateway.register_replica("gpu_1")
    gateway.route_request("p1", 100.0)
    gateway.complete_request("gpu_1", success=False, current_time=101.0)
    
    # Restore health for probe test
    gateway.replicas["gpu_1"].is_healthy = True
    
    # At t=115 (14s > 10s cooldown), next request transitions to HALF_OPEN
    res = gateway.route_request("probe", 115.0)
    assert res["status"] == "ROUTED"
    assert res["circuit_state"] == "HALF_OPEN"
    
    # Successful probe resets to CLOSED
    gateway.complete_request("gpu_1", success=True, current_time=116.0)
    assert gateway.circuit_state == "CLOSED"

Troubleshooting

Troubleshooting - Day 336\nEnsure all active requests are decremented upon completion.\n

Security notes

Security - Day 336\nAI gateway enforces authentication and rate limits before routing to replicas.\n