Deployment, MLOps, and Security › Containers and Cloud › Day 336
Hands-on lab — Day 336: A Containerized AI Deployment
- ← Back to the Day 336 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deployment-mlops-and-security/day-336-a-containerized-ai-deployment/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/ai_gateway.py Test
bash tests/run_tests.sh File tree
examples/ai_gateway.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/ai_gateway.py tests/run_tests.sh tests/test_ai_gateway.py troubleshooting.md
Lab README
Day 336 Lab: A Deployed AI System
Lesson
- Lesson title: A Containerized AI Deployment
- Day number: 336 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-336-a-containerized-ai-deployment
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-336-a-containerized-ai-deploymentwhen the site is running.
Purpose
Build a Production AI Serving Gateway in Python featuring Least-Outstanding-Requests (LOR) load balancing, multi-replica health tracking, and circuit breaking.
Learning objectives
- Register and monitor multi-replica GPU inference workers.
- Implement Least-Outstanding-Requests (LOR) load balancing.
- Manage circuit breaker state transitions (CLOSED, OPEN, HALF_OPEN).
- Enforce fallback responses during backend service degradation.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/ai_gateway.py: Starter implementation skeletonexamples/ai_gateway.py: Verified reference implementationtests/test_ai_gateway.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/ai_gateway.py
What the commands do
- Evaluates LOR routing, simulates replica failures, and verifies circuit breaker recovery.
Expected output
{'status': 'ROUTED', 'assigned_replica': 'w1', 'active_on_replica': 1, 'circuit_state': 'CLOSED'}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- LOR dynamic load balancing selecting least-active replicas
- Active request count increment and decrement
- Circuit breaker tripping to OPEN upon failure threshold
- Graceful fallback response emission when circuit is OPEN
- Self-healing transition to HALF_OPEN and CLOSED after probe success
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure active_requests is decremented in complete_request.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement weighted capacity LOR balancing.
Navigation
Day number: 336 of 365
Expected output
FIELDS.md
# Measured Values
- `requests_routed`: 12
- `circuit_trips`: 1
examples-run.txt
{'status': 'ROUTED', 'assigned_replica': 'w1', 'active_on_replica': 1, 'circuit_state': 'CLOSED'}
measured-values.txt
requests_routed=12
circuit_trips=1
run-output.txt
{'status': 'ROUTED', 'assigned_replica': 'w1', 'active_on_replica': 1, 'circuit_state': 'CLOSED'}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.ai_gateway.ProductionAIGateway object at 0x1089b6780>
replica_id = 'gpu_1'
def register_replica(self, replica_id: str):
> raise NotImplementedError('TASK 1: implement register_replica.')
E NotImplementedError: TASK 1: implement register_replica.
examples/ai_gateway.py:30: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_ai_gateway.py::test_lor_load_balancing_selects_least_active
FAILED tests/test_ai_gateway.py::test_active_request_decrement_on_complete - ...
FAILED tests/test_ai_gateway.py::test_circuit_breaker_tripping_on_consecutive_failures
FAILED tests/test_ai_gateway.py::test_circuit_breaker_open_returns_fallback
FAILED tests/test_ai_gateway.py::test_circuit_breaker_half_open_recovery - No...
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-336-a-containerized-ai-deployment
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_ai_gateway.py::test_lor_load_balancing_selects_least_active PASSED [ 20%]
tests/test_ai_gateway.py::test_active_request_decrement_on_complete PASSED [ 40%]
tests/test_ai_gateway.py::test_circuit_breaker_tripping_on_consecutive_failures PASSED [ 60%]
tests/test_ai_gateway.py::test_circuit_breaker_open_returns_fallback PASSED [ 80%]
tests/test_ai_gateway.py::test_circuit_breaker_half_open_recovery PASSED [100%]
============================== 5 passed in 0.01s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-336-a-containerized-ai-deployment
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_ai_gateway.py::test_lor_load_balancing_selects_least_active PASSED [ 20%]
tests/test_ai_gateway.py::test_active_request_decrement_on_complete PASSED [ 40%]
tests/test_ai_gateway.py::test_circuit_breaker_tripping_on_consecutive_failures PASSED [ 60%]
tests/test_ai_gateway.py::test_circuit_breaker_open_returns_fallback PASSED [ 80%]
tests/test_ai_gateway.py::test_circuit_breaker_half_open_recovery PASSED [100%]
============================== 5 passed in 0.01s ===============================
Source files
examples/ai_gateway.py (3235 bytes)
import time
from typing import Dict, Any, List, Optional
class ReplicaWorker:
def __init__(self, replica_id: str):
self.replica_id = replica_id
self.active_requests = 0
self.is_healthy = True
self.consecutive_failures = 0
class ProductionAIGateway:
def __init__(self, failure_threshold: int = 3, cooldown_seconds: float = 5.0):
self.failure_threshold = int(failure_threshold)
self.cooldown_sec = float(cooldown_seconds)
self.replicas: Dict[str, ReplicaWorker] = {}
self.circuit_state = "CLOSED"
self.last_state_change = time.time()
self.telemetry_metrics = {
"total_routed": 0,
"circuit_fallbacks": 0,
"lor_selections": []
}
def register_replica(self, replica_id: str):
self.replicas[replica_id] = ReplicaWorker(replica_id)
def select_replica_lor(self) -> Optional[ReplicaWorker]:
healthy = [r for r in self.replicas.values() if r.is_healthy]
if not healthy:
return None
return min(healthy, key=lambda r: r.active_requests)
def route_request(self, prompt: str, current_time: float) -> Dict[str, Any]:
if self.circuit_state == "OPEN":
if (current_time - self.last_state_change) >= self.cooldown_sec:
self.circuit_state = "HALF_OPEN"
self.last_state_change = current_time
else:
self.telemetry_metrics["circuit_fallbacks"] += 1
return {"status": "FALLBACK", "response": "Cached graceful fallback response."}
worker = self.select_replica_lor()
if not worker:
self.circuit_state = "OPEN"
self.last_state_change = current_time
self.telemetry_metrics["circuit_fallbacks"] += 1
return {"status": "FALLBACK", "response": "All replicas unhealthy; returning fallback."}
worker.active_requests += 1
self.telemetry_metrics["total_routed"] += 1
self.telemetry_metrics["lor_selections"].append(worker.replica_id)
return {
"status": "ROUTED",
"assigned_replica": worker.replica_id,
"active_on_replica": worker.active_requests,
"circuit_state": self.circuit_state
}
def complete_request(self, replica_id: str, success: bool, current_time: float):
worker = self.replicas.get(replica_id)
if not worker:
return
worker.active_requests = max(0, worker.active_requests - 1)
if success:
worker.consecutive_failures = 0
if self.circuit_state == "HALF_OPEN":
self.circuit_state = "CLOSED"
self.last_state_change = current_time
else:
worker.consecutive_failures += 1
if worker.consecutive_failures >= self.failure_threshold:
worker.is_healthy = False
if all(not r.is_healthy for r in self.replicas.values()):
self.circuit_state = "OPEN"
self.last_state_change = current_time
if __name__ == "__main__":
g = ProductionAIGateway(2, 5.0)
g.register_replica("w1")
print(g.route_request("hi", 100.0))
metadata.yml (397 bytes)
lesson_id: D336
day: 336
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/ai_gateway.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/ai_gateway.py (1691 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import time
from typing import Dict, Any, List, Optional
class ReplicaWorker:
def __init__(self, replica_id: str):
self.replica_id = replica_id
self.active_requests = 0
self.is_healthy = True
self.consecutive_failures = 0
class ProductionAIGateway:
def __init__(self, failure_threshold: int=3, cooldown_seconds: float=5.0):
self.failure_threshold = int(failure_threshold)
self.cooldown_sec = float(cooldown_seconds)
self.replicas: Dict[str, ReplicaWorker] = {}
self.circuit_state = 'CLOSED'
self.last_state_change = time.time()
self.telemetry_metrics = {'total_routed': 0, 'circuit_fallbacks': 0, 'lor_selections': []}
def register_replica(self, replica_id: str):
raise NotImplementedError('TASK 1: implement register_replica.')
def select_replica_lor(self) -> Optional[ReplicaWorker]:
raise NotImplementedError('TASK 2: implement select_replica_lor.')
def route_request(self, prompt: str, current_time: float) -> Dict[str, Any]:
raise NotImplementedError('TASK 3: implement route_request.')
def complete_request(self, replica_id: str, success: bool, current_time: float):
raise NotImplementedError('TASK 4: implement complete_request.')
if __name__ == '__main__':
g = ProductionAIGateway(2, 5.0)
g.register_replica('w1')
print(g.route_request('hi', 100.0))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_ai_gateway.py (2723 bytes)
import pytest
from examples.ai_gateway import ProductionAIGateway
def test_lor_load_balancing_selects_least_active():
gateway = ProductionAIGateway()
gateway.register_replica("gpu_1")
gateway.register_replica("gpu_2")
# First request goes to gpu_1 (active 0 -> 1)
res1 = gateway.route_request("p1", 100.0)
assert res1["status"] == "ROUTED"
assert res1["assigned_replica"] == "gpu_1"
# Second request goes to gpu_2 (active 0 < active 1)
res2 = gateway.route_request("p2", 100.1)
assert res2["status"] == "ROUTED"
assert res2["assigned_replica"] == "gpu_2"
def test_active_request_decrement_on_complete():
gateway = ProductionAIGateway()
gateway.register_replica("gpu_1")
gateway.route_request("p1", 100.0)
assert gateway.replicas["gpu_1"].active_requests == 1
gateway.complete_request("gpu_1", success=True, current_time=101.0)
assert gateway.replicas["gpu_1"].active_requests == 0
def test_circuit_breaker_tripping_on_consecutive_failures():
gateway = ProductionAIGateway(failure_threshold=2)
gateway.register_replica("gpu_1")
gateway.route_request("p1", 100.0)
gateway.complete_request("gpu_1", success=False, current_time=100.5)
assert gateway.replicas["gpu_1"].is_healthy is True
gateway.route_request("p2", 101.0)
gateway.complete_request("gpu_1", success=False, current_time=101.5)
assert gateway.replicas["gpu_1"].is_healthy is False
assert gateway.circuit_state == "OPEN"
def test_circuit_breaker_open_returns_fallback():
gateway = ProductionAIGateway(failure_threshold=1, cooldown_seconds=10.0)
gateway.register_replica("gpu_1")
gateway.route_request("p1", 100.0)
gateway.complete_request("gpu_1", success=False, current_time=101.0) # Tripped to OPEN
# Attempt request during cooldown
res = gateway.route_request("p2", 105.0)
assert res["status"] == "FALLBACK"
assert "fallback" in res["response"].lower()
def test_circuit_breaker_half_open_recovery():
gateway = ProductionAIGateway(failure_threshold=1, cooldown_seconds=10.0)
gateway.register_replica("gpu_1")
gateway.route_request("p1", 100.0)
gateway.complete_request("gpu_1", success=False, current_time=101.0)
# Restore health for probe test
gateway.replicas["gpu_1"].is_healthy = True
# At t=115 (14s > 10s cooldown), next request transitions to HALF_OPEN
res = gateway.route_request("probe", 115.0)
assert res["status"] == "ROUTED"
assert res["circuit_state"] == "HALF_OPEN"
# Successful probe resets to CLOSED
gateway.complete_request("gpu_1", success=True, current_time=116.0)
assert gateway.circuit_state == "CLOSED"