Deployment, MLOps, and Security β€Ί Containers and Cloud β€Ί Day 336

Day 336: A Containerized AI Deployment

Day 336 of 365 β€” A Containerized AI Deployment

Unify all deployment, containerization, and scaling principles: architect an end-to-end multi-replica AI serving gateway with load balancing, circuit breaking, health probes, and live telemetry in Python.

Course
Deployment, MLOps, and Security
Category
Containers and Cloud
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deployment-mlops-and-security/day-336-a-containerized-ai-deployment

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deployment-mlops-and-security/day-336-a-containerized-ai-deployment
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Building isolated inference scripts, writing standalone Dockerfiles, and tuning quantization parameters are meaningless if they cannot operate cohesively in a mission-critical production environment. A fragile deployment causes:

Building a Unified Deployed AI Serving System integrates containerization, vLLM continuous serving, KEDA queue autoscaling, Least-Outstanding-Requests (LOR) load balancing, circuit breaking, and complete SRE telemetry to deliver enterprise-grade 99.99% availability.

End-to-End Production AI Serving Architecture showing Client Requests hitting AI Gateway Reverse Proxy with LOR Load Balancing, Multi-Replica vLLM GPU Pods, Prometheus Telemetry, and Circuit Breaker Failover

The idea in plain language

Diagram: the four responsibilities of an AI serving gateway

Think of a fully deployed AI serving system like an international commercial airline flight operations center:

Historical background

Diagram: the progression from Flask wrappers to intelligent AI gateways

AI serving systems evolved through three generations of production architecture:

1. Monolithic Flask / Gunicorn Wrappers (2018–2021)

Engineers wrapped PyTorch models inside single Gunicorn containers. Load balancers used simple round-robin routing. Any worker exception caused unhandled 500 errors and dropped streaming connections.

2. Multi-Container Microservices (2021–2023)

Companies separated frontend gateways from backend Triton / TorchServe inference pods. However, load balancing remained unaware of internal GPU KV cache saturation.

3. Modern AI Reverse Proxies & Intelligent Gateways (2024–Present)

Modern production systems deploy dedicated AI Gateways (Envoy AI Gateway, Cloudflare AI Gateway, Kong) featuring LOR concurrency routing, streaming SSE pass-through, circuit breaking, and unified SRE metrics.

What it is β€” and what it is not

Let us define the core architecture of an enterprise deployed AI system:

What a Deployed AI System Is:

What a Deployed AI System Is Not:

Diagram: a production AI deployment compared with a container exposed directly

+-------------------------------------------------------------------------------+
|                    PRODUCTION AI SERVING COMPONENT MATRIX                     |
+-------------------+-----------------------+-----------------------------------+
| Component         | Primary Function      | Reliability Guardrail             |
+-------------------+-----------------------+-----------------------------------+
| AI Gateway Proxy  | Auth, Rate Limiting,  | Least-Outstanding-Requests (LOR)  |
|                   | Routing & Caching     | Load Balancing                    |
| Circuit Breaker   | Cascading Failure     | Tripping to OPEN on 50% errors;   |
|                   | Prevention            | graceful cached fallbacks         |
| Kubernetes Pods   | Multi-Replica vLLM    | Startup Probe (180s) + PreStop    |
|                   | GPU Serving           | Graceful Connection Draining (30s)|
| KEDA Autoscaler   | Queue-Depth Elasticity| Scales replicas based on          |
|                   |                       | `vllm:num_requests_waiting`       |
| Prometheus / SRE  | Real-Time Observability| Alerts on TTFT > 500ms, Error > 1%|
+-------------------+-----------------------+-----------------------------------+

Why it was created and what problems it solves

A unified deployment architecture solves the three fatal failure modes of production AI services:

1. Eliminating Tail Latency Spikes via LOR Routing

Because LLM query processing times vary from 50ms to 20 seconds, routing requests to the worker with the fewest active streams prevents queue congestion and slashes P99 latency by 65%.

Diagram: round-robin routing compared with least-outstanding-requests routing

2. Preventing CrashLoopBackOff via Startup Probes

By allowing up to 180 seconds for initial model weight loading and CUDA context initialization, Startup Probes prevent Kubernetes from prematurely killing healthy initializing pods.

Diagram: the three Kubernetes probes and the distinct question each answers

3. Seamless Zero-Downtime Rolling Deployments

With graceful connection draining (PreStop hooks), when a new model version is rolled out, existing streaming token responses complete without truncation.

How it works

Diagram: the closed, open and half-open states of a circuit breaker

Let us inspect the complete Circuit Breaker and Health Probe State Machine:

[CLOSED STATE: Healthy]
  β”‚ - 100% of user traffic forwarded to backend GPU replicas
  β”‚ - Readiness Probes return HTTP 200 OK
  β”‚
  β”œβ”€β”€ 5 Consecutive Backend 500 Errors Occur βž” TRIPS CIRCUIT BREAKER!
  β”‚
  β–Ό
[OPEN STATE: Failing Fast]
  β”‚ - Zero traffic forwarded to failing backend
  β”‚ - AI Gateway returns immediate fallback / cached response (0ms delay)
  β”‚ - Cooldown Timer starts: 30 seconds
  β”‚
  β”œβ”€β”€ 30-Second Cooldown Expires βž” TRANSITIONS TO HALF-OPEN
  β”‚
  β–Ό
[HALF-OPEN STATE: Canary Trial]
  β”‚ - Dispatches 1 single canary probe request to backend
  β”‚
  β”œβ”€β”€ Probe Succeeds βž” Resets failure counters βž” Returns to CLOSED (Normal)
  β”‚
  └── Probe Fails βž” Re-trips immediately βž” Returns to OPEN (30s Cooldown)

Circuit Breaker and Health Probe State Machine showing Closed (Normal), Open (Failing Fast), and Half-Open (Canary Testing) States with Startup and Readiness Probes

An everyday analogy

Diagram: hospital triage as an analogy for gateway routing and circuit breaking

Think of a deployed AI system like a multi-lane emergency room at a major hospital:

Examples in practice

Diagram: the path of a request through circuit evaluation, replica selection and dispatch

Let us build a complete Python Production AI Serving Gateway Simulator with Least-Outstanding-Requests load balancing, multi-replica health tracking, circuit breaking, and telemetry recording:

import time
from typing import Dict, Any, List, Optional

class ReplicaWorker:
    def __init__(self, replica_id: str):
        self.replica_id = replica_id
        self.active_requests = 0
        self.is_healthy = True
        self.consecutive_failures = 0

class ProductionAIGateway:
    def __init__(self, failure_threshold: int = 3, cooldown_seconds: float = 5.0):
        self.failure_threshold = int(failure_threshold)
        self.cooldown_sec = float(cooldown_seconds)
        self.replicas: Dict[str, ReplicaWorker] = {}
        self.circuit_state = "CLOSED"  # CLOSED, OPEN, HALF_OPEN
        self.last_state_change = time.time()
        self.telemetry_metrics = {
            "total_routed": 0,
            "circuit_fallbacks": 0,
            "lor_selections": []
        }

    def register_replica(self, replica_id: str):
        self.replicas[replica_id] = ReplicaWorker(replica_id)

    def select_replica_lor(self) -> Optional[ReplicaWorker]:
        # Filter healthy replicas
        healthy = [r for r in self.replicas.values() if r.is_healthy]
        if not healthy:
            return None
        # Select replica with least outstanding active requests
        return min(healthy, key=lambda r: r.active_requests)

    def route_request(self, prompt: str, current_time: float) -> Dict[str, Any]:
        # 1. Evaluate Circuit Breaker State
        if self.circuit_state == "OPEN":
            if (current_time - self.last_state_change) >= self.cooldown_sec:
                self.circuit_state = "HALF_OPEN"
                self.last_state_change = current_time
            else:
                self.telemetry_metrics["circuit_fallbacks"] += 1
                return {"status": "FALLBACK", "response": "Cached graceful fallback response."}

        # 2. Select Replica via LOR
        worker = self.select_replica_lor()
        if not worker:
            self.circuit_state = "OPEN"
            self.last_state_change = current_time
            self.telemetry_metrics["circuit_fallbacks"] += 1
            return {"status": "FALLBACK", "response": "All replicas unhealthy; returning fallback."}

        # 3. Dispatch Request
        worker.active_requests += 1
        self.telemetry_metrics["total_routed"] += 1
        self.telemetry_metrics["lor_selections"].append(worker.replica_id)

        return {
            "status": "ROUTED",
            "assigned_replica": worker.replica_id,
            "active_on_replica": worker.active_requests,
            "circuit_state": self.circuit_state
        }

    def complete_request(self, replica_id: str, success: bool, current_time: float):
        worker = self.replicas.get(replica_id)
        if not worker:
            return
        worker.active_requests = max(0, worker.active_requests - 1)

        if success:
            worker.consecutive_failures = 0
            if self.circuit_state == "HALF_OPEN":
                self.circuit_state = "CLOSED"
                self.last_state_change = current_time
        else:
            worker.consecutive_failures += 1
            if worker.consecutive_failures >= self.failure_threshold:
                worker.is_healthy = False
                # Check if all replicas failed
                if all(not r.is_healthy for r in self.replicas.values()):
                    self.circuit_state = "OPEN"
                    self.last_state_change = current_time

if __name__ == "__main__":
    gw = ProductionAIGateway(failure_threshold=2, cooldown_seconds=5.0)
    gw.register_replica("gpu_worker_1")
    gw.register_replica("gpu_worker_2")

    # Worker 1 takes a long request (active=1)
    res1 = gw.route_request("prompt 1", 100.0)
    # Worker 2 should be selected for next request (active=0 < 1)
    res2 = gw.route_request("prompt 2", 100.1)
    
    print("Route 1:", res1)
    print("Route 2 (LOR):", res2)

Production Architecture Case Study: Enterprise Multi-AZ AI Cluster

Consider an enterprise fintech infrastructure serving 5,000 requests per second across 3 availability zones with zero downtime during rolling upgrades.

+-------------------------------------------------------------------------------+
|                    ENTERPRISE AI DEPLOYMENT BENCHMARKS                        |
+-------------------+-----------------------+-----------------------------------+
| Architecture      | Availability SLA      | P99 Latency under Surge           |
+-------------------+-----------------------+-----------------------------------+
| Monolithic PyTorch| 98.2% (Frequent Drops)| 4,850 ms (Queue Backlogs)         |
| Naive Round-Robin | 99.1% (504s on Upgrd) | 2,100 ms (Imbalanced Load)        |
| AI Gateway + LOR +| 99.99% (Zero Downtime)| 320 ms (Smooth Balanced Traffic)  |
| Circuit Breaker   |                       |                                   |
+-------------------+-----------------------+-----------------------------------+

1. Kubernetes Production Pod Spec with Health Probes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-inference-worker
spec:
  replicas: 4
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: 1
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 30"]
        startupProbe:
          httpGet:
            path: /health
            port: 8000
          failureThreshold: 36
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          periodSeconds: 10

Detailed Failure Mode Taxonomy: Deployed AI System Hazards

Diagram: the four production failure modes of an AI serving system and their controls

+-------------------------------------------------------------------------------+
|                      DEPLOYED SYSTEM FAILURE TAXONOMY                         |
+-------------------+-----------------------+-----------------------------------+
| Failure Mode      | Root Cause Analysis   | Architectural Mitigation          |
+-------------------+-----------------------+-----------------------------------+
| CrashLoopBackOff  | Liveness probe kills  | Configure Startup Probe with      |
| on Pod Boot       | pod during weight dl  | 180s failureThreshold             |
| Truncated Stream  | Pod terminated during | Implement `preStop` 30s connection|
| on Rolling Update | rolling restart       | draining sleep hook in deployment |
| Cascading Worker  | 1 failed worker dumps | Circuit breaker trips to OPEN;    |
| Meltdown          | 100% traffic on rest  | fail-fast with cached fallbacks   |
| Congested Worker  | Round-Robin routing   | Enforce Least-Outstanding-Requests|
| Latency Spike     | ignores active streams| (LOR) load balancing at gateway   |
+-------------------+-----------------------+-----------------------------------+

Deep Architectural Breakdown: AI Gateway Architecture and SRE Observability

An enterprise AI deployment requires a unified gateway layer bridging client traffic with backend GPU clusters:

1. Least-Outstanding-Requests (LOR) Load Balancing Algorithm

Let W = \{w_1, w_2, \dots, w_k\} be the set of healthy worker replicas, and A(w_i) be the count of active in-flight streaming requests on worker w_i.

2. Complete SRE Telemetry & Alerting Matrix

Production AI systems continuously monitor the four golden signals:

Diagram: the four telemetry signals an AI serving system must record

Implications: security, privacy, performance, scalability, and cost

1. Centralized Audit & Prompt Injection Scrubbing

The AI Gateway acts as the single security perimeter where all incoming prompts undergo API key authentication, rate-limiting, and automated prompt-injection guardrail screening.

2. High-Availability Zero-Downtime Economics

By pairing intelligent LOR routing with KEDA queue autoscaling, infrastructure maintains 99.99% availability while keeping idle GPU footprint minimal.

Alternatives: free, open source, and commercial

Tool / FrameworkTypeBest Suited ForKey Capabilities
Envoy AI GatewayOpen SourceHigh-Performance RoutingLOR balancing, rate limiting, SSE stream
Kong AI GatewayOpen Source / CommEnterprise API GatewayMulti-LLM routing, failover, analytics
Cloudflare AI GateCommercial SaaSEdge Multi-Model ProxyGlobal caching, rate-limiting, logging
LiteLLM ProxyOpen SourceMulti-Provider Unified APIOpenAI-compatible proxy with failovers
+-----------------------+-----------------------+-----------------------+
| Dimension             | Round-Robin Proxy     | AI Gateway with LOR   |
+-----------------------+-----------------------+-----------------------+
| Routing Logic         | Cyclic Index (1, 2, 3)| Min Active Streams    |
| Awareness of GPU Load | Zero Awareness        | Real-Time Concurrency |
| Failure Protection    | Drops connections     | Circuit Breaker Open  |
| Pod Startup Safety    | Vulnerable to restarts| Startup Probes (180s) |
+-----------------------+-----------------------+-----------------------+

When to use it β€” and when not to

Deploy a Full AI Serving Gateway for:

Do NOT require multi-replica gateways for:

Diagram: when to deploy a full serving gateway and when a single container suffices

Production Checklist: Deployed AI Serving System Readiness

Diagram: the readiness checklist for a deployed AI serving system

Enterprise Gateway Routing Policies

The AI connection

Knowledge check

  1. Why is Least-Outstanding-Requests (LOR) superior to Round-Robin for LLMs? Because LLM request durations vary drastically; LOR routes traffic to the worker with the fewest active streams, preventing queue congestion.
  2. What problem do Startup Probes solve? They give models up to 180 seconds to download weights and initialize VRAM without being prematurely killed by liveness probes.
  3. What happens when a Circuit Breaker trips to OPEN? It stops forwarding traffic to failing backends and fails fast with graceful fallback responses, preventing cascading outages.

Hands-on exercise

In this lab, you will build an End-to-End Production AI Serving Gateway in Python. Your implementation will:

  1. Register multiple backend GPU inference replica workers.
  2. Implement Least-Outstanding-Requests (LOR) load balancing routing traffic to the replica with the lowest active streams.
  3. Implement a Circuit Breaker state machine transitioning from CLOSED to OPEN upon consecutive worker failures.
  4. Verify graceful fallback responses during OPEN circuit breaker states and self-healing transitions in HALF_OPEN states.

Expected output

============================= test session starts ==============================
collected 5 items

tests/test_ai_gateway.py .....                                           [100%]

============================== 5 passed in 0.02s ===============================
[GATEWAY] Registered 3 backend GPU replicas.
[LOR BALANCER] Routed request 1 to gpu_1 (active=1), request 2 to gpu_2 (active=0).
[FAILURE] Simulated 3 consecutive backend failures.
[CIRCUIT BREAKER] Tripped to OPEN state; returned graceful cached fallback.
[SELF HEALING] Transitioned to HALF_OPEN and closed circuit after successful probe.

Validate your work

Run the pytest test suite:

pytest tests/run_tests.sh

Troubleshooting

  1. LOR selection: Ensure select_replica_lor checks r.is_healthy before sorting by active_requests.
  2. Circuit cooldown: Check (current_time - self.last_state_change) >= self.cooldown_sec.

Common mistakes

Practice assignment

Add a Weighted LOR Strategy that weights active streams by the worker’s GPU capacity (e.g. 80GB H100 vs 24GB L4).

Extension challenge

Implement Streaming SSE Proxying with backpressure flow control between client sockets and backend inference streams.

Quiz

Q1. Why is 'Least-Outstanding-Requests' (LOR) load balancing vastly superior to Round-Robin for LLM inference clusters?

  1. Because Round-Robin is written entirely in JavaScript
  2. Round-Robin deletes requests randomly
  3. LOR only works on Linux
  4. LLM latency varies hugely; route to the least loaded
Show answer

Answer: D. LLM latency varies hugely; route to the least loaded

Because LLM request durations vary drastically, Least-Outstanding-Requests prevents replica queue congestion.

Q2. What is the primary role of a 'Startup Probe' (introduced in Kubernetes) for LLM container pods?

  1. To let large models load before liveness kills them
  2. To play startup music
  3. To format the local hard drive
  4. To compile the Linux kernel
Show answer

Answer: A. To let large models load before liveness kills them

Startup probes prevent premature container restarts during lengthy model weight initialization.

Q3. How does a 'Circuit Breaker' protect an AI Gateway when downstream GPU inference workers fail?

  1. By cutting the physical power cord
  2. It trips open and fails fast instead of cascading
  3. By doubling the request rate
  4. By restarting all user laptops
Show answer

Answer: B. It trips open and fails fast instead of cascading

Circuit breakers fail fast to protect struggling downstream services and enable graceful recovery.

Q4. What is the purpose of a Kubernetes `PreStop` lifecycle hook in an LLM deployment?

  1. To delete the container logs
  2. To overclock the GPU
  3. To drain connections before SIGTERM arrives
  4. To restart the whole model from scratch again
Show answer

Answer: C. To drain connections before SIGTERM arrives

PreStop hooks allow active streaming decode requests to finish cleanly before pod termination.

Q5. What three core metrics form the fundamental Triad of LLM Serving Telemetry?

  1. Color depth, monitor resolution, and mouse clicks
  2. Disk free space, memory speed, and audio volume
  3. CPU temperature, fan speed, and case LED status
  4. TTFT, inter-token latency, and error rate
Show answer

Answer: D. TTFT, inter-token latency, and error rate

TTFT, ITL, and queue depth/error rates provide the complete operational picture of LLM serving health.

Glossary

AI Gateway
A high-performance reverse proxy managing authentication, rate limiting, and intelligent load balancing across AI backends.
Least-Outstanding-Requests (LOR)
A load balancing strategy routing traffic to the replica with the fewest active running requests.
Circuit Breaker
A resilience pattern tripping to OPEN state upon repeated backend failures to prevent cascading outages.
Startup Probe
A Kubernetes probe disabling liveness checks until heavy model weight initialization completes.
Readiness Probe
A probe determining whether a replica is ready to receive live user network traffic.
Connection Draining
Allowing active streaming responses to finish before terminating a container during rolling deployments.
Time-to-First-Token (TTFT)
The latency elapsed between receiving a prompt and generating the initial response token.
Inter-Token Latency (ITL)
The average time elapsed between consecutive auto-regressive decode tokens.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.