Deployment, MLOps, and Security › Operating AI in Production › Day 338

Hands-on lab — Day 338: GPU Serving and Inference Infrastructure

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/gpu_infrastructure.py

Test

bash tests/run_tests.sh

File tree

examples/gpu_infrastructure.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/gpu_infrastructure.py
tests/run_tests.sh
tests/test_gpu_infrastructure.py
troubleshooting.md

Lab README

Day 338 Lab: GPU Serving and Inference Infrastructure

Lesson

Purpose

Build a Multi-Model GPU Inference Infrastructure Simulator in Python modeling VRAM pool allocation, dynamic server-side batching, and queue delay timeouts.

Learning objectives

  • Manage GPU VRAM pool allocations and prevent OOMs.
  • Enqueue incoming requests with arrival timestamps.
  • Aggregate dynamic batches based on capacity and delay thresholds.
  • Measure average queue latency and batch packing efficiency.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/gpu_infrastructure.py: Starter implementation skeleton
  • examples/gpu_infrastructure.py: Verified reference implementation
  • tests/test_gpu_infrastructure.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/gpu_infrastructure.py

What the commands do

  • Executes dynamic batch aggregation, evaluates queue timeouts, and tests VRAM limits.

Expected output

{'batch_size': 2, 'request_ids': ['r1', 'r2'], 'avg_queue_delay_ms': 1.5, 'timestamp': 100.002}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • VRAM allocation and OOM protection
  • Immediate batch dispatch when max_batch_size is met
  • Timed batch flush when max_queue_delay_ms expires
  • Holding state while requests are below batch size and timeout
  • Empty queue handling

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure arrival timestamps are recorded in seconds.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement a priority queue dispatch policy.

Day number: 338 of 365

Expected output

FIELDS.md

# Measured Values
- `batches_dispatched`: 6
- `avg_delay_ms`: 3.4

examples-run.txt

{'batch_size': 2, 'request_ids': ['r1', 'r2'], 'avg_queue_delay_ms': 1.5, 'timestamp': 100.002}

measured-values.txt

batches_dispatched=6
avg_delay_ms=3.4

run-output.txt

{'batch_size': 2, 'request_ids': ['r1', 'r2'], 'avg_queue_delay_ms': 1.5, 'timestamp': 100.002}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


self = <examples.gpu_infrastructure.GPUServingInfrastructureSimulator object at 0x108fea650>
req_id = 'r1', prompt = 'p1', arrival_time = 100.0

    def enqueue_request(self, req_id: str, prompt: str, arrival_time: float):
>       raise NotImplementedError('TASK 2: implement enqueue_request.')
E       NotImplementedError: TASK 2: implement enqueue_request.

examples/gpu_infrastructure.py:25: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_gpu_infrastructure.py::test_vram_allocation_and_oom_protection
FAILED tests/test_gpu_infrastructure.py::test_dynamic_batch_full_immediate_dispatch
FAILED tests/test_gpu_infrastructure.py::test_dynamic_batch_queue_delay_timeout_flush
FAILED tests/test_gpu_infrastructure.py::test_empty_queue_returns_none - NotI...
FAILED tests/test_gpu_infrastructure.py::test_partial_batch_draining_when_exceeding_max_batch
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-338-gpu-serving-and-inference-infrastructure
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_gpu_infrastructure.py::test_vram_allocation_and_oom_protection PASSED [ 20%]
tests/test_gpu_infrastructure.py::test_dynamic_batch_full_immediate_dispatch PASSED [ 40%]
tests/test_gpu_infrastructure.py::test_dynamic_batch_queue_delay_timeout_flush PASSED [ 60%]
tests/test_gpu_infrastructure.py::test_empty_queue_returns_none PASSED   [ 80%]
tests/test_gpu_infrastructure.py::test_partial_batch_draining_when_exceeding_max_batch PASSED [100%]

============================== 5 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-338-gpu-serving-and-inference-infrastructure
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_gpu_infrastructure.py::test_vram_allocation_and_oom_protection PASSED [ 20%]
tests/test_gpu_infrastructure.py::test_dynamic_batch_full_immediate_dispatch PASSED [ 40%]
tests/test_gpu_infrastructure.py::test_dynamic_batch_queue_delay_timeout_flush PASSED [ 60%]
tests/test_gpu_infrastructure.py::test_empty_queue_returns_none PASSED   [ 80%]
tests/test_gpu_infrastructure.py::test_partial_batch_draining_when_exceeding_max_batch PASSED [100%]

============================== 5 passed in 0.01s ===============================

Source files

examples/gpu_infrastructure.py (2151 bytes)
import time
from typing import Dict, Any, List, Optional

class GPUServingInfrastructureSimulator:
    def __init__(self, max_batch_size: int = 4, max_queue_delay_ms: float = 5.0):
        self.max_batch = int(max_batch_size)
        self.max_delay_ms = float(max_queue_delay_ms)
        self.vram_pool_mb = 16384
        self.allocated_models: Dict[str, int] = {}
        self.incoming_queue: List[Dict[str, Any]] = []
        self.execution_log: List[Dict[str, Any]] = []

    def load_model_to_vram(self, model_name: str, required_vram_mb: int) -> bool:
        used_vram = sum(self.allocated_models.values())
        if (used_vram + required_vram_mb) > self.vram_pool_mb:
            return False
        self.allocated_models[model_name] = required_vram_mb
        return True

    def enqueue_request(self, req_id: str, prompt: str, arrival_time: float):
        self.incoming_queue.append({
            "req_id": req_id,
            "prompt": prompt,
            "arrival_time": arrival_time
        })

    def process_dynamic_batch(self, current_time: float) -> Optional[Dict[str, Any]]:
        if not self.incoming_queue:
            return None

        oldest_arrival = self.incoming_queue[0]["arrival_time"]
        delay_elapsed_ms = (current_time - oldest_arrival) * 1000.0

        if len(self.incoming_queue) >= self.max_batch or delay_elapsed_ms >= self.max_delay_ms:
            batch_size = min(len(self.incoming_queue), self.max_batch)
            batch = [self.incoming_queue.pop(0) for _ in range(batch_size)]
            
            event = {
                "batch_size": len(batch),
                "request_ids": [b["req_id"] for b in batch],
                "avg_queue_delay_ms": round(sum((current_time - b["arrival_time"]) * 1000.0 for b in batch) / len(batch), 2),
                "timestamp": current_time
            }
            self.execution_log.append(event)
            return event

        return None

if __name__ == "__main__":
    s = GPUServingInfrastructureSimulator(2, 5.0)
    s.enqueue_request("r1", "p1", 100.0)
    s.enqueue_request("r2", "p2", 100.001)
    print(s.process_dynamic_batch(100.002))
metadata.yml (405 bytes)
lesson_id: D338
day: 338
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/gpu_infrastructure.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/gpu_infrastructure.py (1427 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import time
from typing import Dict, Any, List, Optional

class GPUServingInfrastructureSimulator:

    def __init__(self, max_batch_size: int=4, max_queue_delay_ms: float=5.0):
        self.max_batch = int(max_batch_size)
        self.max_delay_ms = float(max_queue_delay_ms)
        self.vram_pool_mb = 16384
        self.allocated_models: Dict[str, int] = {}
        self.incoming_queue: List[Dict[str, Any]] = []
        self.execution_log: List[Dict[str, Any]] = []

    def load_model_to_vram(self, model_name: str, required_vram_mb: int) -> bool:
        raise NotImplementedError('TASK 1: implement load_model_to_vram.')

    def enqueue_request(self, req_id: str, prompt: str, arrival_time: float):
        raise NotImplementedError('TASK 2: implement enqueue_request.')

    def process_dynamic_batch(self, current_time: float) -> Optional[Dict[str, Any]]:
        raise NotImplementedError('TASK 3: implement process_dynamic_batch.')
if __name__ == '__main__':
    s = GPUServingInfrastructureSimulator(2, 5.0)
    s.enqueue_request('r1', 'p1', 100.0)
    s.enqueue_request('r2', 'p2', 100.001)
    print(s.process_dynamic_batch(100.002))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_gpu_infrastructure.py (2110 bytes)
import pytest
from examples.gpu_infrastructure import GPUServingInfrastructureSimulator

def test_vram_allocation_and_oom_protection():
    sim = GPUServingInfrastructureSimulator()
    assert sim.load_model_to_vram("embed_model", 8192) is True
    assert sim.load_model_to_vram("rerank_model", 8192) is True
    # Total 16384 MB full -> next model should fail OOM
    assert sim.load_model_to_vram("extra_model", 1024) is False

def test_dynamic_batch_full_immediate_dispatch():
    sim = GPUServingInfrastructureSimulator(max_batch_size=3, max_queue_delay_ms=10.0)
    sim.enqueue_request("r1", "prompt 1", 100.000)
    sim.enqueue_request("r2", "prompt 2", 100.001)
    sim.enqueue_request("r3", "prompt 3", 100.002)
    
    # 3 items >= max_batch_size (3) -> dispatches immediately at t=100.003
    res = sim.process_dynamic_batch(current_time=100.003)
    assert res is not None
    assert res["batch_size"] == 3
    assert res["request_ids"] == ["r1", "r2", "r3"]
    assert len(sim.incoming_queue) == 0

def test_dynamic_batch_queue_delay_timeout_flush():
    sim = GPUServingInfrastructureSimulator(max_batch_size=5, max_queue_delay_ms=5.0)
    sim.enqueue_request("r1", "prompt 1", 100.000)
    
    # At t=100.002 (2ms < 5ms delay and 1 < 5 batch), should hold
    assert sim.process_dynamic_batch(current_time=100.002) is None
    
    # At t=100.006 (6ms > 5ms delay), should flush single-item batch!
    res = sim.process_dynamic_batch(current_time=100.006)
    assert res is not None
    assert res["batch_size"] == 1
    assert res["request_ids"] == ["r1"]

def test_empty_queue_returns_none():
    sim = GPUServingInfrastructureSimulator()
    assert sim.process_dynamic_batch(100.0) is None

def test_partial_batch_draining_when_exceeding_max_batch():
    sim = GPUServingInfrastructureSimulator(max_batch_size=2, max_queue_delay_ms=10.0)
    sim.enqueue_request("r1", "p1", 100.0)
    sim.enqueue_request("r2", "p2", 100.0)
    sim.enqueue_request("r3", "p3", 100.0)
    
    res = sim.process_dynamic_batch(100.001)
    assert res["batch_size"] == 2
    assert len(sim.incoming_queue) == 1

Troubleshooting

Troubleshooting - Day 338\nEnsure queue delay is computed in milliseconds against arrival timestamp.\n

Security notes

Security - Day 338\nMulti-model serving enforces VRAM memory boundaries and MIG isolation.\n