Deployment, MLOps, and Security › Operating AI in Production › Day 338
Hands-on lab — Day 338: GPU Serving and Inference Infrastructure
- ← Back to the Day 338 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deployment-mlops-and-security/day-338-gpu-serving-and-inference-infrastructure/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/gpu_infrastructure.py Test
bash tests/run_tests.sh File tree
examples/gpu_infrastructure.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/gpu_infrastructure.py tests/run_tests.sh tests/test_gpu_infrastructure.py troubleshooting.md
Lab README
Day 338 Lab: GPU Serving and Inference Infrastructure
Lesson
- Lesson title: GPU Serving and Inference Infrastructure
- Day number: 338 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-338-gpu-serving-and-inference-infrastructure
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-338-gpu-serving-and-inference-infrastructurewhen the site is running.
Purpose
Build a Multi-Model GPU Inference Infrastructure Simulator in Python modeling VRAM pool allocation, dynamic server-side batching, and queue delay timeouts.
Learning objectives
- Manage GPU VRAM pool allocations and prevent OOMs.
- Enqueue incoming requests with arrival timestamps.
- Aggregate dynamic batches based on capacity and delay thresholds.
- Measure average queue latency and batch packing efficiency.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/gpu_infrastructure.py: Starter implementation skeletonexamples/gpu_infrastructure.py: Verified reference implementationtests/test_gpu_infrastructure.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/gpu_infrastructure.py
What the commands do
- Executes dynamic batch aggregation, evaluates queue timeouts, and tests VRAM limits.
Expected output
{'batch_size': 2, 'request_ids': ['r1', 'r2'], 'avg_queue_delay_ms': 1.5, 'timestamp': 100.002}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- VRAM allocation and OOM protection
- Immediate batch dispatch when max_batch_size is met
- Timed batch flush when max_queue_delay_ms expires
- Holding state while requests are below batch size and timeout
- Empty queue handling
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure arrival timestamps are recorded in seconds.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement a priority queue dispatch policy.
Navigation
Day number: 338 of 365
Expected output
FIELDS.md
# Measured Values
- `batches_dispatched`: 6
- `avg_delay_ms`: 3.4
examples-run.txt
{'batch_size': 2, 'request_ids': ['r1', 'r2'], 'avg_queue_delay_ms': 1.5, 'timestamp': 100.002}
measured-values.txt
batches_dispatched=6
avg_delay_ms=3.4
run-output.txt
{'batch_size': 2, 'request_ids': ['r1', 'r2'], 'avg_queue_delay_ms': 1.5, 'timestamp': 100.002}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.gpu_infrastructure.GPUServingInfrastructureSimulator object at 0x108fea650>
req_id = 'r1', prompt = 'p1', arrival_time = 100.0
def enqueue_request(self, req_id: str, prompt: str, arrival_time: float):
> raise NotImplementedError('TASK 2: implement enqueue_request.')
E NotImplementedError: TASK 2: implement enqueue_request.
examples/gpu_infrastructure.py:25: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_gpu_infrastructure.py::test_vram_allocation_and_oom_protection
FAILED tests/test_gpu_infrastructure.py::test_dynamic_batch_full_immediate_dispatch
FAILED tests/test_gpu_infrastructure.py::test_dynamic_batch_queue_delay_timeout_flush
FAILED tests/test_gpu_infrastructure.py::test_empty_queue_returns_none - NotI...
FAILED tests/test_gpu_infrastructure.py::test_partial_batch_draining_when_exceeding_max_batch
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-338-gpu-serving-and-inference-infrastructure
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_gpu_infrastructure.py::test_vram_allocation_and_oom_protection PASSED [ 20%]
tests/test_gpu_infrastructure.py::test_dynamic_batch_full_immediate_dispatch PASSED [ 40%]
tests/test_gpu_infrastructure.py::test_dynamic_batch_queue_delay_timeout_flush PASSED [ 60%]
tests/test_gpu_infrastructure.py::test_empty_queue_returns_none PASSED [ 80%]
tests/test_gpu_infrastructure.py::test_partial_batch_draining_when_exceeding_max_batch PASSED [100%]
============================== 5 passed in 0.01s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-338-gpu-serving-and-inference-infrastructure
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_gpu_infrastructure.py::test_vram_allocation_and_oom_protection PASSED [ 20%]
tests/test_gpu_infrastructure.py::test_dynamic_batch_full_immediate_dispatch PASSED [ 40%]
tests/test_gpu_infrastructure.py::test_dynamic_batch_queue_delay_timeout_flush PASSED [ 60%]
tests/test_gpu_infrastructure.py::test_empty_queue_returns_none PASSED [ 80%]
tests/test_gpu_infrastructure.py::test_partial_batch_draining_when_exceeding_max_batch PASSED [100%]
============================== 5 passed in 0.01s ===============================
Source files
examples/gpu_infrastructure.py (2151 bytes)
import time
from typing import Dict, Any, List, Optional
class GPUServingInfrastructureSimulator:
def __init__(self, max_batch_size: int = 4, max_queue_delay_ms: float = 5.0):
self.max_batch = int(max_batch_size)
self.max_delay_ms = float(max_queue_delay_ms)
self.vram_pool_mb = 16384
self.allocated_models: Dict[str, int] = {}
self.incoming_queue: List[Dict[str, Any]] = []
self.execution_log: List[Dict[str, Any]] = []
def load_model_to_vram(self, model_name: str, required_vram_mb: int) -> bool:
used_vram = sum(self.allocated_models.values())
if (used_vram + required_vram_mb) > self.vram_pool_mb:
return False
self.allocated_models[model_name] = required_vram_mb
return True
def enqueue_request(self, req_id: str, prompt: str, arrival_time: float):
self.incoming_queue.append({
"req_id": req_id,
"prompt": prompt,
"arrival_time": arrival_time
})
def process_dynamic_batch(self, current_time: float) -> Optional[Dict[str, Any]]:
if not self.incoming_queue:
return None
oldest_arrival = self.incoming_queue[0]["arrival_time"]
delay_elapsed_ms = (current_time - oldest_arrival) * 1000.0
if len(self.incoming_queue) >= self.max_batch or delay_elapsed_ms >= self.max_delay_ms:
batch_size = min(len(self.incoming_queue), self.max_batch)
batch = [self.incoming_queue.pop(0) for _ in range(batch_size)]
event = {
"batch_size": len(batch),
"request_ids": [b["req_id"] for b in batch],
"avg_queue_delay_ms": round(sum((current_time - b["arrival_time"]) * 1000.0 for b in batch) / len(batch), 2),
"timestamp": current_time
}
self.execution_log.append(event)
return event
return None
if __name__ == "__main__":
s = GPUServingInfrastructureSimulator(2, 5.0)
s.enqueue_request("r1", "p1", 100.0)
s.enqueue_request("r2", "p2", 100.001)
print(s.process_dynamic_batch(100.002))
metadata.yml (405 bytes)
lesson_id: D338
day: 338
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/gpu_infrastructure.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/gpu_infrastructure.py (1427 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import time
from typing import Dict, Any, List, Optional
class GPUServingInfrastructureSimulator:
def __init__(self, max_batch_size: int=4, max_queue_delay_ms: float=5.0):
self.max_batch = int(max_batch_size)
self.max_delay_ms = float(max_queue_delay_ms)
self.vram_pool_mb = 16384
self.allocated_models: Dict[str, int] = {}
self.incoming_queue: List[Dict[str, Any]] = []
self.execution_log: List[Dict[str, Any]] = []
def load_model_to_vram(self, model_name: str, required_vram_mb: int) -> bool:
raise NotImplementedError('TASK 1: implement load_model_to_vram.')
def enqueue_request(self, req_id: str, prompt: str, arrival_time: float):
raise NotImplementedError('TASK 2: implement enqueue_request.')
def process_dynamic_batch(self, current_time: float) -> Optional[Dict[str, Any]]:
raise NotImplementedError('TASK 3: implement process_dynamic_batch.')
if __name__ == '__main__':
s = GPUServingInfrastructureSimulator(2, 5.0)
s.enqueue_request('r1', 'p1', 100.0)
s.enqueue_request('r2', 'p2', 100.001)
print(s.process_dynamic_batch(100.002))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_gpu_infrastructure.py (2110 bytes)
import pytest
from examples.gpu_infrastructure import GPUServingInfrastructureSimulator
def test_vram_allocation_and_oom_protection():
sim = GPUServingInfrastructureSimulator()
assert sim.load_model_to_vram("embed_model", 8192) is True
assert sim.load_model_to_vram("rerank_model", 8192) is True
# Total 16384 MB full -> next model should fail OOM
assert sim.load_model_to_vram("extra_model", 1024) is False
def test_dynamic_batch_full_immediate_dispatch():
sim = GPUServingInfrastructureSimulator(max_batch_size=3, max_queue_delay_ms=10.0)
sim.enqueue_request("r1", "prompt 1", 100.000)
sim.enqueue_request("r2", "prompt 2", 100.001)
sim.enqueue_request("r3", "prompt 3", 100.002)
# 3 items >= max_batch_size (3) -> dispatches immediately at t=100.003
res = sim.process_dynamic_batch(current_time=100.003)
assert res is not None
assert res["batch_size"] == 3
assert res["request_ids"] == ["r1", "r2", "r3"]
assert len(sim.incoming_queue) == 0
def test_dynamic_batch_queue_delay_timeout_flush():
sim = GPUServingInfrastructureSimulator(max_batch_size=5, max_queue_delay_ms=5.0)
sim.enqueue_request("r1", "prompt 1", 100.000)
# At t=100.002 (2ms < 5ms delay and 1 < 5 batch), should hold
assert sim.process_dynamic_batch(current_time=100.002) is None
# At t=100.006 (6ms > 5ms delay), should flush single-item batch!
res = sim.process_dynamic_batch(current_time=100.006)
assert res is not None
assert res["batch_size"] == 1
assert res["request_ids"] == ["r1"]
def test_empty_queue_returns_none():
sim = GPUServingInfrastructureSimulator()
assert sim.process_dynamic_batch(100.0) is None
def test_partial_batch_draining_when_exceeding_max_batch():
sim = GPUServingInfrastructureSimulator(max_batch_size=2, max_queue_delay_ms=10.0)
sim.enqueue_request("r1", "p1", 100.0)
sim.enqueue_request("r2", "p2", 100.0)
sim.enqueue_request("r3", "p3", 100.0)
res = sim.process_dynamic_batch(100.001)
assert res["batch_size"] == 2
assert len(sim.incoming_queue) == 1