LLMs and Generative AI βΊ Customizing and Running Models βΊ Day 279
Day 279: Serving Open Models
Master high-throughput production LLM serving architectures: understand PagedAttention virtual memory block allocation, implement continuous/iteration-level batching schedulers, configure tensor parallelism across multi-GPU setups, and expose OpenAI-compatible streaming endpoints.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-279-serving-open-models
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/llms-and-generative-ai/day-279-serving-open-models - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Deconstruct PagedAttention: virtual memory paging for KV attention caches and memory fragmentation elimination.
- Compare Static Request Batching vs Continuous / Iteration-Level Batching in high-concurrency serving.
- Analyze multi-GPU scaling strategies: Tensor Parallelism (TP) vs Pipeline Parallelism (PP).
- Implement a PagedAttention Block Table and Continuous Batching Scheduler simulator in Python.
- Measure GPU token throughput and queue latency under varying request distributions.
Prerequisites
Why this matters
Deploying open-weight language models in production is radically different from deploying standard web applications or classical machine learning classifiers. In classical web services, serving 1,000 concurrent requests simply requires scaling stateless container pods behind a load balancer.
In LLM serving, however, every active generation request maintains a massive, dynamic Key-Value (KV) Cache in precious GPU High-Bandwidth Memory (HBM).
Under legacy serving architectures, serving engines pre-allocated contiguous memory blocks assuming every incoming request might generate up to the maximum context window (e.g. 8,192 tokens). If a user submitted a short prompt and received a 50-token answer, over 95% of the allocated GPU memory was wasted in internal fragmentation. As a result, an $8,000 NVIDIA A100 GPU capable of computing thousands of tokens per second was artificially constrained to serving only 4 to 8 concurrent users before crashing with Out-Of-Memory errors.
In 2023, researchers at UC Berkeley introduced vLLM and the PagedAttention algorithm. By borrowing virtual memory paging concepts developed in 1960s operating systems, PagedAttention fragmented the KV cache into fixed-size virtual blocks, slashing memory waste to under 4% and boosting serving throughput by 200% to 400%.
Understanding high-throughput serving engines (vLLM, TGI, SGLang), continuous batching schedulers, and multi-GPU tensor parallelism is the defining engineering competency required to operate cost-effective private AI infrastructure at enterprise scale.
The idea in plain language
Imagine a busy downtown parking garage with 100 parking spaces:
- Static Batching (Legacy Serving): When a customer enters the garage, the parking manager does not know if they drive a compact Mini Cooper or a 60-foot articulated tour bus. So, the manager reserves 5 consecutive parking spaces for every single car, just in case. If 20 compact cars enter, the garage is declared βFullβ (Out of Memory), even though 80% of the asphalt is completely empty. Furthermore, if 19 cars want to leave after 5 minutes, they are forbidden from leaving until the slowest 20th car finishes parking (Static Batch Lockstep).
- PagedAttention & Continuous Batching (vLLM): The parking manager treats the garage like a smart grid of modular 10-foot tiles (Virtual Memory Blocks).
- When a compact car enters, it is given exactly 2 tiles. As it needs more space, the manager assigns additional non-contiguous tiles anywhere in the garage via a Block Table.
- The moment any car finishes and leaves, its tiles are instantly freed and assigned to the next car waiting in line (Continuous Batching).
By eliminating reserved waste and allowing cars to enter and exit dynamically on every second, the garage easily parks 120 cars simultaneously without expanding physical space.
Historical background
The evolution of generative LLM serving progressed through three distinct architectural generations:
In the First Generation (2020β2022), teams served models using standard deep learning frameworks like PyTorch and Hugging Face transformers.pipeline wrapped in FastAPI or Flask. Inference executed sequentially: one request entered, generated tokens, and completed before the next request was evaluated. GPU compute utilization hovered at an abysmal 5% to 10%, because matrix multiplication cores sat idle while memory streamed across the bus.
In the Second Generation (2022), researchers at Microsoft and Stanford published Orca (OSDI β22), introducing Iteration-Level (Continuous) Batching. Instead of waiting for an entire batch of requests to finish generating, Orca scheduled inference at the level of individual token iterations: every time the model finished a forward pass, completed sequences were evicted and newly arrived requests from the queue were injected into the active batch. While continuous batching doubled throughput, memory allocation was still constrained by contiguous pre-allocations.
In the Third Generation (2023βPresent), Woosuk Kwon et al. at UC Berkeley published βEfficient Memory Management for Large Language Model Serving with PagedAttentionβ, founding the open-source vLLM project. PagedAttention solved memory fragmentation once and for all, introducing virtual page tables for KV caches, prefix caching across shared system prompts, and multi-GPU Tensor Parallelism.
Today, vLLM, TensorRT-LLM (NVIDIA), and SGLang represent the gold standard for production LLM serving, powering major AI cloud providers and enterprise on-premises infrastructure.
What it is β and what it is not
Let us define high-throughput serving systems clearly:
What Modern LLM Serving Is:
- An asynchronous inference server (e.g. vLLM, TGI, SGLang) that coordinates continuous batch scheduling and GPU memory execution.
- A virtual memory manager (PagedAttention) that dynamically allocates fixed-size KV cache blocks (e.g. 16 tokens per block) in GPU VRAM.
- A multi-GPU sharding engine that executes Tensor Parallelism (TP) across high-speed NVLink interconnects.
- An OpenAI-compatible HTTP REST server exposing standard
/v1/chat/completionsand Server-Sent Events (SSE) streaming.
What Modern LLM Serving Is Not:
- A simple Python FastAPI wrapper around
model.generate()(naive loops lack continuous batching and suffer from massive memory fragmentation). - A model training framework (serving engines are optimized exclusively for forward-pass prefill and autoregressive decoding).
- Restricted to single-user execution (designed specifically for high-concurrency multi-tenant production traffic).
+-------------------------------------------------------------------------------+
| SERVING ENGINE FEATURE MATRIX |
+----------------------+--------------------+-------------------+---------------+
| Feature | vLLM | Hugging Face TGI | Ollama |
+----------------------+--------------------+-------------------+---------------+
| Primary Focus | Enterprise Scale | Enterprise Scale | Local Dev |
| PagedAttention | Native (Original) | Yes (v2) | No (Simple) |
| Continuous Batching | State-of-the-Art | State-of-the-Art | Basic Queue |
| Tensor Parallelism | Native Multi-GPU | Native Multi-GPU | Single-GPU/CPU|
| Apple Silicon Metal | No (CUDA/ROCm) | No (CUDA/ROCm) | Native Metal |
| Target Deployment | Kubernetes GPU Pod | Kubernetes Pod | Developer Mac |
+----------------------+--------------------+-------------------+---------------+
Why it was created and what problems it solves
Production serving engines solve the primary operational bottlenecks of large-scale LLM deployment:
1. Internal & External Memory Fragmentation (Solved by PagedAttention)
In standard attention, Key and Value vectors for all past tokens must be stored contiguously in memory. Because request lengths are unpredictable, engines reserved the maximum sequence length (e.g. 4,096 tokens). In practice, actual generation lengths varied wildly, leaving 60% to 80% of VRAM allocated but completely unused (internal fragmentation). PagedAttention allocates memory on-demand in tiny 16-token chunks, reducing fragmentation waste to under 4%.
2. The Slowest-Request Head-of-Line Blocking (Solved by Continuous Batching)
In static request batching, if Request A needs 10 tokens and Request B needs 1,000 tokens, Request A finishes in 100ms but cannot return its response or free its memory until Request B finishes 10 seconds later. Continuous batching operates at the token step level, evicting Request A immediately after token 10 and inserting Request C from the waiting queue on the very next forward pass.
3. Redundant System Prompt Processing (Solved by Automatic Prefix Caching)
When thousands of users interact with an enterprise assistant that carries a 2,000-token system prompt, standard engines re-evaluate the full 2,000-token attention prefill on every single turn. Prefix Caching hashes the prompt prefix and points new requests to existing pre-computed KV cache blocks, reducing Time-To-First-Token (TTFT) from 800ms to 20ms and cutting compute load by 90%.
How it works
Let us inspect the internal architectural components of a modern vLLM serving node:
[HTTP Clients] ββ> [OpenAI-Compatible API Server (FastAPI / Uvicorn)]
β
βΌ
[Engine Orchestrator & Scheduler]
β - Continuous Batch Queue
β - Block Table Allocator
βΌ
βββββββββββββββββββββββββββββββββββ
β Worker GPU Execution Pool β
β βββββββββββββββββββββββββββββ β
β β Transformer Model Weights β β
β βββββββββββββββββββββββββββββ β
β βββββββββββββββββββββββββββββ β
β β PagedAttention KV Blocks β β
β β [Block 0] [Block 1] ... β β
β βββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββ
β
βΌ
[Server-Sent Events (SSE) Stream]
1. The Two Phases of LLM Inference
Every LLM request consists of two fundamentally distinct computational phases:
- The Prefill Phase (Compute-Bound):
- The engine processes the entire input prompt simultaneously.
- Computes Key and Value attention projections for all prompt tokens in parallel using high-performance matrix multiplication (GEMM).
- Characterized by high GPU compute utilization (TFLOPS) and low memory bandwidth pressure.
- The Decode Phase (Memory-Bandwidth Bound):
- The engine generates output tokens autoregressively, one token at a time.
- For each new token, the model reads the entire model weight tensor and all previous KV cache tokens from memory.
- Characterized by low compute utilization and extreme memory bandwidth saturation.
Continuous batching co-schedules prefill requests with active decode requests, perfectly balancing compute-heavy operations with bandwidth-heavy memory loads on the same GPU cores.
2. PagedAttention Block Allocation Mechanics
PagedAttention organizes KV cache storage analogously to virtual memory pages:
- Physical KV memory is partitioned into a fixed array of Physical Blocks (e.g. each block holds Keys and Values for 16 tokens).
- Each active request maintains a Logical Block Table that maps logical token positions to arbitrary physical block IDs.
- When a sequence generates its 17th token, the scheduler allocates a new physical block from the free list and updates the requestβs block table.
Logical Token Indices: [0 ... 15] [16 ... 31] [32 ... 47]
Logical Blocks: Block 0 Block 1 Block 2
β β β
βΌ βΌ βΌ
Physical Memory Cache: [Phys #4] [Phys #12] [Phys #7] (Non-contiguous in VRAM!)
During attention computation, the custom PagedAttention CUDA kernel fetches keys and values directly from non-contiguous physical memory blocks via the block table without copying or defragmenting memory.
3. Multi-GPU Sharding: Tensor Parallelism (TP)
When a model is too large for a single GPU (e.g. Llama 3 70B in 16-bit requires 140GB VRAM, exceeding a single 80GB A100), the model is sharded across multiple GPUs using Tensor Parallelism:
- Attention Projections ($W_q, W_k, W_v$): Column-parallel sharded across GPUs. Each GPU computes its local attention heads independently.
- Attention Output ($W_o$): Row-parallel sharded. An All-Reduce communication collective sums the outputs across GPUs over the high-speed NVLink bus.
- MLP Layers: Gate/Up projections are column-sharded; Down projections are row-sharded with an All-Reduce sum.
Because NVLink provides 900 GB/s inter-GPU bandwidth, Tensor Parallelism achieves near-linear scaling across 2, 4, or 8 GPUs within a single server chassis.
An everyday analogy
Think of a bustling Italian restaurant kitchen during dinner rush:
- Static Batching: The chef prepares 8 orders of lasagna at once. If one customer ordered a simple salad that takes 2 minutes, the salad sits under a heat lamp for 45 minutes until all 8 lasagnas are baked and plated. No new orders can enter the kitchen until all 8 plates leave together.
- Continuous Batching & PagedAttention: The head chef treats cooking stations as modular, dynamically scheduled steps.
- The moment a salad is plated, it is immediately served to the customer.
- The empty prep station is instantly assigned to chop garlic for an incoming pasta order.
- Sauce pans and baking dishes (KV memory blocks) are assigned modularly from a shared pantry and washed the second they are finished.
The kitchen stays at 100% capacity, orders stream out continuously, and customers receive their food in minimal time.
Examples in practice
Let us inspect standard enterprise deployment patterns for vLLM:
1. Launching a Production vLLM Server on Kubernetes
python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-70B-Instruct --tensor-parallel-size 4 --max-model-len 8192 --gpu-memory-utilization 0.95 --enable-prefix-caching --port 8000
2. Consuming Streaming Tokens via Python OpenAI SDK
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8000/v1",
api_key="token-not-required-for-internal-vllm"
)
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3-70B-Instruct",
messages=[
{"role": "system", "content": "You are an enterprise support bot."},
{"role": "user", "content": "Explain continuous batching."}
],
stream=True,
temperature=0.2
)
for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
Implications: security, privacy, performance, scalability, and cost
1. Security & Multi-Tenant Isolation
In shared multi-tenant serving clusters, ensure that memory page pools are strictly scrubbed or zeroed between tenant allocations to prevent residual KV attention states from leaking across customer sessions.
2. Privacy & Air-Gapped Deployments
Deploying vLLM inside private enterprise VPCs guarantees that zero telemetry, customer prompts, or completion vectors leave corporate boundary firewalls.
3. Latency Metrics: TTFT vs Inter-Token Latency (ITL)
- Time-To-First-Token (TTFT): Governed by prompt prefill length and queue wait time. Optimizing prefix caching reduces TTFT by 90%.
- Inter-Token Latency (ITL): The time required per output token (e.g. 15ms per token = 66 tokens/sec). Governed by model size, quantization level, and memory bandwidth.
4. Financial Economics
Serving Architecture Max Concurrency (A100 80GB) Cost per 1M Tokens
-----------------------------------------------------------------------------
Naive PyTorch FastAPI 4 - 8 concurrent requests $0.024 / 1k requests
TGI (Text Generation Inf) 30 - 50 concurrent requests $0.005 / 1k requests
vLLM (PagedAttention + TP) 120 - 200 concurrent requests $0.0012 / 1k requests
Alternatives: free, open source, and commercial
| Serving Framework | Developer / Org | Key Capabilities | Best Use Case | Tradeoff |
|---|---|---|---|---|
| vLLM | UC Berkeley / Open Source | PagedAttention, Prefix Caching, LoRA Swapping, TP | Standard cloud enterprise serving | NVIDIA CUDA / AMD ROCm focused |
| TensorRT-LLM | NVIDIA | Deep kernel fusion, In-Flight Batching | Peak throughput on NVIDIA hardware | Tightly coupled to NVIDIA GPUs |
| SGLang | LMSYS / UC Berkeley | RadixAttention, complex multi-call structured workflows | High-frequency agent pipelines | Rapidly evolving API surface |
| TGI (Text Generation Inference) | Hugging Face | Production Docker images, token streaming, watermarking | Turnkey Hugging Face enterprise pods | Less modular custom layer support |
| llama.cpp Server | Georgi Gerganov | Lightweight C++ server with Apple Metal support | Edge devices and developer MacBooks | Lower multi-user concurrency |
Comparison with related concepts
+-----------------------+-----------------------+-----------------------+-----------------------+
| Feature | Static Batching | Continuous Batching | PagedAttention (vLLM) |
+-----------------------+-----------------------+-----------------------+-----------------------+
| Scheduling Granularity| Request-Level | Iteration (Token)-Level| Iteration-Level |
| Memory Layout | Contiguous Fixed | Contiguous Fixed | Non-Contiguous Paged |
| Memory Waste | 60% - 80% (Severe) | 40% - 60% (Moderate) | < 4% (Negligible) |
| Max Concurrency | Low (4 - 8) | Medium (15 - 30) | Extremely High (100+) |
| Head-of-Line Blocking | Yes (Severe) | No (Evicts early) | No (Evicts early) |
+-----------------------+-----------------------+-----------------------+-----------------------+
When to use it β and when not to
Choose Production Serving Engines (vLLM / TGI) when:
- You are hosting models in the cloud or on private server hardware serving multiple concurrent users or background agent workers.
- You need high throughput (hundreds of queries per second) and low operational cost per token.
- You need dynamic multi-LoRA adapter serving from a shared base model checkpoint.
- You want a drop-in OpenAI-compatible API gateway.
Do NOT use Production Serving Engines when:
- You are developing single-user desktop applications running locally on Apple Silicon (use Ollama / llama.cpp).
- You are executing one-off offline batch evaluation scripts over a small CSV file.
- You have minimal hardware (e.g. single 4GB GPU without CUDA support).
The AI connection
- Serving efficiency, not model quality, decides whether self-hosting is affordable β the gap between naive and optimised serving is several-fold.
- Paged attention memory is operating-system design applied to inference, and it is the clearest case of an old idea transferring cleanly.
- Continuous batching removes head-of-line blocking, which is what turns a demonstration server into a production one.
- Prefix caching makes long system prompts affordable, which changes what a prompt can contain rather than merely what it costs.
- The two latency numbers are optimised by different things, so a serving target has to name which one it means.
Knowledge check
- Why does static batching waste memory? Because it pre-allocates maximum-length contiguous memory blocks for every request regardless of actual prompt or response lengths.
- What is the role of a Block Table in PagedAttention? It maps logical token positions in a sequence to non-contiguous physical memory blocks in GPU VRAM, enabling dynamic memory allocation without memory fragmentation.
- What is the difference between Tensor Parallelism and Pipeline Parallelism? Tensor Parallelism shards individual layer matrices across GPUs within a server via NVLink All-Reduce. Pipeline Parallelism assigns sequential layers to different GPUs or nodes in a linear chain.
Hands-on exercise
In this lab, you will implement a PagedAttention Block Allocator and Continuous Batching Scheduler Simulator in Python. Your system will:
- Manage a pool of fixed-size physical memory blocks (e.g. 16 tokens per block).
- Allocate, expand, and free logical block tables dynamically as sequences generate tokens.
- Simulate continuous iteration-level batching over variable-length requests.
- Calculate memory fragmentation savings and throughput metrics compared to static batching.
Expected output
============================= test session starts ==============================
collected 6 items
tests/test_serving_scheduler.py ...... [100%]
============================== 6 passed in 0.04s ===============================
[PAGED ATTENTION ENGINE] Pool Size: 64 Blocks (1,024 Tokens Capacity)
[CONTINUOUS BATCH SIMULATION] Processed 10 Concurrent Requests:
-> Total Tokens Emitted: 480 tokens
-> Peak Physical Memory Used: 32 Blocks (50.0% of pool)
-> Static Batching Memory Requirement: 80 Blocks (125.0% - OOM Crash)
-> Memory Fragmentation Waste: 2.8% (vs 68.5% in Static)
Validate your work
Run the pytest harness to verify block table allocation, iteration scheduling, and memory accounting:
pytest tests/run_tests.sh
Troubleshooting
- Physical block leak on completion: Ensure request eviction calls
free_request()to return physical block IDs to the free pool. - Block allocation overflow: Verify that
allocate_block()raises an OutOfMemory error whenfree_blocksis empty.
Common mistakes
- Failing to allocate a new physical block when
token_count % block_size == 1. - Forgetting to deallocate physical blocks when a request completes its final token.
Practice assignment
Extend the PagedAttentionSimulator to implement Shared Prefix Caching: when two requests share identical initial prompt tokens, point their initial logical blocks to the same physical block ID with reference counting.
Extension challenge
Implement a Tensor Parallel Matrix Multiplication Simulator in Python that shards a $2048 \times 2048$ linear projection across 4 simulated GPU workers, performs row/column parallel matrix multiplications, and executes an All-Reduce sum reduction.
Quiz
Q1. What primary problem in transformer inference does PagedAttention solve?
- It accelerates floating-point gradient descent during pretraining
- It compresses PDF documents into Markdown
- It automatically translates Python code into C++
- It removes the fragmentation from reserving max-length caches
Show answer
Answer: D. It removes the fragmentation from reserving max-length caches
PagedAttention treats the Key-Value (KV) cache like virtual memory pages in an operating system. It stores KV tensors in non-contiguous physical blocks, reducing memory waste from 60-80% down to under 4%.
Q2. How does Continuous (Iteration-Level) Batching differ from legacy Static Batching?
- Continuous batching swaps requests in at token level
- Static batching processes one request per day; continuous batching runs 24/7
- Continuous batching only runs on Apple Silicon
- Static batching does not use GPUs
Show answer
Answer: A. Continuous batching swaps requests in at token level
Continuous batching schedules execution at the level of individual token generation iterations. When a sequence finishes emitting tokens, its memory is freed immediately and a waiting request enters the active batch without stalling.
Q3. What is Tensor Parallelism (TP) in multi-GPU LLM serving?
- Training multiple different models on separate GPUs
- Splitting weight matrices across GPUs over NVLink
- Running inference on CPU and GPU concurrently
- Compressing models into ZIP files
Show answer
Answer: B. Splitting weight matrices across GPUs over NVLink
Tensor Parallelism shards individual layer matrices across multiple GPUs (e.g. 2, 4, or 8 GPUs in an NVLink mesh), computing attention and MLP projections in parallel with All-Reduce communication.
Q4. Why is Prefix Caching especially valuable in multi-turn conversations and system-prompt heavy applications?
- It deletes old messages to save memory
- It forces all users to send identical prompts
- It reuses cached KV blocks for shared prefixes
- It bypasses model tokenization
Show answer
Answer: C. It reuses cached KV blocks for shared prefixes
Prefix caching identifies shared token prefixes across requests (such as static system prompts or previous conversation turns) and directly references their existing KV blocks in memory, slashing Time-To-First-Token.
Q5. Which open-source inference engine pioneered PagedAttention and is considered the standard for high-throughput enterprise LLM serving?
- PyTorch TorchScript
- SQLite
- Nginx
- vLLM
Show answer
Answer: D. vLLM
vLLM, developed by UC Berkeley (Kwon et al., 2023), introduced PagedAttention and continuous batching, achieving 2x to 4x higher throughput than Hugging Face TGI and legacy serving engines.
Glossary
- PagedAttention
- An attention algorithm that stores KV cache tensors in non-contiguous physical memory blocks managed via virtual page tables.
- Continuous Batching
- An iteration-level scheduling mechanism that dynamically adds new requests and removes completed requests on every forward pass.
- KV Cache (Key-Value Cache)
- Cached attention key and value projection tensors stored in GPU memory to avoid recomputing past tokens during autoregressive generation.
- Tensor Parallelism (TP)
- Sharding individual weight matrices across multiple GPUs with intra-node All-Reduce communication.
- Pipeline Parallelism (PP)
- Assigning sequential transformer layers across multiple GPUs or nodes.
- Prefix Caching
- Reusing pre-computed KV attention blocks for static prompt prefixes shared across multiple requests.
- Time-To-First-Token (TTFT)
- The latency required to process the input prompt (prefill phase) and emit the very first output token.
- Inter-Token Latency (ITL)
- The time duration required to compute and stream each subsequent generated token during the decode phase.
Sources and further reading
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al. / vLLM) β SOSP / arXiv (accessed 2026-08-30)
- vLLM Documentation and Architecture Overview β vLLM Team (accessed 2026-08-30)
- Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al.) β USENIX OSDI (accessed 2026-08-30)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.