LLMs and Generative AI β€Ί Customizing and Running Models β€Ί Day 279

Day 279: Serving Open Models

Day 279 of 365 β€” Serving Open Models

Master high-throughput production LLM serving architectures: understand PagedAttention virtual memory block allocation, implement continuous/iteration-level batching schedulers, configure tensor parallelism across multi-GPU setups, and expose OpenAI-compatible streaming endpoints.

Course
LLMs and Generative AI
Category
Customizing and Running Models
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-279-serving-open-models

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-279-serving-open-models
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Deploying open-weight language models in production is radically different from deploying standard web applications or classical machine learning classifiers. In classical web services, serving 1,000 concurrent requests simply requires scaling stateless container pods behind a load balancer.

In LLM serving, however, every active generation request maintains a massive, dynamic Key-Value (KV) Cache in precious GPU High-Bandwidth Memory (HBM).

Under legacy serving architectures, serving engines pre-allocated contiguous memory blocks assuming every incoming request might generate up to the maximum context window (e.g. 8,192 tokens). If a user submitted a short prompt and received a 50-token answer, over 95% of the allocated GPU memory was wasted in internal fragmentation. As a result, an $8,000 NVIDIA A100 GPU capable of computing thousands of tokens per second was artificially constrained to serving only 4 to 8 concurrent users before crashing with Out-Of-Memory errors.

In 2023, researchers at UC Berkeley introduced vLLM and the PagedAttention algorithm. By borrowing virtual memory paging concepts developed in 1960s operating systems, PagedAttention fragmented the KV cache into fixed-size virtual blocks, slashing memory waste to under 4% and boosting serving throughput by 200% to 400%.

Understanding high-throughput serving engines (vLLM, TGI, SGLang), continuous batching schedulers, and multi-GPU tensor parallelism is the defining engineering competency required to operate cost-effective private AI infrastructure at enterprise scale.

PagedAttention and Continuous Batching Architecture comparing static batching fragmentation with virtual paged KV blocks

The idea in plain language

Diagram: the four capabilities that distinguish a serving engine from naive inference

Imagine a busy downtown parking garage with 100 parking spaces:

By eliminating reserved waste and allowing cars to enter and exit dynamically on every second, the garage easily parks 120 cars simultaneously without expanding physical space.

Historical background

Diagram: the historical progression of open model serving infrastructure

The evolution of generative LLM serving progressed through three distinct architectural generations:

In the First Generation (2020–2022), teams served models using standard deep learning frameworks like PyTorch and Hugging Face transformers.pipeline wrapped in FastAPI or Flask. Inference executed sequentially: one request entered, generated tokens, and completed before the next request was evaluated. GPU compute utilization hovered at an abysmal 5% to 10%, because matrix multiplication cores sat idle while memory streamed across the bus.

In the Second Generation (2022), researchers at Microsoft and Stanford published Orca (OSDI β€˜22), introducing Iteration-Level (Continuous) Batching. Instead of waiting for an entire batch of requests to finish generating, Orca scheduled inference at the level of individual token iterations: every time the model finished a forward pass, completed sequences were evicted and newly arrived requests from the queue were injected into the active batch. While continuous batching doubled throughput, memory allocation was still constrained by contiguous pre-allocations.

In the Third Generation (2023–Present), Woosuk Kwon et al. at UC Berkeley published β€œEfficient Memory Management for Large Language Model Serving with PagedAttention”, founding the open-source vLLM project. PagedAttention solved memory fragmentation once and for all, introducing virtual page tables for KV caches, prefix caching across shared system prompts, and multi-GPU Tensor Parallelism.

Today, vLLM, TensorRT-LLM (NVIDIA), and SGLang represent the gold standard for production LLM serving, powering major AI cloud providers and enterprise on-premises infrastructure.

What it is β€” and what it is not

Let us define high-throughput serving systems clearly:

What Modern LLM Serving Is:

What Modern LLM Serving Is Not:

+-------------------------------------------------------------------------------+
|                       SERVING ENGINE FEATURE MATRIX                           |
+----------------------+--------------------+-------------------+---------------+
| Feature              | vLLM               | Hugging Face TGI  | Ollama        |
+----------------------+--------------------+-------------------+---------------+
| Primary Focus        | Enterprise Scale   | Enterprise Scale  | Local Dev     |
| PagedAttention       | Native (Original)  | Yes (v2)          | No (Simple)   |
| Continuous Batching  | State-of-the-Art   | State-of-the-Art  | Basic Queue   |
| Tensor Parallelism   | Native Multi-GPU   | Native Multi-GPU  | Single-GPU/CPU|
| Apple Silicon Metal  | No (CUDA/ROCm)     | No (CUDA/ROCm)    | Native Metal  |
| Target Deployment    | Kubernetes GPU Pod | Kubernetes Pod    | Developer Mac |
+----------------------+--------------------+-------------------+---------------+

Why it was created and what problems it solves

Production serving engines solve the primary operational bottlenecks of large-scale LLM deployment:

1. Internal & External Memory Fragmentation (Solved by PagedAttention)

In standard attention, Key and Value vectors for all past tokens must be stored contiguously in memory. Because request lengths are unpredictable, engines reserved the maximum sequence length (e.g. 4,096 tokens). In practice, actual generation lengths varied wildly, leaving 60% to 80% of VRAM allocated but completely unused (internal fragmentation). PagedAttention allocates memory on-demand in tiny 16-token chunks, reducing fragmentation waste to under 4%.

2. The Slowest-Request Head-of-Line Blocking (Solved by Continuous Batching)

In static request batching, if Request A needs 10 tokens and Request B needs 1,000 tokens, Request A finishes in 100ms but cannot return its response or free its memory until Request B finishes 10 seconds later. Continuous batching operates at the token step level, evicting Request A immediately after token 10 and inserting Request C from the waiting queue on the very next forward pass.

Diagram: static request batching compared with continuous batching

3. Redundant System Prompt Processing (Solved by Automatic Prefix Caching)

When thousands of users interact with an enterprise assistant that carries a 2,000-token system prompt, standard engines re-evaluate the full 2,000-token attention prefill on every single turn. Prefix Caching hashes the prompt prefix and points new requests to existing pre-computed KV cache blocks, reducing Time-To-First-Token (TTFT) from 800ms to 20ms and cutting compute load by 90%.

Diagram: how a serving engine detects and reuses shared prompt prefixes

How it works

Let us inspect the internal architectural components of a modern vLLM serving node:

[HTTP Clients] ──> [OpenAI-Compatible API Server (FastAPI / Uvicorn)]
                                β”‚
                                β–Ό
                   [Engine Orchestrator & Scheduler]
                                β”‚  - Continuous Batch Queue
                                β”‚  - Block Table Allocator
                                β–Ό
               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
               β”‚    Worker GPU Execution Pool    β”‚
               β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
               β”‚  β”‚ Transformer Model Weights β”‚  β”‚
               β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
               β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
               β”‚  β”‚ PagedAttention KV Blocks  β”‚  β”‚
               β”‚  β”‚ [Block 0] [Block 1] ...   β”‚  β”‚
               β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
               β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                β”‚
                                β–Ό
               [Server-Sent Events (SSE) Stream]

1. The Two Phases of LLM Inference

Every LLM request consists of two fundamentally distinct computational phases:

Diagram: the prompt processing phase compared with the token generation phase

  1. The Prefill Phase (Compute-Bound):
    • The engine processes the entire input prompt simultaneously.
    • Computes Key and Value attention projections for all prompt tokens in parallel using high-performance matrix multiplication (GEMM).
    • Characterized by high GPU compute utilization (TFLOPS) and low memory bandwidth pressure.
  2. The Decode Phase (Memory-Bandwidth Bound):
    • The engine generates output tokens autoregressively, one token at a time.
    • For each new token, the model reads the entire model weight tensor and all previous KV cache tokens from memory.
    • Characterized by low compute utilization and extreme memory bandwidth saturation.

Continuous batching co-schedules prefill requests with active decode requests, perfectly balancing compute-heavy operations with bandwidth-heavy memory loads on the same GPU cores.

2. PagedAttention Block Allocation Mechanics

PagedAttention organizes KV cache storage analogously to virtual memory pages:

Diagram: contiguous attention cache allocation compared with paged block allocation

Logical Token Indices:  [0 ... 15]   [16 ... 31]   [32 ... 47]
Logical Blocks:          Block 0       Block 1       Block 2
                            β”‚             β”‚             β”‚
                            β–Ό             β–Ό             β–Ό
Physical Memory Cache:   [Phys #4]     [Phys #12]    [Phys #7]  (Non-contiguous in VRAM!)

During attention computation, the custom PagedAttention CUDA kernel fetches keys and values directly from non-contiguous physical memory blocks via the block table without copying or defragmenting memory.

3. Multi-GPU Sharding: Tensor Parallelism (TP)

When a model is too large for a single GPU (e.g. Llama 3 70B in 16-bit requires 140GB VRAM, exceeding a single 80GB A100), the model is sharded across multiple GPUs using Tensor Parallelism:

Diagram: how a model too large for one device is split across several

Because NVLink provides 900 GB/s inter-GPU bandwidth, Tensor Parallelism achieves near-linear scaling across 2, 4, or 8 GPUs within a single server chassis.

Continuous Batching Request Lifecycle showing scheduling queue, PagedAttention worker, and SSE streaming

An everyday analogy

Diagram: seating guests in fixed waves compared with seating them as tables free

Think of a bustling Italian restaurant kitchen during dinner rush:

The kitchen stays at 100% capacity, orders stream out continuously, and customers receive their food in minimal time.

Examples in practice

Let us inspect standard enterprise deployment patterns for vLLM:

1. Launching a Production vLLM Server on Kubernetes

python3 -m vllm.entrypoints.openai.api_server     --model meta-llama/Meta-Llama-3-70B-Instruct     --tensor-parallel-size 4     --max-model-len 8192     --gpu-memory-utilization 0.95     --enable-prefix-caching     --port 8000

2. Consuming Streaming Tokens via Python OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="token-not-required-for-internal-vllm"
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    messages=[
        {"role": "system", "content": "You are an enterprise support bot."},
        {"role": "user", "content": "Explain continuous batching."}
    ],
    stream=True,
    temperature=0.2
)

for chunk in response:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Implications: security, privacy, performance, scalability, and cost

Diagram: the throughput, cost, isolation and complexity implications of a serving engine

1. Security & Multi-Tenant Isolation

In shared multi-tenant serving clusters, ensure that memory page pools are strictly scrubbed or zeroed between tenant allocations to prevent residual KV attention states from leaking across customer sessions.

2. Privacy & Air-Gapped Deployments

Deploying vLLM inside private enterprise VPCs guarantees that zero telemetry, customer prompts, or completion vectors leave corporate boundary firewalls.

3. Latency Metrics: TTFT vs Inter-Token Latency (ITL)

Diagram: time to first token compared with inter-token latency as serving targets

4. Financial Economics

Serving Architecture        Max Concurrency (A100 80GB)   Cost per 1M Tokens
-----------------------------------------------------------------------------
Naive PyTorch FastAPI       4 - 8 concurrent requests     $0.024 / 1k requests
TGI (Text Generation Inf)   30 - 50 concurrent requests   $0.005 / 1k requests
vLLM (PagedAttention + TP)  120 - 200 concurrent requests $0.0012 / 1k requests

Alternatives: free, open source, and commercial

Serving FrameworkDeveloper / OrgKey CapabilitiesBest Use CaseTradeoff
vLLMUC Berkeley / Open SourcePagedAttention, Prefix Caching, LoRA Swapping, TPStandard cloud enterprise servingNVIDIA CUDA / AMD ROCm focused
TensorRT-LLMNVIDIADeep kernel fusion, In-Flight BatchingPeak throughput on NVIDIA hardwareTightly coupled to NVIDIA GPUs
SGLangLMSYS / UC BerkeleyRadixAttention, complex multi-call structured workflowsHigh-frequency agent pipelinesRapidly evolving API surface
TGI (Text Generation Inference)Hugging FaceProduction Docker images, token streaming, watermarkingTurnkey Hugging Face enterprise podsLess modular custom layer support
llama.cpp ServerGeorgi GerganovLightweight C++ server with Apple Metal supportEdge devices and developer MacBooksLower multi-user concurrency
+-----------------------+-----------------------+-----------------------+-----------------------+
| Feature               | Static Batching       | Continuous Batching   | PagedAttention (vLLM) |
+-----------------------+-----------------------+-----------------------+-----------------------+
| Scheduling Granularity| Request-Level         | Iteration (Token)-Level| Iteration-Level       |
| Memory Layout         | Contiguous Fixed      | Contiguous Fixed      | Non-Contiguous Paged  |
| Memory Waste          | 60% - 80% (Severe)    | 40% - 60% (Moderate)  | < 4% (Negligible)     |
| Max Concurrency       | Low (4 - 8)           | Medium (15 - 30)      | Extremely High (100+) |
| Head-of-Line Blocking | Yes (Severe)          | No (Evicts early)     | No (Evicts early)     |
+-----------------------+-----------------------+-----------------------+-----------------------+

When to use it β€” and when not to

Diagram: when a production serving engine is warranted

Choose Production Serving Engines (vLLM / TGI) when:

Do NOT use Production Serving Engines when:

The AI connection

Knowledge check

  1. Why does static batching waste memory? Because it pre-allocates maximum-length contiguous memory blocks for every request regardless of actual prompt or response lengths.
  2. What is the role of a Block Table in PagedAttention? It maps logical token positions in a sequence to non-contiguous physical memory blocks in GPU VRAM, enabling dynamic memory allocation without memory fragmentation.
  3. What is the difference between Tensor Parallelism and Pipeline Parallelism? Tensor Parallelism shards individual layer matrices across GPUs within a server via NVLink All-Reduce. Pipeline Parallelism assigns sequential layers to different GPUs or nodes in a linear chain.

Hands-on exercise

In this lab, you will implement a PagedAttention Block Allocator and Continuous Batching Scheduler Simulator in Python. Your system will:

  1. Manage a pool of fixed-size physical memory blocks (e.g. 16 tokens per block).
  2. Allocate, expand, and free logical block tables dynamically as sequences generate tokens.
  3. Simulate continuous iteration-level batching over variable-length requests.
  4. Calculate memory fragmentation savings and throughput metrics compared to static batching.

Expected output

============================= test session starts ==============================
collected 6 items

tests/test_serving_scheduler.py ......                                   [100%]

============================== 6 passed in 0.04s ===============================
[PAGED ATTENTION ENGINE] Pool Size: 64 Blocks (1,024 Tokens Capacity)
[CONTINUOUS BATCH SIMULATION] Processed 10 Concurrent Requests:
  -> Total Tokens Emitted: 480 tokens
  -> Peak Physical Memory Used: 32 Blocks (50.0% of pool)
  -> Static Batching Memory Requirement: 80 Blocks (125.0% - OOM Crash)
  -> Memory Fragmentation Waste: 2.8% (vs 68.5% in Static)

Validate your work

Run the pytest harness to verify block table allocation, iteration scheduling, and memory accounting:

pytest tests/run_tests.sh

Troubleshooting

  1. Physical block leak on completion: Ensure request eviction calls free_request() to return physical block IDs to the free pool.
  2. Block allocation overflow: Verify that allocate_block() raises an OutOfMemory error when free_blocks is empty.

Common mistakes

Practice assignment

Extend the PagedAttentionSimulator to implement Shared Prefix Caching: when two requests share identical initial prompt tokens, point their initial logical blocks to the same physical block ID with reference counting.

Extension challenge

Implement a Tensor Parallel Matrix Multiplication Simulator in Python that shards a $2048 \times 2048$ linear projection across 4 simulated GPU workers, performs row/column parallel matrix multiplications, and executes an All-Reduce sum reduction.

Quiz

Q1. What primary problem in transformer inference does PagedAttention solve?

  1. It accelerates floating-point gradient descent during pretraining
  2. It compresses PDF documents into Markdown
  3. It automatically translates Python code into C++
  4. It removes the fragmentation from reserving max-length caches
Show answer

Answer: D. It removes the fragmentation from reserving max-length caches

PagedAttention treats the Key-Value (KV) cache like virtual memory pages in an operating system. It stores KV tensors in non-contiguous physical blocks, reducing memory waste from 60-80% down to under 4%.

Q2. How does Continuous (Iteration-Level) Batching differ from legacy Static Batching?

  1. Continuous batching swaps requests in at token level
  2. Static batching processes one request per day; continuous batching runs 24/7
  3. Continuous batching only runs on Apple Silicon
  4. Static batching does not use GPUs
Show answer

Answer: A. Continuous batching swaps requests in at token level

Continuous batching schedules execution at the level of individual token generation iterations. When a sequence finishes emitting tokens, its memory is freed immediately and a waiting request enters the active batch without stalling.

Q3. What is Tensor Parallelism (TP) in multi-GPU LLM serving?

  1. Training multiple different models on separate GPUs
  2. Splitting weight matrices across GPUs over NVLink
  3. Running inference on CPU and GPU concurrently
  4. Compressing models into ZIP files
Show answer

Answer: B. Splitting weight matrices across GPUs over NVLink

Tensor Parallelism shards individual layer matrices across multiple GPUs (e.g. 2, 4, or 8 GPUs in an NVLink mesh), computing attention and MLP projections in parallel with All-Reduce communication.

Q4. Why is Prefix Caching especially valuable in multi-turn conversations and system-prompt heavy applications?

  1. It deletes old messages to save memory
  2. It forces all users to send identical prompts
  3. It reuses cached KV blocks for shared prefixes
  4. It bypasses model tokenization
Show answer

Answer: C. It reuses cached KV blocks for shared prefixes

Prefix caching identifies shared token prefixes across requests (such as static system prompts or previous conversation turns) and directly references their existing KV blocks in memory, slashing Time-To-First-Token.

Q5. Which open-source inference engine pioneered PagedAttention and is considered the standard for high-throughput enterprise LLM serving?

  1. PyTorch TorchScript
  2. SQLite
  3. Nginx
  4. vLLM
Show answer

Answer: D. vLLM

vLLM, developed by UC Berkeley (Kwon et al., 2023), introduced PagedAttention and continuous batching, achieving 2x to 4x higher throughput than Hugging Face TGI and legacy serving engines.

Glossary

PagedAttention
An attention algorithm that stores KV cache tensors in non-contiguous physical memory blocks managed via virtual page tables.
Continuous Batching
An iteration-level scheduling mechanism that dynamically adds new requests and removes completed requests on every forward pass.
KV Cache (Key-Value Cache)
Cached attention key and value projection tensors stored in GPU memory to avoid recomputing past tokens during autoregressive generation.
Tensor Parallelism (TP)
Sharding individual weight matrices across multiple GPUs with intra-node All-Reduce communication.
Pipeline Parallelism (PP)
Assigning sequential transformer layers across multiple GPUs or nodes.
Prefix Caching
Reusing pre-computed KV attention blocks for static prompt prefixes shared across multiple requests.
Time-To-First-Token (TTFT)
The latency required to process the input prompt (prefill phase) and emit the very first output token.
Inter-Token Latency (ITL)
The time duration required to compute and stream each subsequent generated token during the decode phase.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.