Deployment, MLOps, and Security β€Ί Operating AI in Production β€Ί Day 338

Day 338: GPU Serving and Inference Infrastructure

Day 338 of 365 β€” GPU Serving and Inference Infrastructure

Master high-performance GPU serving architectures: architect multi-model Triton Inference Server pipelines, optimize NVLink inter-GPU bandwidth, configure dynamic batching, and profile CUDA streams in Python.

Course
Deployment, MLOps, and Security
Category
Operating AI in Production
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deployment-mlops-and-security/day-338-gpu-serving-and-inference-infrastructure

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deployment-mlops-and-security/day-338-gpu-serving-and-inference-infrastructure
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Deploying deep learning and generative AI models at scale is fundamentally a hardware and systems engineering challenge:

Mastering GPU Serving and Inference Infrastructure (NVIDIA Triton Inference Server, NVLink 900 GB/s inter-GPU fabrics, dynamic server-side batching, and CUDA stream overlap) extracts 100% of theoretical hardware performance, slashing per-token serving costs by 70%.

NVLink Multi-GPU Topology and CUDA Streams showing 4x H100 GPUs connected via 900 GB/s NVSwitch Fabric with Concurrent CUDA Stream Overlap between DMA Memory Copy and Tensor Core Compute

The idea in plain language

Diagram: the three hardware bottlenecks in GPU inference

Think of GPU serving infrastructure like managing a high-speed Formula 1 racing pit crew and multi-lane highway system:

Historical background

Diagram: the progression from synchronous PyTorch serving to NVLink and TensorRT-LLM

GPU inference infrastructure evolved across three major generational leaps:

1. Synchronous PyTorch / Caffe APIs (2015–2019)

Models ran in single Python worker processes. Memory copies blocked CPU execution completely, and running multiple models required multiple separate physical GPU instances.

2. Specialized C++ Inference Engines (Triton & TorchServe) (2020–2023)

NVIDIA open-sourced Triton Inference Server, introducing C++ backend execution, multi-model concurrent serving on single GPUs, dynamic server-side batching, and shared memory ensemble graphs.

Modern multi-GPU systems leverage 900 GB/s NVSwitch fabrics, FP8 matrix acceleration, and unified memory kernels (FlashAttention-3), executing 405B frontier models across 8-way GPU clusters with microsecond inter-GPU AllReduce latencies.

What it is β€” and what it is not

Let us define the core architecture of production GPU inference systems:

What GPU Serving Infrastructure Is:

What GPU Serving Infrastructure Is Not:

Diagram: GPU serving infrastructure compared with the prototyping habits it replaces

+-------------------------------------------------------------------------------+
|                    GPU SERVING RUNTIME COMPARISON MATRIX                      |
+-------------------+-----------------------+-------------------+---------------+
| Runtime Engine    | Primary Strength      | Multi-GPU Support | Dynamic Batch |
+-------------------+-----------------------+-------------------+---------------+
| NVIDIA Triton     | Multi-model ensembles,| Native TensorRT & | Yes (Custom   |
| Inference Server  | C++ high throughput   | vLLM backends     | queue delay)  |
| TorchServe (AWS)  | Native PyTorch handler| Multi-worker      | Yes (Batch    |
|                   | ecosystem             | process pool      | aggregator)   |
| TensorRT-LLM      | Peak ultra-low latency| 8-way NVLink TP   | In-flight     |
| (NVIDIA)          | on Tensor Cores (FP8) | & Pipeline Par    | iteration     |
| vLLM Server       | PagedAttention memory | Tensor Parallel   | Continuous    |
|                   | & prefix caching      | via Ray/NCCL      | iteration-lvl |
+-------------------+-----------------------+-------------------+---------------+

Why it was created and what problems it solves

GPU infrastructure engineering resolves the three primary physical bottlenecks of modern deep learning deployments:

1. Squeezing Maximum Throughput from $30,000+ GPUs

High-end AI GPUs (NVIDIA H100 / A100) are expensive investments. Dynamic batching ensures that GPU compute cores operate at 95%+ utilization rather than sitting idle waiting for single queries.

2. Eliminating Inter-Model Network Hops via Ensembles

Chaining a tokenizer model, embedding generator, and cross-encoder classifier within a Triton Model Ensemble executes all three stages in shared GPU memory in 8ms, eliminating 50ms of network serialization overhead.

3. Zero-Copy Host-to-Device Streaming

Using page-locked (pinned) memory and non-blocking CUDA streams allows the CPU to continuously stream incoming requests directly to GPU VRAM while the GPU is actively computing prior batches.

How it works

Let us inspect the complete Triton Dynamic Batching and Model Ensemble Lifecycle:

[Incoming Concurrent Client Queries (Req A, Req B, Req C, Req D)]
                               β”‚
                               β–Ό
        [Triton Dynamic Batch Scheduler (5ms Window)]
        - Waits up to 5ms or until max_batch_size (64) reached
        - Combines 4 queries into contiguous batch tensor [4, 128]
                               β”‚
                               β–Ό
        [IN-MEMORY ENSEMBLE PIPELINE (Shared VRAM)]
        Stage 1: HuggingFace C++ Tokenizer Backend
                 - Converts 4 raw strings to Input IDs & Attention Mask
                 - Output passed via pointer to Stage 2 (0ms copy)
                               β”‚
        Stage 2: TensorRT-LLM Model Engine
                 - Executes transformer forward pass on Tensor Cores
                 - Produces raw Logit Tensors [4, 128, 32000]
                               β”‚
        Stage 3: Softmax & Top-P Sampling Kernel
                 - Samples top token IDs for each sequence
                               β”‚
                               β–Ό
[Emits formatted JSON responses to all 4 clients simultaneously in 18ms]

Triton Inference Server Dynamic Batching and Ensemble Flow showing Dynamic Batch Scheduler aggregating incoming requests into unified batch tensors and passing them through Tokenizer, Model, and Postprocessing Ensembles

Why that scheme breaks for text generation

Diagram: static request-level batching compared with continuous token-level scheduling

The pipeline above is correct for a model that produces its whole answer in one forward pass β€” a classifier, an embedding model, a vision backbone. Batch four requests, run once, return four answers.

Autoregressive generation is not that. Each request needs one forward pass per token it emits, and different requests emit wildly different numbers of them. Batch four requests where three finish in 20 tokens and one runs to 800, and with static batching the whole batch occupies the GPU until the longest one finishes. Three clients wait for a request that is not theirs, and three quarters of the batch slots compute padding.

Continuous batching β€” also called iteration-level scheduling β€” changes the unit of scheduling from the request to the token. After every single forward pass the scheduler evicts finished sequences and admits waiting ones into the freed slots. A request that arrives mid-generation joins the very next iteration rather than waiting for the current batch to drain. Reported throughput gains over static batching are commonly in the range of ten to twenty times for mixed-length workloads, which is why every serious LLM serving stack does it and why max_batch_size alone is the wrong dial.

The KV cache is the thing you are actually managing

Diagram: how PagedAttention allocates the KV cache in fixed-size blocks

Diagram: why the KV cache rather than model size bounds concurrent requests

Generating token n requires attending to every previous token. Recomputing those keys and values each step would make generation quadratic, so they are kept β€” the KV cache β€” and generation becomes linear.

That cache is large and it grows with every token. Its size is roughly:

2 (K and V) x layers x heads x head_dim x sequence_length x batch_size x bytes_per_value

For a 7B model at 16-bit with a few thousand tokens of context, this reaches gigabytes per sequence. On a 24 GB card the weights are 14 GB, which leaves the KV cache as the binding constraint on how many concurrent requests fit β€” not FLOPs, and not the model size everybody quotes.

Naive allocation makes it worse. Reserving max_sequence_length per request means a client who asks for 20 tokens holds capacity for 4,096, and measured utilisation of that reserved memory has been reported as low as 20–40%.

PagedAttention applies the operating-system idea from Day 003 directly: allocate the KV cache in fixed-size blocks, keep a per-sequence block table, and let a sequence’s blocks be non-contiguous. Internal fragmentation drops to within one block, and blocks can be shared β€” two requests with the same system prompt point at the same physical blocks until they diverge, which is copy-on-write for attention state.

The practical consequence is worth stating plainly: on an LLM serving box, memory management is the performance work. Tensor cores are rarely the limit.

An everyday analogy

Diagram: an elevator carrying one passenger compared with one that waits to fill

Think of GPU serving and dynamic batching like a modern commercial elevator system in an 80-story skyscraper:

Examples in practice

Let us build a complete Python Multi-Model GPU Inference Infrastructure Simulator that models GPU VRAM allocation, pinned memory buffers, dynamic batch window aggregation, and model ensemble execution:

import time
from typing import Dict, Any, List, Optional

class GPUServingInfrastructureSimulator:
    def __init__(self, max_batch_size: int = 4, max_queue_delay_ms: float = 5.0):
        self.max_batch = int(max_batch_size)
        self.max_delay_ms = float(max_queue_delay_ms)
        self.vram_pool_mb = 16384  # 16 GB
        self.allocated_models: Dict[str, int] = {}
        self.incoming_queue: List[Dict[str, Any]] = []
        self.execution_log: List[Dict[str, Any]] = []

    def load_model_to_vram(self, model_name: str, required_vram_mb: int) -> bool:
        used_vram = sum(self.allocated_models.values())
        if (used_vram + required_vram_mb) > self.vram_pool_mb:
            return False  # VRAM OOM
        self.allocated_models[model_name] = required_vram_mb
        return True

    def enqueue_request(self, req_id: str, prompt: str, arrival_time: float):
        self.incoming_queue.append({
            "req_id": req_id,
            "prompt": prompt,
            "arrival_time": arrival_time
        })

    def process_dynamic_batch(self, current_time: float) -> Optional[Dict[str, Any]]:
        if not self.incoming_queue:
            return None

        # Determine if batch is full or oldest request reached max delay
        oldest_arrival = self.incoming_queue[0]["arrival_time"]
        delay_elapsed_ms = (current_time - oldest_arrival) * 1000.0

        if len(self.incoming_queue) >= self.max_batch or delay_elapsed_ms >= self.max_delay_ms:
            # Drain batch
            batch_size = min(len(self.incoming_queue), self.max_batch)
            batch = [self.incoming_queue.pop(0) for _ in range(batch_size)]
            
            event = {
                "batch_size": len(batch),
                "request_ids": [b["req_id"] for b in batch],
                "avg_queue_delay_ms": round(sum((current_time - b["arrival_time"]) * 1000.0 for b in batch) / len(batch), 2),
                "timestamp": current_time
            }
            self.execution_log.append(event)
            return event

        return None  # Waiting for more requests or delay threshold

if __name__ == "__main__":
    sim = GPUServingInfrastructureSimulator(max_batch_size=3, max_queue_delay_ms=10.0)
    sim.load_model_to_vram("embedding_model", 4096)
    sim.load_model_to_vram("reranker_model", 8192)

    sim.enqueue_request("req_1", "query 1", 100.000)
    sim.enqueue_request("req_2", "query 2", 100.002)
    # At t=100.005, batch not full (2 < 3) and delay (5ms < 10ms) -> Holds
    print("Step 1 (Hold):", sim.process_dynamic_batch(100.005))
    
    # 3rd request arrives -> Batch full (3 == 3) -> Dispatches immediately!
    sim.enqueue_request("req_3", "query 3", 100.006)
    print("Step 2 (Dispatched):", sim.process_dynamic_batch(100.007))

Production Architecture Case Study: Triton Model Configuration for TensorRT

Consider an enterprise multi-model serving deployment on NVIDIA Triton running TensorRT-LLM with dynamic batching.

# config.pbtxt: Triton Dynamic Batching Model Configuration
name: "ensemble_text_classification"
platform: "ensemble"
max_batch_size: 64

dynamic_batching {
  max_queue_delay_microseconds: 5000  # 5ms max queue aggregation
  preferred_batch_size: [ 16, 32, 64 ]
}

input [
  {
    name: "RAW_TEXT"
    data_type: TYPE_STRING
    dims: [ -1 ]
  }
]

output [
  {
    name: "CLASSIFICATION_PROBS"
    data_type: TYPE_FP32
    dims: [ 10 ]
  }
]

ensemble_scheduling {
  step [
    {
      model_name: "cxx_tokenizer"
      model_version: 1
      input_map { key: "RAW_TEXT" value: "RAW_TEXT" }
      output_map { key: "INPUT_IDS" value: "TOKEN_IDS" }
    },
    {
      model_name: "tensorrt_encoder"
      model_version: 1
      input_map { key: "INPUT_IDS" value: "TOKEN_IDS" }
      output_map { key: "LOGITS" value: "MODEL_LOGITS" }
    },
    {
      model_name: "softmax_postprocessor"
      model_version: 1
      input_map { key: "LOGITS" value: "MODEL_LOGITS" }
      output_map { key: "PROBABILITIES" value: "CLASSIFICATION_PROBS" }
    }
  ]
}

Detailed Failure Mode Taxonomy: GPU Infrastructure Hazards

Diagram: the four failure modes of GPU serving infrastructure and their controls

+-------------------------------------------------------------------------------+
|                      GPU INFRASTRUCTURE FAILURE TAXONOMY                      |
+-------------------+-----------------------+-----------------------------------+
| Failure Mode      | Root Cause Analysis   | Architectural Mitigation          |
+-------------------+-----------------------+-----------------------------------+
| Host Memory Page  | Unpinned memory forces| Use `cudaMallocHost` / pinned RAM |
| Thrashing         | CPU page-lock copy    | for all host-device DMA transfers |
| Dynamic Batch     | `max_queue_delay` set | Tune delay to 2–5ms; flush batch  |
| Latency Penalty   | too high (e.g. 50ms)  | immediately when batch size full  |
| PCIe Gen Bottleneck| GPU placed in Gen 3   | Verify PCIe Gen 5 link speed via  |
| Speed Drop        | slot instead of Gen 5 | `nvidia-smi -q -d LINK` on boot   |
| CUDA Out of Memory| Dynamic batch exceeds | Configure strict Triton memory    |
| on Long Context   | remaining VRAM        | limiters and max_batch_size bounds|
+-------------------+-----------------------+-----------------------------------+

Deep Architectural Breakdown: Multi-Instance GPU (MIG) and GPU Virtualization

When deploying multiple small models on massive 80GB GPUs:

1. Multi-Instance GPU (MIG) Partitioning

On NVIDIA A100 / H100 GPUs, running small models on a single shared GPU causes resource contention. MIG physically partitions the GPU into up to 7 isolated GPU instances:

Diagram: what hardware partitioning of a GPU provides

+-------------------------------------------------------------------------------+
|                    MIG HARDWARE PARTITIONING ON NVIDIA H100                   |
+-------------------+-------------------+-------------------+-------------------+
| MIG Slice Profile | Compute (SMs)     | Dedicated VRAM    | Ideal Workload    |
+-------------------+-------------------+-------------------+-------------------+
| 1g.10gb           | 14 SMs (~14%)     | 10 GB VRAM        | BERT Tokenizer    |
| 2g.20gb           | 28 SMs (~28%)     | 20 GB VRAM        | Embedding Model   |
| 3g.40gb           | 42 SMs (~42%)     | 40 GB VRAM        | 8B LLM Generation |
| 7g.80gb (Full)    | 132 SMs (100%)    | 80 GB VRAM        | 70B Tensor Parallel|
+-------------------+-------------------+-------------------+-------------------+

GPU Kernel Optimization: FlashAttention-3 and FP8 Tensor Cores

To maximize throughput on NVIDIA Hopper (H100) and Blackwell architectures:

+-------------------------------------------------------------------------------+
|                    CUDA KERNEL EXECUTION TIMELINE ON H100                     |
+-------------------+-------------------+-------------------+-------------------+
| Compute Phase     | Precision Format  | Hardware Unit     | Latency Overhead  |
+-------------------+-------------------+-------------------+-------------------+
| QKV Projection    | FP8 (E4M3)        | Tensor Cores      | 1.2 microseconds  |
| FlashAttention-3  | FP16 / FP8 Hybrid | TMA + Tensor Core | 3.4 microseconds  |
| MLP Feed-Forward  | FP8 GEMM          | Tensor Cores      | 2.8 microseconds  |
| LayerNorm Fusion  | FP32 Accumulate   | CUDA Vector Cores | 0.4 microseconds  |
+-------------------+-------------------+-------------------+-------------------+

Multi-Worker Threading and IPC Shared Memory

When serving multiple models across Python processes on a single node:

Implications: security, privacy, performance, scalability, and cost

1. Hardware MIG Security Sandboxing

In multi-tenant SaaS environments, hardware MIG partitions guarantee complete memory isolation, preventing one tenant from probing physical memory channels belonging to another tenant.

Diagram: hardware-partitioned multi-tenancy compared with software sharing on one GPU

2. Microsecond Server-Side Batch Economics

Dynamic server-side batching boosts serving capacity by 4x to 6x on identical hardware, cutting the number of required GPU server instances by 75%.

Alternatives: free, open source, and commercial

Diagram: Triton Inference Server compared with the vLLM engine

Tool / FrameworkTypeBest Suited ForKey Capabilities
NVIDIA TritonOpen SourceMulti-Model High ThroughputDynamic batching, C++ ensembles, GPU MIG
TorchServeOpen SourcePyTorch-Centric ServingPython handlers, dynamic batching, AWS native
TensorRT-LLMOpen SourcePeak LLM PerformanceFP8 kernels, in-flight batching, NVLink TP
vLLM EngineOpen SourceHigh-Concurrency LLMsPagedAttention, continuous iteration batching
+-----------------------+-----------------------+-----------------------+
| Dimension             | Client-Side Batching  | Server-Side Dyn Batch |
+-----------------------+-----------------------+-----------------------+
| Client Complexity     | High (Must aggregate) | Zero (Simple single req)|
| Latency Overhead      | Uncontrolled          | Strictly Bounded (5ms)|
| Multi-Client Sharing  | Impossible            | Shared across all users|
| GPU Saturation        | Low / Burst-dependent | Consistently High     |
+-----------------------+-----------------------+-----------------------+

When to use it β€” and when not to

Diagram: when GPU serving infrastructure is worth its complexity

Deploy Triton & GPU Serving Infrastructure for:

Do NOT require complex Triton configurations for:

Production Checklist: GPU Serving Infrastructure Hardening

The AI connection

Knowledge check

  1. What is the bandwidth advantage of NVLink 4 over PCIe Gen 5? NVLink provides up to 900 GB/s per GPU (14x faster than PCIe Gen 5’s 64 GB/s), eliminating inter-GPU AllReduce bottlenecks.
  2. How does dynamic batching in Triton improve throughput? By aggregating independent client requests into a single tensor execution batch within a microsecond window (e.g. 5ms).
  3. What is a Triton Model Ensemble? An in-memory pipeline connecting multiple models (tokenizer, neural net, post-processor) with zero inter-process network copying.

Hands-on exercise

In this lab, you will build a Multi-Model GPU Inference Infrastructure Simulator in Python. Your implementation will:

  1. Manage GPU VRAM allocation pools and prevent Out-of-Memory (OOM) over-allocation.
  2. Enqueue incoming client requests with millisecond arrival timestamps.
  3. Aggregate requests dynamically based on max_batch_size or max_queue_delay_ms thresholds.
  4. Dispatch executed batches with recorded queue delays and throughput metrics.

Expected output

============================= test session starts ==============================
collected 5 items

tests/test_gpu_infrastructure.py .....                                   [100%]

============================== 5 passed in 0.02s ===============================
[VRAM POOL] Loaded embedding_model (4GB) and reranker_model (8GB) into 16GB VRAM.
[DYNAMIC BATCH] Enqueued 3 requests; dispatched batch immediately upon max_batch reached.
[DELAY FLUSH] Flushed partial batch (1 item) after 10ms queue delay expired.
[OOM PROTECTION] Rejected oversized model exceeding available VRAM.

Validate your work

Run the pytest test suite:

pytest tests/run_tests.sh

Troubleshooting

  1. VRAM check: Verify (sum(allocated) + new) <= vram_pool_mb.
  2. Delay calculation: Compute (current_time - oldest_arrival) * 1000.0.

Common mistakes

Practice assignment

Add a Priority Queue Policy that dispatches high-priority requests immediately with zero queue delay.

Extension challenge

Implement a CUDA Stream Overlap Model tracking concurrent execution of memory transfers and compute kernels.

Quiz

Q1. What is the critical hardware throughput advantage of NVLink 4 / NVSwitch over standard PCIe Gen 5 buses in multi-GPU clusters?

  1. NVLink is over ten times PCIe bandwidth per GPU
  2. NVLink uses less electricity
  3. PCIe does not support GPUs
  4. NVLink is only ever used for audio processing
Show answer

Answer: A. NVLink is over ten times PCIe bandwidth per GPU

NVLink provides 900 GB/s bandwidth, eliminating inter-GPU communication bottlenecks during tensor parallel inference.

Q2. What is 'Dynamic Batching' in Triton Inference Server?

  1. Deleting requests randomly
  2. It merges arriving requests into one batch tensor
  3. Running the inference on the client's own laptop
  4. Restarting the GPU container every hour
Show answer

Answer: B. It merges arriving requests into one batch tensor

Dynamic batching aggregates independent incoming requests into a single tensor execution cycle within a microsecond window.

Q3. What is a Triton 'Model Ensemble' pipeline?

  1. A musical concert played by computers
  2. A cluster of 1,000 Raspberry Pis
  3. Chaining several models inside the server's memory
  4. An encrypted database backup
Show answer

Answer: C. Chaining several models inside the server's memory

Ensembles chain multiple processing steps directly in server memory, minimizing I/O latency.

Q4. What is the purpose of 'Page-Locked' (Pinned) Host Memory in CUDA data transfers?

  1. To lock the host computer screen
  2. To turn off the cooling fans
  3. To prevent files from being deleted
  4. Pinned host RAM lets DMA transfer without the CPU
Show answer

Answer: D. Pinned host RAM lets DMA transfer without the CPU

Pinned host memory enables asynchronous DMA transfers directly to GPU memory.

Q5. How do CUDA Streams enable concurrency on a single GPU?

  1. Streams let transfer and compute overlap
  2. By splitting the GPU into physical pieces
  3. By running Python scripts in parallel threads
  4. By overclocking the memory clock
Show answer

Answer: A. Streams let transfer and compute overlap

CUDA streams allow concurrent overlap of memory transfers and compute kernel execution.

Glossary

Triton Inference Server
NVIDIA's open-source multi-framework inference engine supporting TensorRT, PyTorch, ONNX, and vLLM.
NVLink / NVSwitch
NVIDIA's high-speed direct inter-GPU interconnect delivering up to 900 GB/s per GPU.
Dynamic Batching
Server-side aggregation of concurrent client requests into unified GPU execution batches.
Model Ensemble
A DAG pipeline connecting multiple models in shared server memory with zero network overhead.
CUDA Streams
Independent GPU execution queues enabling overlapping compute and memory transfer operations.
Pinned Memory (Page-Locked)
Host RAM mapped directly for high-speed asynchronous DMA transfer to GPU VRAM.
TensorRT-LLM
NVIDIA's highly optimized runtime library for ultra-low latency LLM inference on Tensor Cores.
MIG (Multi-Instance GPU)
Hardware partitioning of an A100/H100 GPU into isolated hardware instances with dedicated compute and memory.
Static Batching
Grouping whole requests and running them together until the longest finishes. Correct for single-pass models and wasteful for generation, where the batch is held by its slowest member.
Continuous Batching
Scheduling at the token rather than the request level β€” after every forward pass, finished sequences are evicted and waiting ones admitted into the freed slots.
KV Cache
The stored keys and values for every previous token, which makes generation linear rather than quadratic. Its size, not FLOPs, usually limits how many requests fit on a card.
PagedAttention
Allocating the KV cache in fixed-size blocks with a per-sequence block table, so memory need not be contiguous and identical prefixes can share physical blocks.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.