AI Engineering: Agents and Applications βΊ Full-Stack AI Applications βΊ Day 320
Day 320: Latency and Caching
Master multi-tier latency optimization and caching strategies for LLM applications: implement exact prompt hashing, vector semantic caching, provider prompt caching, and speculative decoding acceleration.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/ai-engineering/day-320-latency-and-caching
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/ai-engineering/day-320-latency-and-caching - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Deconstruct end-to-end latency in generative AI (Network RTT, Prefill/TTFT, Inter-Token Decode Latency).
- Implement Multi-Tier Caching: Tier 1 Exact Match (Redis SHA-256), Tier 2 Semantic Similarity Cache (Vector distance threshold), and Tier 3 Provider Prompt Caching.
- Configure Anthropic and OpenAI Prompt Caching prefixes to achieve 90% input token cost reductions and 50% TTFT latency improvements.
- Build a production-grade Semantic & Exact LLM Cache Engine in Python with cosine similarity thresholds and TTL eviction.
- Quantify cache hit trade-offs: cost reduction, latency savings, and semantic drift risks.
Prerequisites
Why this matters
In interactive software products, latency is the single greatest determinant of user engagement and retention. While traditional web APIs return responses in 30ms, large language model generation requires:
- 300ms to 2,000ms just to process the input prompt (Time to First Token / Prefill Latency).
- 10ms to 30ms per output token (Inter-Token Decode Latency).
- A 500-word response takes 5 to 10 seconds of continuous GPU computation.
Deploying a Multi-Tier Latency & Caching Architecture (Exact Redis Hashing, Semantic Vector Caching, and Provider Prompt Caching) slashes p95 latency by up to 80% and reduces API billing expenses by over 60%.
The idea in plain language
Think of multi-tier LLM caching like a professional chef preparing dishes during dinner rush:
- Tier 1: Ready-Made Dishes on the Pass (Exact Hash Cache): If a customer orders the daily soup, the chef ladles a hot bowl in 2 seconds without turning on the stove (1ms Redis Cache Hit, $0.00 Cost).
- Tier 2: Assembled Cold Prep (Semantic Cache): If a customer asks for βa green saladβ vs βhouse tossed greensβ, the chef recognizes they are the same dish and serves the prepped bowl in 10 seconds (15ms Vector Cache Hit, $0.00 Cost).
- Tier 3: Pre-Simmered Master Stock (Provider Prompt Cache): If a customer orders a complex risotto, the chef scoops from a giant simmering pot of 24-hour stock, cooking the dish in 3 minutes instead of starting from raw bones (150ms TTFT, 90% Cost Discount).
- Tier 4: Cooking from Scratch (Full Model Inference): A completely novel custom order requires firing all burners from scratch (800ms+ Full GPU Inference).
Historical background
Application caching strategies evolved through three key generations:
1. HTTP Reverse Proxy Caching (2000β2020)
Varnish, Squid, and CDNs cached identical HTTP GET URLs with Cache-Control: max-age=3600. Responses were 100% deterministic.
2. The LLM Non-Determinism Dilemma (2022β2023)
Because LLMs generate text stochastically and take dynamic prompts, traditional HTTP URL caching failed completely. Every single user query incurred full model execution costs.
3. Multi-Tier Semantic & KV Caching (2024βPresent)
Modern systems combine exact prompt hashing in Redis, embedding vector semantic caching (GPTCache), and GPU Key-Value (KV) activation caching at model providers (Anthropic Prompt Caching, vLLM PagedAttention).
What it is β and what it is not
Let us define the core components of modern generative AI caching:
What Multi-Tier AI Caching Is:
- Tier 1: Sub-millisecond exact string matching via SHA-256 hashes in Redis.
- Tier 2: Embedding vector similarity search matching paraphrased queries above a strict threshold (
similarity >= 0.95). - Tier 3: GPU KV cache reuse across static system prompt prefixes at cloud model providers.
What Multi-Tier AI Caching Is Not:
- Setting loose similarity thresholds (e.g. 0.70) that return inaccurate answers to distinct questions.
- Caching private, user-specific data across organizational tenant boundaries.
+-------------------------------------------------------------------------------+
| MULTI-TIER CACHE SPECIFICATIONS |
+-------------------+-------------------+-------------------+-------------------+
| Tier Name | Lookup Latency | Hit Criteria | Cost Reduction |
+-------------------+-------------------+-------------------+-------------------+
| 1. Exact Hash | under 1ms | Exact SHA-256 | 100% ($0.00 cost) |
| 2. Semantic Cache | 10 - 20ms | Cosine Sim >= 0.95| 100% ($0.00 cost) |
| 3. Provider KV | 100 - 200ms TTFT | Static Prefix KV | 90% Input Discount|
| 4. Full GPU Infer | 800 - 3,000ms | None (Cache Miss) | 0% (Full Price) |
+-------------------+-------------------+-------------------+-------------------+
Why it was created and what problems it solves
Multi-tier caching resolves the three primary performance and cost bottlenecks of generative AI products:
1. Slashing Time to First Token (TTFT)
Serving common queries from exact or semantic cache drops TTFT from 1,500ms down to under 15ms, creating an instantly responsive application.
2. Drastic Cloud GPU Cost Reductions
In customer support and knowledge retrieval applications, 30% to 50% of user questions are paraphrased variations of frequent inquiries. Caching these queries saves thousands of dollars monthly in API expenses.
3. Mitigating Upstream Rate Limit Throttle
Serving cached responses requires zero downstream provider API calls, protecting your application from hitting provider Requests-Per-Minute (RPM) and Tokens-Per-Minute (TPM) rate limits during traffic spikes.
How it works
Let us inspect the complete GPU Execution Anatomy: Prefill vs Autoregressive Decode:
User Submits Prompt: "Explain photosynthesis in 200 words" (500 tokens context)
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PHASE 1: PREFILL PHASE (Parallel) β
β - GPU computes self-attention across all 500 prompt tokens in parallel β
β - Compute-bound on Tensor Cores β
β - Generates Key-Value (KV) cache for prompt context β
β - Duration: 150ms -> Produces First Token: "Photosynthesis" β
βββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββ
β (First token streamed to client)
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PHASE 2: AUTOREGRESSIVE DECODE PHASE (Sequential) β
β - GPU generates Token 2: " is" (attending to KV cache) β
β - GPU generates Token 3: " the" β
β - GPU generates Token 4: " process..." β
β - Memory-bandwidth bound (15ms per token) β
β - Duration: 200 tokens * 15ms = 3,000ms total streaming time β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Speculative decoding: the other way to attack the decode phase
Caching removes work you have already done. It cannot help with the 3,000ms of streaming above, because those 200 tokens have never been generated before.
The opening for attacking that phase comes from why decode is slow. It is not compute-bound β it is memory-bandwidth bound. Each token requires streaming the entire weight matrix from HBM into the compute units, and having paid that cost, the GPU is barely occupied. A forward pass that processes one token and a forward pass that processes eight cost almost the same wall-clock time, because the bottleneck is moving the weights rather than doing the arithmetic.
That asymmetry is exploitable. Speculative decoding runs a small, cheap draft model to propose the next k tokens, then has the large target model verify all k in a single forward pass β verification is parallel, exactly like prefill. Every proposed token the target model agrees with is kept; at the first disagreement, the targetβs own token is used and the rest are discarded.
The output is mathematically identical to what the target model would have produced alone. This is the property that matters and the one people disbelieve: with the standard rejection-sampling scheme the accepted sequence is drawn from exactly the target modelβs distribution. It is a latency optimisation, not a quality trade.
What decides the speedup is the acceptance rate β how often the draft model guesses what the target would have said. Easy, predictable spans (boilerplate, closing brackets, common phrasing) accept readily; genuinely uncertain spans do not. Reported end-to-end gains are typically in the range of two to three times for a well-matched draft model, and the cost is real: a second model in memory, and wasted draft compute on every rejection.
It is worth knowing when it does not help. At high batch sizes the GPU is already saturated with useful work from other requests, so the spare capacity speculative decoding exploits has been spent β the technique pays best exactly when serving few concurrent users, which is the opposite of the regime where continuous batching pays.
An everyday analogy
Think of prompt caching like a mathematician answering math questions with a whiteboard:
- Without Prompt Caching: Every time you ask a question, the mathematician erases the entire board, re-derives 50 pages of calculus proofs from scratch, and then solves your problem (Recomputing the entire prefill context).
- With Prompt Caching: The mathematician leaves the 50 pages of calculus proofs permanently drawn on the left side of the board, glances at them instantly, and writes only the final solution on the right (Instant KV Activation Reuse).
Examples in practice
Let us build a complete Python Exact & Semantic Multi-Tier LLM Cache Engine supporting SHA-256 hashing, mock vector embedding similarity evaluation, and time-to-live (TTL) expiration:
import hashlib
import time
from typing import Dict, Any, List, Optional, Tuple
class MultiTierLLMCache:
def __init__(self, semantic_similarity_threshold: float = 0.90, default_ttl_seconds: float = 300.0):
self.exact_cache: Dict[str, Dict[str, Any]] = {}
self.semantic_cache: List[Dict[str, Any]] = []
self.similarity_threshold = float(semantic_similarity_threshold)
self.ttl = float(default_ttl_seconds)
@staticmethod
def _compute_hash(text: str) -> str:
return hashlib.sha256(text.strip().lower().encode()).hexdigest()
@staticmethod
def _mock_embedding(text: str) -> List[float]:
# Simple deterministic character-frequency mock embedding normalized
words = text.lower().split()
length = len(words)
return [float(len(text)), float(length), float(sum(ord(c) for c in text[:5]))]
@staticmethod
def _cosine_similarity(vec1: List[float], vec2: List[float]) -> float:
dot = sum(a * b for a, b in zip(vec1, vec2))
norm1 = sum(a * a for a in vec1) ** 0.5
norm2 = sum(b * b for b in vec2) ** 0.5
if norm1 == 0 or norm2 == 0:
return 0.0
return round(dot / (norm1 * norm2), 4)
def get(self, query: str) -> Tuple[Optional[str], str]:
now = time.time()
# 1. Tier 1: Exact Hash Lookup
q_hash = self._compute_hash(query)
if q_hash in self.exact_cache:
entry = self.exact_cache[q_hash]
if now - entry["timestamp"] <= self.ttl:
return entry["response"], "TIER_1_EXACT_HIT"
# 2. Tier 2: Semantic Vector Lookup
q_vec = self._mock_embedding(query)
for entry in self.semantic_cache:
if now - entry["timestamp"] <= self.ttl:
sim = self._cosine_similarity(q_vec, entry["embedding"])
if sim >= self.similarity_threshold:
return entry["response"], f"TIER_2_SEMANTIC_HIT (sim={sim})"
return None, "CACHE_MISS"
def put(self, query: str, response: str):
now = time.time()
q_hash = self._compute_hash(query)
q_vec = self._mock_embedding(query)
# Store in Tier 1
self.exact_cache[q_hash] = {
"response": response,
"timestamp": now
}
# Store in Tier 2
self.semantic_cache.append({
"query": query,
"embedding": q_vec,
"response": response,
"timestamp": now
})
if __name__ == "__main__":
cache = MultiTierLLMCache(semantic_similarity_threshold=0.95)
cache.put("What is machine learning?", "ML is a subset of AI.")
# Exact Hit
resp, tier = cache.get("what is machine learning?")
print("Lookup 1:", resp, "|", tier)
# Cache Miss
resp2, tier2 = cache.get("How do rockets work?")
print("Lookup 2:", resp2, "|", tier2)
Implications: security, privacy, performance, scalability, and cost
1. Tenant Privacy and Cache Key Scoping
Never share cache keys across organizations. Always prepend the tenant organization ID to exact hashes and vector metadata (cache_key = f"tenant_id:query_hash") to prevent cross-tenant data leakage.
2. Cache Invalidation on Knowledge Updates
When underlying company documentation or product policies are updated, invalidate relevant semantic cache partitions immediately to prevent serving stale or deprecated answers.
Production Latency Case Study: Prompt Caching at Massive Context Scale
Consider an enterprise legal AI assistant analyzing 150-page contracts (approximately 80,000 tokens of static context per document). Without prompt caching, every follow-up question from a lawyer re-processes all 80,000 tokens through the modelβs self-attention matrix.
+-------------------------------------------------------------------------------+
| LONG-CONTEXT PROMPT CACHING COMPARISON |
+-------------------+-----------------------+-----------------------------------+
| Performance Metric| Without Prompt Cache | With Provider Prompt Cache (KV) |
+-------------------+-----------------------+-----------------------------------+
| Input Token Price | $3.00 / M ($0.24/req) | $0.30 / M ($0.024/req) [90% saved]|
| Time to First | 4,200 milliseconds | 450 milliseconds [89% faster] |
| Token (TTFT) | | |
| GPU Compute Load | 80,000 token prefill | 0 prefill (Reuses GPU KV tensors) |
| Session Cost (20Q)| $4.80 total | $0.48 total |
+-------------------+-----------------------+-----------------------------------+
1. Structuring Prompts for Prefix Cache Maximization
Modern provider prompt caches (such as Anthropic Claude Prompt Caching and OpenAI Cached Tokens) require static prefixes to match exactly from token 0. Placing dynamic timestamps or user IDs at the top of the prompt completely destroys cache reuse.
[POOR STRUCTURE - DESTROYS CACHE ON EVERY REQUEST]
System: You are an assistant. Current Time: 2026-08-30 14:45:12. User ID: u_94812.
[80,000 tokens of static legal documentation...]
User: Summarize clause 4.
[OPTIMAL STRUCTURE - 100% PROMPT CACHE HIT]
System: You are an assistant.
[80,000 tokens of static legal documentation...] (Marked with cache_control: {"type": "ephemeral"})
User: Current Time: 2026-08-30 14:45:12. User ID: u_94812. Summarize clause 4.
2. Semantic Cache Vector Invalidation Strategies
Semantic caches store embeddings of past queries and their validated responses. To prevent stale data from being served after knowledge updates, production systems partition vector collections by document hash and tenant ID. When a document is re-indexed, all associated semantic cache vectors are purged using collection namespace deletes (vector_index.delete(filter={"doc_id": "contract_94"})).
Production Failure Mode Taxonomy: Multi-Tier Caching Hazards
Deploying multi-tier caching in generative AI requires careful governance to avoid latency degradation and data poisoning:
+-------------------------------------------------------------------------------+
| LLM CACHING FAILURE TAXONOMY |
+-------------------+-----------------------+-----------------------------------+
| Caching Hazard | Root Cause Analysis | Cache Governance Mitigation |
+-------------------+-----------------------+-----------------------------------+
| False-Positive | Similarity threshold | Enforce strict cosine similarity |
| Semantic Hits | set too loose (< 0.92)| threshold (>= 0.95) for answers |
| Cross-Tenant Data | Cache key lacks tenant| Prepend tenant UUID to all Redis |
| Leakage | organization partition| keys and vector metadata filters |
| Stale Knowledge | Policy doc changes | Purge associated vector collection|
| Delivery | but cache persists | on document re-indexing |
| Cache Stampede | Hot key expires; 500 | Mutex lock on cache miss ensures |
| (Thundering Herd) | requests hit model GPU| only 1 request computes answer |
+-------------------+-----------------------+-----------------------------------+
Caching Architecture Operational Checklist
- All Tier 1 exact cache keys incorporate tenant IDs (
org_id:hash). - Vector similarity comparisons for Tier 2 caching use normalized unit vectors for accurate cosine distances.
- Provider prompt caches structure static system context as continuous prefixes exceeding 1,024 tokens.
- Cache entries have explicit TTL expiration timestamps to prevent infinite persistence of obsolete data.
Real-World Caching Deep Dive: Semantic Cache Vector Indexing and Pruning
Deploying semantic caching at production scale requires optimizing vector indexing, distance metrics, and cache hit verification:
+-------------------------------------------------------------------------------+
| SEMANTIC CACHE PERFORMANCE BENCHMARKS |
+-------------------+-----------------------+-----------------------------------+
| Vector Index Type | Lookup Latency | Accuracy & Memory Consumption |
+-------------------+-----------------------+-----------------------------------+
| Flat HNSW Index | 1.8 milliseconds | 99.4% recall | 240 MB per 100k |
| IVF-PQ Index | 0.6 milliseconds | 96.1% recall | 32 MB per 100k |
| Exact Brute Force | 14.5 milliseconds | 100% recall | High CPU load |
+-------------------+-----------------------+-----------------------------------+
Semantic Cache Engineering Rules
- Embedding Dimension Selection: Use compact embedding models (such as
text-embedding-3-smallwith 512 dimensions) to minimize vector storage memory and accelerate cosine dot products. - Length-Normalized Thresholding: Require higher similarity thresholds (e.g. $\ge 0.96$) for short queries ($< 5$ words) to avoid false positives, while allowing $\ge 0.93$ for long multi-sentence queries.
- Dynamic Response Updating: If an updated model generates a superior response for a cached query, update the stored response vector in place while retaining the original query embedding.
Alternatives: free, open source, and commercial
| Tool / Layer | Type | Best Suited For | Key Capabilities |
|---|---|---|---|
| Redis | Open Source | Tier 1 Exact Hash Caching | Sub-millisecond GET/SET with TTL |
| GPTCache | Open Source | Tier 2 Semantic Caching | LangChain/LlamaIndex integration |
| Anthropic Prompt Caching | Commercial Cloud | Tier 3 Provider KV Cache | 90% cost savings on system prompts |
| vLLM (PagedAttention) | Open Source | Self-hosted Model Serving | Automatic GPU KV cache management |
Comparison with related concepts
+-----------------------+-----------------------+-----------------------+
| Dimension | Exact Hash Cache | Semantic Vector Cache |
+-----------------------+-----------------------+-----------------------+
| Match Mechanism | Cryptographic SHA-256 | Cosine vector distance|
| Query Flexibility | 100% Identical only | Paraphrased variations|
| Lookup Latency | under 1 millisecond | 10 - 20 milliseconds |
| False Positive Risk | Zero (Deterministic) | Possible if sim < 0.92|
+-----------------------+-----------------------+-----------------------+
When to use it β and when not to
Deploy Multi-Tier Caching for:
- High-volume customer support bots, documentation assistants, and applications with large static system prompts ($> 1,000$ tokens).
Do NOT use semantic caching for:
- Highly dynamic, personalized queries containing real-time stock prices, private user IDs, or fast-changing state.
The AI connection
-
Caching is uniquely valuable here because generation is uniquely expensive β a cache hit is not a saved database query, it is a saved GPU-second.
-
Semantic caching only exists because of embeddings. No traditional cache could recognise that two differently worded questions want the same answer.
-
Prompt caching is a property of the modelβs KV cache, which is why prefix ordering matters: a dynamic timestamp at the top destroys reuse of everything below it.
-
Speculative decoding attacks the other half. Caching cannot help with tokens never generated before, and drafting-then-verifying can.
-
A loose similarity threshold is a correctness bug, not a tuning choice: 0.70 returns confident answers to questions nobody asked.
Knowledge check
- What is the difference between Prefill latency and Decode latency? Prefill latency is parallel GPU processing of input prompt tokens (TTFT); Decode latency is sequential generation of output tokens (ITL).
- Why does Provider Prompt Caching reduce costs by up to 90%? Because it caches the Key-Value (KV) activation tensors of static system prompt prefixes in GPU VRAM across requests.
- What is the main danger of an overly permissive semantic cache threshold? False positive cache hits that serve incorrect answers to subtly distinct questions.
Hands-on exercise
In this lab, you will build a Multi-Tier LLM Caching Engine in Python. Your implementation will:
- Compute SHA-256 hashes and store exact prompt-response pairs with TTL expiration.
- Implement semantic similarity lookup comparing query embedding vectors against cached entries.
- Enforce strict similarity thresholds to distinguish hits from misses.
- Support TTL eviction when cached items expire.
Expected output
============================= test session starts ==============================
collected 5 items
tests/test_caching_engine.py ..... [100%]
============================== 5 passed in 0.02s ===============================
[CACHE TIER 1] Exact hash hit returned in 0.4ms.
[CACHE TIER 2] Semantic hit detected with similarity = 0.98.
[CACHE MISS] Novel query correctly triggered model execution.
[TTL EVICTION] Expired entries successfully evicted.
Validate your work
Run the pytest test suite:
pytest tests/run_tests.sh
Troubleshooting
- Case-insensitive hashing: Normalize all queries with
.strip().lower()before hashing. - Vector dot product: Ensure vectors have non-zero norms before division to avoid division-by-zero errors.
Common mistakes
- Storing cache entries without timestamp metadata, making TTL eviction impossible.
Practice assignment
Add a Cache Invalidation Method (invalidate_by_pattern) that purges all cached entries matching a specific keyword or category.
Extension challenge
Implement a Prompt Caching Prefix Optimizer that splits incoming system prompts into static cacheable blocks ($> 1,024$ tokens) and dynamic user suffixes for Anthropic API headers.
Quiz
Q1. What are the three distinct phases of latency in an LLM request lifecycle?
- Compilation, Assembly, Execution
- Reading, Writing, Deleting
- Network, prefill, and decode
- HTML, CSS, JavaScript
Show answer
Answer: C. Network, prefill, and decode
LLM latency breaks down into network transit, prompt prefill (TTFT), and autoregressive decode generation.
Q2. What is the key difference between an 'Exact Hash Cache' and a 'Semantic Cache'?
- They are identical
- Exact cache is only for images
- Semantic cache works without a computer at all
- Hash matches bytes; semantic matches meaning
Show answer
Answer: D. Hash matches bytes; semantic matches meaning
Exact cache requires identical string matches; semantic cache matches semantically equivalent phrasing via vector similarity.
Q3. How does 'Provider Prompt Caching' (e.g. Anthropic Prompt Caching or OpenAI Cached Tokens) reduce latency and cost?
- It keeps the prefix KV cache on the provider's GPUs
- By deleting the prompt
- By speeding up user internet speed
- By compressing text into ZIP files
Show answer
Answer: A. It keeps the prefix KV cache on the provider's GPUs
Provider prompt caching reuses KV activations across requests, dramatically cutting TTFT and input costs.
Q4. What is the primary risk of setting a semantic cache similarity threshold too low (e.g. threshold = 0.70)?
- The database will run out of disk space
- False hits: near-miss queries get the wrong answer
- The server will overheat
- Tokens become unreadable
Show answer
Answer: B. False hits: near-miss queries get the wrong answer
Low similarity thresholds risk returning incorrect responses to distinct queries.
Q5. Where should static system instructions and few-shot examples be placed to maximize prompt caching hit rates?
- At the very end of the prompt
- Scattered randomly throughout the text
- At the start, with dynamic parts last
- In a separate audio recording
Show answer
Answer: C. At the start, with dynamic parts last
KV caches match from the beginning (prefix) of the prompt; static prefixes maximize cache hits.
Glossary
- Time to First Token (TTFT)
- The latency from request dispatch until the first token is received by the client.
- Inter-Token Latency (ITL)
- The average time in milliseconds between consecutive autoregressively generated tokens.
- Prefill Phase
- The initial GPU compute phase processing all input prompt tokens in parallel.
- Decode Phase
- The sequential autoregressive generation of output tokens one by one.
- Exact Hash Cache
- Caching responses based on exact cryptographic hashes (SHA-256) of input prompts.
- Semantic Cache
- Caching responses based on embedding vector cosine similarity above a strict threshold.
- KV Cache
- In-memory Key-Value activation tensors cached on GPU VRAM to avoid recomputing attention over past tokens.
- Prompt Caching
- Provider feature reusing KV caches for static prefix context across distinct user requests.
- Speculative Decoding
- Having a small draft model propose several tokens and the large model verify them in one forward pass. The output is mathematically identical to the large model alone.
- Draft Model
- The small, cheap model that proposes candidate tokens in speculative decoding. Its similarity to the target model decides how often its guesses are accepted.
- Acceptance Rate
- How often the target model agrees with the draft model proposal. It decides the speedup, and it is high on predictable spans and low on genuinely uncertain ones.
- Memory-Bandwidth Bound
- Limited by moving weights out of memory rather than by arithmetic. The decode phase is bandwidth bound, which is the asymmetry speculative decoding exploits.
Sources and further reading
- Anthropic: Prompt Caching Documentation (Anthropic, 2024) β Anthropic Documentation (accessed 2026-08-30)
- GPTCache: An Open-Source Semantic Cache for LLMs (Bang et al., 2023) β GitHub (accessed 2026-08-30)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023) β SOSP (accessed 2026-08-30)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.