LLMs and Generative AI › The LLM Landscape › Day 243
Hands-on lab — Day 243: Tokens, Context Windows, and Sampling
- ← Back to the Day 243 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-243-tokens-context-windows-and-sampling/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/tokens_context_windows_and_sampling_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/test_tokens_context_windows_and_sampling_lib.py examples/tokens_context_windows_and_sampling_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/test_tokens_context_windows_and_sampling_lib.py starter/tokens_context_windows_and_sampling_lib.py tests/run_tests.sh tests/test_tokens_context_windows_and_sampling_lib.py troubleshooting.md
Lab README
Lab: Day 243 -- Tokens, Context Windows, and Sampling
Lesson
Day number: 243 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Tokens, Context Windows, KV Cache, and Probabilistic Sampling.
Purpose
Build and test a modular probabilistic sampling and KV-cache calculation engine in PyTorch. Implement Temperature scaling, Top-$k$ masking, Top-$p$ Nucleus truncation, and calculate exact KV-cache VRAM consumption across model architectures.
Learning objectives
- Formulate the exact mathematical KV-cache memory scaling equation.
- Implement Temperature logit scaling ($z_i / T$) and analyze entropy changes.
- Build Top-$k$ and Top-$p$ (Nucleus) cumulative mass logit masking filters.
- Understand the hardware difference between compute-bound prefill and memory-bound token decoding.
Prerequisites
- Day 242 (Open Weights versus Closed APIs).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/tokens_context_windows_and_sampling_lib.py: Student scaffold file.examples/tokens_context_windows_and_sampling_lib.py: Complete reference implementation.tests/test_tokens_context_windows_and_sampling_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/tokens_context_windows_and_sampling_lib.py
What the commands do
- Computes exact KV-cache VRAM footprint for specified context lengths.
- Executes Greedy, Top-$k$, and Top-$p$ sampling over test logit distributions.
- Runs unit test assertions.
Expected output
Sampling Demo: KV Cache (8k context) = 2.50 GB, Greedy Token = 42, Sampled = 42
Validation steps
- Verify
calculate_kv_cache_bytescomputes exact Key + Value byte sizes. - Confirm
sample_next_tokenreturns deterministic argmax when $T = 0.0$. - Confirm Top-$k$ and Top-$p$ filters mask tail probabilities to $-\infty$.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Probability sum error: Ensure
F.softmaxis applied after masking logits with $-\infty$.
Security notes
All sampling algorithms execute locally in process memory.
Extension exercises
- Implement the Min-P dynamic sampling filter.
- Build an automated KV-cache VRAM capacity estimator for multi-GPU servers.
Navigation
- Lesson title: Tokens, Context Windows, and Sampling
- Day number: 243 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-243-tokens-context-windows-and-sampling
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-243-tokens-context-windows-and-samplingwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 243
- `KV Cache`: Total VRAM footprint in Gigabytes for the specified sequence length.
- `Greedy Token`: Selected token index under greedy decoding ($T = 0$).
- `Sampled`: Selected token index under probabilistic nucleus sampling.
examples-run.txt
Sampling Demo: KV Cache (8k context) = 2.50 GB, Greedy Token = 42, Sampled = 42
measured-values.txt
KV Cache: 2.50 GB
Greedy Token: 42
Sampled: 42
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_tokens_context_windows_and_sampling_lib.py::test_kv_cache_calculation PASSED [ 33%]
tests/test_tokens_context_windows_and_sampling_lib.py::test_greedy_sampling_determinism PASSED [ 66%]
tests/test_tokens_context_windows_and_sampling_lib.py::test_top_k_sampling_bounds PASSED [100%]
============================== 3 passed in 0.43s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/test_tokens_context_windows_and_sampling_lib.py (996 bytes)
import pytest
import torch
from examples.tokens_context_windows_and_sampling_lib import (
calculate_kv_cache_bytes,
sample_next_token
)
def test_kv_cache_calculation():
# 1 batch, 1024 seq, 32 layers, 32 heads, 64 dim, 2 bytes
# 2 * 1 * 1024 * 32 * (32 * 64) * 2 = 2 * 1024 * 32 * 2048 * 2 = 268,435,456 bytes = 256 MB
kv_bytes = calculate_kv_cache_bytes(1, 1024, 32, 32, 64, bytes_per_elem=2)
assert kv_bytes == 268435456
def test_greedy_sampling_determinism():
logits = torch.tensor([1.0, 2.0, 5.0, 3.0])
for _ in range(10):
# Index 2 has highest logit (5.0)
tok = sample_next_token(logits, temperature=0.0)
assert tok == 2
def test_top_k_sampling_bounds():
torch.manual_seed(42)
# Logits where top 2 are indices 0 and 1, rest are very negative
logits = torch.tensor([10.0, 9.0, -100.0, -100.0, -100.0])
for _ in range(10):
tok = sample_next_token(logits, temperature=1.0, top_k=2)
assert tok in [0, 1]
examples/tokens_context_windows_and_sampling_lib.py (2216 bytes)
import torch
import torch.nn.functional as F
from typing import List, Optional
def calculate_kv_cache_bytes(
batch_size: int,
seq_len: int,
num_layers: int,
num_kv_heads: int,
head_dim: int,
bytes_per_elem: int = 2
) -> int:
# 2 (Keys + Values) * Batch * SeqLen * NumLayers * (NumKVHeads * HeadDim) * BytesPerElem
return 2 * batch_size * seq_len * num_layers * (num_kv_heads * head_dim) * bytes_per_elem
def sample_next_token(
logits: torch.Tensor,
temperature: float = 1.0,
top_k: int = 0,
top_p: float = 1.0
) -> int:
logits = logits.clone().float()
if temperature <= 1e-5:
return int(torch.argmax(logits).item())
logits = logits / temperature
# Top-k filtering
if top_k > 0:
topk_vals, _ = torch.topk(logits, min(top_k, logits.shape[-1]))
min_topk = topk_vals[-1]
logits[logits < min_topk] = -float('inf')
# Top-p filtering
if 0.0 < top_p < 1.0:
sorted_logits, sorted_indices = torch.sort(logits, descending=True)
cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
sorted_indices_to_remove = cumulative_probs > top_p
sorted_indices_to_remove[1:] = sorted_indices_to_remove[:-1].clone()
sorted_indices_to_remove[0] = False
indices_to_remove = sorted_indices[sorted_indices_to_remove]
logits[indices_to_remove] = -float('inf')
probs = F.softmax(logits, dim=-1)
return int(torch.multinomial(probs, num_samples=1).item())
def run_sampling_demo():
torch.manual_seed(42)
# LLaMA-3-70B: 80 layers, 8 KV heads, head_dim 128, 8k context
kv_bytes = calculate_kv_cache_bytes(1, 8192, 80, 8, 128, bytes_per_elem=2)
kv_gb = kv_bytes / (1024**3)
# Simulated vocabulary logits (vocab size 100)
logits = torch.randn(100)
logits[42] = 10.0 # Clear winner
tok_greedy = sample_next_token(logits, temperature=0.0)
tok_temp = sample_next_token(logits, temperature=0.7, top_p=0.9)
print(f"Sampling Demo: KV Cache (8k context) = {kv_gb:.2f} GB, Greedy Token = {tok_greedy}, Sampled = {tok_temp}")
return kv_gb, tok_greedy, tok_temp
if __name__ == "__main__":
run_sampling_demo()
metadata.yml (448 bytes)
lesson_id: D243
day: 243
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/tokens_context_windows_and_sampling_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_tokens_context_windows_and_sampling_lib.py (996 bytes)
import pytest
import torch
from examples.tokens_context_windows_and_sampling_lib import (
calculate_kv_cache_bytes,
sample_next_token
)
def test_kv_cache_calculation():
# 1 batch, 1024 seq, 32 layers, 32 heads, 64 dim, 2 bytes
# 2 * 1 * 1024 * 32 * (32 * 64) * 2 = 2 * 1024 * 32 * 2048 * 2 = 268,435,456 bytes = 256 MB
kv_bytes = calculate_kv_cache_bytes(1, 1024, 32, 32, 64, bytes_per_elem=2)
assert kv_bytes == 268435456
def test_greedy_sampling_determinism():
logits = torch.tensor([1.0, 2.0, 5.0, 3.0])
for _ in range(10):
# Index 2 has highest logit (5.0)
tok = sample_next_token(logits, temperature=0.0)
assert tok == 2
def test_top_k_sampling_bounds():
torch.manual_seed(42)
# Logits where top 2 are indices 0 and 1, rest are very negative
logits = torch.tensor([10.0, 9.0, -100.0, -100.0, -100.0])
for _ in range(10):
tok = sample_next_token(logits, temperature=1.0, top_k=2)
assert tok in [0, 1]
starter/tokens_context_windows_and_sampling_lib.py (481 bytes)
import torch
from typing import List, Optional
def calculate_kv_cache_bytes(
batch_size: int,
seq_len: int,
num_layers: int,
num_kv_heads: int,
head_dim: int,
bytes_per_elem: int = 2
) -> int:
# TODO: Calculate total KV cache size in bytes
pass
def sample_next_token(
logits: torch.Tensor,
temperature: float = 1.0,
top_k: int = 0,
top_p: float = 1.0
) -> int:
# TODO: Implement Temperature, Top-k, and Top-p sampling
pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 243 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_tokens_context_windows_and_sampling_lib.py (996 bytes)
import pytest
import torch
from examples.tokens_context_windows_and_sampling_lib import (
calculate_kv_cache_bytes,
sample_next_token
)
def test_kv_cache_calculation():
# 1 batch, 1024 seq, 32 layers, 32 heads, 64 dim, 2 bytes
# 2 * 1 * 1024 * 32 * (32 * 64) * 2 = 2 * 1024 * 32 * 2048 * 2 = 268,435,456 bytes = 256 MB
kv_bytes = calculate_kv_cache_bytes(1, 1024, 32, 32, 64, bytes_per_elem=2)
assert kv_bytes == 268435456
def test_greedy_sampling_determinism():
logits = torch.tensor([1.0, 2.0, 5.0, 3.0])
for _ in range(10):
# Index 2 has highest logit (5.0)
tok = sample_next_token(logits, temperature=0.0)
assert tok == 2
def test_top_k_sampling_bounds():
torch.manual_seed(42)
# Logits where top 2 are indices 0 and 1, rest are very negative
logits = torch.tensor([10.0, 9.0, -100.0, -100.0, -100.0])
for _ in range(10):
tok = sample_next_token(logits, temperature=1.0, top_k=2)
assert tok in [0, 1]
Troubleshooting
Troubleshooting: Day 243 - Tokens, Context Windows, and Sampling
Common Issues
- NaN in Softmax after Top-p:
- Cause: Masking out all tokens because cumulative probability condition was inverted.
- Fix: Ensure
sorted_indices_to_remove[0] = Falseto always keep the top token.
Security notes
Security & Privacy: Day 243 - Tokens, Context Windows, and Sampling
Security Guidance
- All sampling and cache computations execute locally in memory.