LLMs and Generative AI › The LLM Landscape › Day 243

Hands-on lab — Day 243: Tokens, Context Windows, and Sampling

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/tokens_context_windows_and_sampling_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/test_tokens_context_windows_and_sampling_lib.py
examples/tokens_context_windows_and_sampling_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_tokens_context_windows_and_sampling_lib.py
starter/tokens_context_windows_and_sampling_lib.py
tests/run_tests.sh
tests/test_tokens_context_windows_and_sampling_lib.py
troubleshooting.md

Lab README

Lab: Day 243 -- Tokens, Context Windows, and Sampling

Lesson

Day number: 243 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Tokens, Context Windows, KV Cache, and Probabilistic Sampling.

Purpose

Build and test a modular probabilistic sampling and KV-cache calculation engine in PyTorch. Implement Temperature scaling, Top-$k$ masking, Top-$p$ Nucleus truncation, and calculate exact KV-cache VRAM consumption across model architectures.

Learning objectives

  • Formulate the exact mathematical KV-cache memory scaling equation.
  • Implement Temperature logit scaling ($z_i / T$) and analyze entropy changes.
  • Build Top-$k$ and Top-$p$ (Nucleus) cumulative mass logit masking filters.
  • Understand the hardware difference between compute-bound prefill and memory-bound token decoding.

Prerequisites

  • Day 242 (Open Weights versus Closed APIs).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/tokens_context_windows_and_sampling_lib.py: Student scaffold file.
  • examples/tokens_context_windows_and_sampling_lib.py: Complete reference implementation.
  • tests/test_tokens_context_windows_and_sampling_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/tokens_context_windows_and_sampling_lib.py

What the commands do

  • Computes exact KV-cache VRAM footprint for specified context lengths.
  • Executes Greedy, Top-$k$, and Top-$p$ sampling over test logit distributions.
  • Runs unit test assertions.

Expected output

Sampling Demo: KV Cache (8k context) = 2.50 GB, Greedy Token = 42, Sampled = 42

Validation steps

  1. Verify calculate_kv_cache_bytes computes exact Key + Value byte sizes.
  2. Confirm sample_next_token returns deterministic argmax when $T = 0.0$.
  3. Confirm Top-$k$ and Top-$p$ filters mask tail probabilities to $-\infty$.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Probability sum error: Ensure F.softmax is applied after masking logits with $-\infty$.

Security notes

All sampling algorithms execute locally in process memory.

Extension exercises

  1. Implement the Min-P dynamic sampling filter.
  2. Build an automated KV-cache VRAM capacity estimator for multi-GPU servers.

Expected output

FIELDS.md

# Expected Output Fields: Day 243

- `KV Cache`: Total VRAM footprint in Gigabytes for the specified sequence length.
- `Greedy Token`: Selected token index under greedy decoding ($T = 0$).
- `Sampled`: Selected token index under probabilistic nucleus sampling.

examples-run.txt

Sampling Demo: KV Cache (8k context) = 2.50 GB, Greedy Token = 42, Sampled = 42

measured-values.txt

KV Cache: 2.50 GB
Greedy Token: 42
Sampled: 42

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_tokens_context_windows_and_sampling_lib.py::test_kv_cache_calculation PASSED [ 33%]
tests/test_tokens_context_windows_and_sampling_lib.py::test_greedy_sampling_determinism PASSED [ 66%]
tests/test_tokens_context_windows_and_sampling_lib.py::test_top_k_sampling_bounds PASSED [100%]

============================== 3 passed in 0.43s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/test_tokens_context_windows_and_sampling_lib.py (996 bytes)
import pytest
import torch
from examples.tokens_context_windows_and_sampling_lib import (
    calculate_kv_cache_bytes,
    sample_next_token
)

def test_kv_cache_calculation():
    # 1 batch, 1024 seq, 32 layers, 32 heads, 64 dim, 2 bytes
    # 2 * 1 * 1024 * 32 * (32 * 64) * 2 = 2 * 1024 * 32 * 2048 * 2 = 268,435,456 bytes = 256 MB
    kv_bytes = calculate_kv_cache_bytes(1, 1024, 32, 32, 64, bytes_per_elem=2)
    assert kv_bytes == 268435456

def test_greedy_sampling_determinism():
    logits = torch.tensor([1.0, 2.0, 5.0, 3.0])
    for _ in range(10):
        # Index 2 has highest logit (5.0)
        tok = sample_next_token(logits, temperature=0.0)
        assert tok == 2

def test_top_k_sampling_bounds():
    torch.manual_seed(42)
    # Logits where top 2 are indices 0 and 1, rest are very negative
    logits = torch.tensor([10.0, 9.0, -100.0, -100.0, -100.0])
    for _ in range(10):
        tok = sample_next_token(logits, temperature=1.0, top_k=2)
        assert tok in [0, 1]
examples/tokens_context_windows_and_sampling_lib.py (2216 bytes)
import torch
import torch.nn.functional as F
from typing import List, Optional

def calculate_kv_cache_bytes(
    batch_size: int,
    seq_len: int,
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    bytes_per_elem: int = 2
) -> int:
    # 2 (Keys + Values) * Batch * SeqLen * NumLayers * (NumKVHeads * HeadDim) * BytesPerElem
    return 2 * batch_size * seq_len * num_layers * (num_kv_heads * head_dim) * bytes_per_elem

def sample_next_token(
    logits: torch.Tensor,
    temperature: float = 1.0,
    top_k: int = 0,
    top_p: float = 1.0
) -> int:
    logits = logits.clone().float()

    if temperature <= 1e-5:
        return int(torch.argmax(logits).item())

    logits = logits / temperature

    # Top-k filtering
    if top_k > 0:
        topk_vals, _ = torch.topk(logits, min(top_k, logits.shape[-1]))
        min_topk = topk_vals[-1]
        logits[logits < min_topk] = -float('inf')

    # Top-p filtering
    if 0.0 < top_p < 1.0:
        sorted_logits, sorted_indices = torch.sort(logits, descending=True)
        cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)

        sorted_indices_to_remove = cumulative_probs > top_p
        sorted_indices_to_remove[1:] = sorted_indices_to_remove[:-1].clone()
        sorted_indices_to_remove[0] = False

        indices_to_remove = sorted_indices[sorted_indices_to_remove]
        logits[indices_to_remove] = -float('inf')

    probs = F.softmax(logits, dim=-1)
    return int(torch.multinomial(probs, num_samples=1).item())

def run_sampling_demo():
    torch.manual_seed(42)
    # LLaMA-3-70B: 80 layers, 8 KV heads, head_dim 128, 8k context
    kv_bytes = calculate_kv_cache_bytes(1, 8192, 80, 8, 128, bytes_per_elem=2)
    kv_gb = kv_bytes / (1024**3)

    # Simulated vocabulary logits (vocab size 100)
    logits = torch.randn(100)
    logits[42] = 10.0 # Clear winner

    tok_greedy = sample_next_token(logits, temperature=0.0)
    tok_temp = sample_next_token(logits, temperature=0.7, top_p=0.9)

    print(f"Sampling Demo: KV Cache (8k context) = {kv_gb:.2f} GB, Greedy Token = {tok_greedy}, Sampled = {tok_temp}")
    return kv_gb, tok_greedy, tok_temp

if __name__ == "__main__":
    run_sampling_demo()
metadata.yml (448 bytes)
lesson_id: D243
day: 243
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/tokens_context_windows_and_sampling_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_tokens_context_windows_and_sampling_lib.py (996 bytes)
import pytest
import torch
from examples.tokens_context_windows_and_sampling_lib import (
    calculate_kv_cache_bytes,
    sample_next_token
)

def test_kv_cache_calculation():
    # 1 batch, 1024 seq, 32 layers, 32 heads, 64 dim, 2 bytes
    # 2 * 1 * 1024 * 32 * (32 * 64) * 2 = 2 * 1024 * 32 * 2048 * 2 = 268,435,456 bytes = 256 MB
    kv_bytes = calculate_kv_cache_bytes(1, 1024, 32, 32, 64, bytes_per_elem=2)
    assert kv_bytes == 268435456

def test_greedy_sampling_determinism():
    logits = torch.tensor([1.0, 2.0, 5.0, 3.0])
    for _ in range(10):
        # Index 2 has highest logit (5.0)
        tok = sample_next_token(logits, temperature=0.0)
        assert tok == 2

def test_top_k_sampling_bounds():
    torch.manual_seed(42)
    # Logits where top 2 are indices 0 and 1, rest are very negative
    logits = torch.tensor([10.0, 9.0, -100.0, -100.0, -100.0])
    for _ in range(10):
        tok = sample_next_token(logits, temperature=1.0, top_k=2)
        assert tok in [0, 1]
starter/tokens_context_windows_and_sampling_lib.py (481 bytes)
import torch
from typing import List, Optional

def calculate_kv_cache_bytes(
    batch_size: int,
    seq_len: int,
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    bytes_per_elem: int = 2
) -> int:
    # TODO: Calculate total KV cache size in bytes
    pass

def sample_next_token(
    logits: torch.Tensor,
    temperature: float = 1.0,
    top_k: int = 0,
    top_p: float = 1.0
) -> int:
    # TODO: Implement Temperature, Top-k, and Top-p sampling
    pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 243 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_tokens_context_windows_and_sampling_lib.py (996 bytes)
import pytest
import torch
from examples.tokens_context_windows_and_sampling_lib import (
    calculate_kv_cache_bytes,
    sample_next_token
)

def test_kv_cache_calculation():
    # 1 batch, 1024 seq, 32 layers, 32 heads, 64 dim, 2 bytes
    # 2 * 1 * 1024 * 32 * (32 * 64) * 2 = 2 * 1024 * 32 * 2048 * 2 = 268,435,456 bytes = 256 MB
    kv_bytes = calculate_kv_cache_bytes(1, 1024, 32, 32, 64, bytes_per_elem=2)
    assert kv_bytes == 268435456

def test_greedy_sampling_determinism():
    logits = torch.tensor([1.0, 2.0, 5.0, 3.0])
    for _ in range(10):
        # Index 2 has highest logit (5.0)
        tok = sample_next_token(logits, temperature=0.0)
        assert tok == 2

def test_top_k_sampling_bounds():
    torch.manual_seed(42)
    # Logits where top 2 are indices 0 and 1, rest are very negative
    logits = torch.tensor([10.0, 9.0, -100.0, -100.0, -100.0])
    for _ in range(10):
        tok = sample_next_token(logits, temperature=1.0, top_k=2)
        assert tok in [0, 1]

Troubleshooting

Troubleshooting: Day 243 - Tokens, Context Windows, and Sampling

Common Issues

  1. NaN in Softmax after Top-p:
    • Cause: Masking out all tokens because cumulative probability condition was inverted.
    • Fix: Ensure sorted_indices_to_remove[0] = False to always keep the top token.

Security notes

Security & Privacy: Day 243 - Tokens, Context Windows, and Sampling

Security Guidance

  • All sampling and cache computations execute locally in memory.