Deep Learning β€Ί Transformers β€Ί Day 229

Day 229: Decoder Models: The GPT Family

Day 229 of 365 β€” Decoder Models: The GPT Family

Master Autoregressive Decoder-only Transformer foundation models: understand GPT-1, GPT-2, and GPT-3, derive Causal Self-Attention masking matrices, analyze KV Caching for O(1) step latency, understand Neural Scaling Laws (Kaplan vs Chinchilla compute-optimal frontier), master generation decoding strategies (Temperature, Top-k, Top-p, Min-p), and build an autoregressive generation loop in PyTorch.

Course
Deep Learning
Category
Transformers
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-229-decoder-models-the-gpt-family

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-229-decoder-models-the-gpt-family
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

While Encoder models (BERT) unlocked bidirectional understanding for classification, they cannot write poetry, author software code, or carry on dynamic human conversation.

To achieve open-ended language generation, researchers turned to Autoregressive Decoder Models:

From GPT-1 (117M params, 2018) to GPT-2 (1.5B params, 2019) to GPT-3 (175B params, 2020) and modern open-weight giants (LLaMA 3, Mistral, DeepSeek), the autoregressive decoder has become the fundamental architecture powering modern artificial intelligence.

Today, you will master Causal Self-Attention, KV-Caching, Neural Scaling Laws, and advanced sampling decoding strategies.


The idea in plain language

Diagram: how autoregressive next-token prediction works

Think of a master storyteller improvising an epic tale before a live audience:


Historical background

GPT causal autoregressive architecture diagram showing lower triangular causal attention mask next token prediction and KV cache mechanism

  1. 2018 (Radford et al. & GPT-1): Demonstrated that unsupervised generative pretraining followed by supervised fine-tuning outperformed task-specific architectures across 9 NLP benchmarks.
  2. 2019 (Radford et al. & GPT-2): β€œLanguage Models are Unsupervised Multitask Learners” proved that simply scaling GPT-1 to 1.5B parameters enabled zero-shot translation, summarization, and question answering without any task-specific fine-tuning.
  3. 2020 (Brown et al. & GPT-3): Scaled to 175B parameters and demonstrated In-Context Few-Shot Learning: prompting the model with 2 to 3 demonstrations in plain text allowed it to perform novel tasks with zero weight updates.
  4. 2022 (Hoffmann et al. & Chinchilla): Corrected Kaplan’s scaling laws, establishing that compute-optimal models require scaling tokens and parameters in equal proportion (20 tokens per parameter), birthing the modern dense LLM era (LLaMA, Mistral).

What it is β€” and what it is not

What Autoregressive Decoders ARE:

What they are NOT:


Why it was created and what problems it solves

Autoregressive decoders solved the fundamental trade-off between task specialization and generality:

  1. Eliminating Task-Specific Output Heads: Instead of engineering custom classification heads for every task, the model outputs natural language answers directly into the token stream.
  2. Infinite Generation Rollout: The autoregressive loop allows the model to produce arbitrarily long paragraphs, essays, code modules, and complex multi-step reasoning traces.

How it works

Let us dissect the causal mechanics, KV-caching acceleration, scaling laws, and decoding algorithms.


1. Causal Self-Attention Masking

To train a Transformer decoder on an entire sequence in parallel while preventing future token leakage:

  1. Construct an upper-triangular mask matrix M \in \mathbb{R}^{N \times N}:

    M_{i, j} = \begin{cases} 0 & \text{if } j \le i \\ -\infty & \text{if } j > i \end{cases}
  2. Add the mask to raw attention scores prior to softmax:

    \text{CausalAttention}(Q, K, V) = \text{Softmax}\left( \frac{Q K^\top}{\sqrt{d_k}} + M \right) V
  3. Because \exp(-\infty) = 0, all attention weights pointing to future tokens become exactly zero.


2. Fast Inference Rollout with KV-Caching

Diagram: how the key-value cache avoids recomputing attention over the prefix

During inference, generating text token-by-token is computationally expensive if recomputed naively:


3. Neural Scaling Laws: Kaplan vs. Chinchilla

Diagram: parameter-heavy scaling compared with compute-optimal scaling

Neural scaling laws comparison showing Kaplan empirical power laws versus Chinchilla compute optimal tokens per parameter balance

How should compute budget C \approx 6 N D (where N is parameter count and D is dataset token count) be allocated?

1. Kaplan et al. (2020) Power Law:

2. Hoffmann et al. (2022) Chinchilla Law:


4. Generation Decoding Strategies

Diagram: four decoding strategies for autoregressive generation

Given final vocabulary logits z \in \mathbb{R}^{V}:

  1. Greedy Search: Always pick \hat{y} = \arg\max z_i. Fast, but suffers from repetitive loops and lack of diversity.

  2. Temperature Scaling (T): Rescales logits:

    P(w_i) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}
    • T \rightarrow 0: Collapses into greedy search.
    • T = 1.0: True model probability.
    • T > 1.0: High entropy, creative randomness.
  3. Top-k Sampling (Fan et al., 2018): Restricts sampling strictly to the k highest-probability tokens.

  4. Top-p (Nucleus) Sampling (Holtzman et al., 2019): Selects the dynamic minimal subset of tokens whose cumulative probability \sum P(w_i) \ge p (e.g. p = 0.9).

  5. Min-p Sampling (Grimsley et al., 2024): Filters out any token whose probability is less than p_{\text{base}} \times P(\text{top token}).


5. Repetition and Frequency/Presence Penalties

Diagram: the causes and remedies of repetitive generation

Autoregressive decoders naturally suffer from degenerate repetition loops when sampling temperature is low:


6. Contrastive Search: Coherence Without Degeneration (Su et al., 2022)

To combine the factual rigor of greedy search with the linguistic diversity of stochastic sampling:


7. PagedAttention & vLLM: Virtual Memory for the KV-Cache (Kwon et al., 2023)

Diagram: contiguous key-value cache allocation compared with paged allocation

In multi-tenant LLM serving:


8. Speculative Decoding Mathematical Mechanics

Diagram: why autoregressive generation is memory-bandwidth bound

Autoregressive generation is strictly memory-bandwidth bound because reading the model parameters from GPU HBM memory takes 100x longer than the actual floating-point GEMM arithmetic:


An everyday analogy

Diagram: improvised narration compared with a written draft

Think of an automated passenger jet flying on autopilot:


Examples in practice

Let us build a Complete GPT Decoder with KV-Cache Autoregressive Generation in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F

class GPTDecoderLayer(nn.Module):
    def __init__(self, d_model: int, num_heads: int):
        super().__init__()
        self.norm1 = nn.LayerNorm(d_model)
        self.self_attn = nn.MultiheadAttention(d_model, num_heads, batch_first=True)
        self.norm2 = nn.LayerNorm(d_model)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, 4 * d_model),
            nn.GELU(),
            nn.Linear(4 * d_model, d_model)
        )

    def forward(self, x: torch.Tensor, mask: torch.Tensor = None) -> torch.Tensor:
        normed = self.norm1(x)
        attn_out, _ = self.self_attn(normed, normed, normed, attn_mask=mask, is_causal=True)
        x = x + attn_out
        x = x + self.ffn(self.norm2(x))
        return x

class MiniatureGPT(nn.Module):
    def __init__(self, vocab_size: int = 1000, d_model: int = 64,
                 num_layers: int = 4, num_heads: int = 4):
        super().__init__()
        self.token_embeddings = nn.Embedding(vocab_size, d_model)
        self.pos_embeddings = nn.Embedding(512, d_model)
        self.layers = nn.ModuleList([
            GPTDecoderLayer(d_model, num_heads) for _ in range(num_layers)
        ])
        self.final_norm = nn.LayerNorm(d_model)
        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
        self.lm_head.weight = self.token_embeddings.weight

    def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
        seq_len = input_ids.size(1)
        pos = torch.arange(seq_len, device=input_ids.device).unsqueeze(0)
        x = self.token_embeddings(input_ids) + self.pos_embeddings(pos)

        for layer in self.layers:
            x = layer(x)

        x = self.final_norm(x)
        logits = self.lm_head(x)
        return logits

    @torch.no_grad()
    def generate(self, prompt_ids: torch.Tensor, max_new_tokens: int = 10,
                 temperature: float = 0.8, top_k: int = 20) -> torch.Tensor:
        current_ids = prompt_ids
        for _ in range(max_new_tokens):
            logits = self.forward(current_ids)[:, -1, :] # Last token logits
            logits = logits / temperature

            if top_k > 0:
                values, _ = torch.topk(logits, top_k)
                min_val = values[:, -1].unsqueeze(-1)
                logits = torch.where(logits < min_val, torch.tensor(-float('Inf'), device=logits.device), logits)

            probs = F.softmax(logits, dim=-1)
            next_token = torch.multinomial(probs, num_samples=1)
            current_ids = torch.cat([current_ids, next_token], dim=1)

        return current_ids

Implications: security, privacy, performance, scalability, and cost

Diagram: the latency, memory, cost and safety implications of autoregressive decoding

  1. Speculative Decoding (Leviathan et al., 2023):
    • Autoregressive decoding is strictly memory-bandwidth bound on GPUs.
    • Speculative Execution: A small draft model (e.g. 1B) rapidly guesses 5 candidate tokens, which are verified in a single parallel forward pass by the large target model (e.g. 70B), speeding up inference 2.5\times to 3\times with bit-level mathematical equivalence.
  2. Hallucination & Sycophancy:
    • Causal LMs optimize purely for likelihood, not objective truth. Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are required to align generation outputs with human intent.

Alternatives: free, open source, and commercial

ArchitectureParametersContext WindowNotable Feature
GPT-4oUndisclosed128kFrontier multimodal omni model
Llama 3.18B / 70B / 405B128kPremier open-weight foundation model
Mistral-Large123B128kState-of-the-art multilingual reasoning
DeepSeek-V3671B (37B active MoE)128kMulti-Head Latent Attention (MLA) efficiency

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   DECODER SAMPLING STRATEGIES COMPARISON               β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Strategy           β”‚ Deterministic?  β”‚ Diversity        β”‚ Best Use     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Greedy Search      β”‚ Yes             β”‚ Very Low         β”‚ Math, SQL    β”‚
β”‚ Temp = 0.2         β”‚ Low Randomness  β”‚ Controlled       β”‚ Code Gen     β”‚
β”‚ Top-p = 0.9, T=0.8 β”‚ High Randomness β”‚ Rich Diversity   β”‚ Storytelling β”‚
β”‚ Min-p = 0.05       β”‚ Dynamic Cutoff  β”‚ High Quality     β”‚ General Chat β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when autoregressive decoders are the right choice

When to USE Autoregressive Decoders:

When NOT to use:


The AI connection

Knowledge check

  1. Why does Causal Masking set future attention scores to -\infty rather than 0?
  2. How does the KV-Cache reduce single-token autoregressive generation complexity from O(t) to O(1)?
  3. What is the Chinchilla scaling law and what is the optimal token-to-parameter ratio?
  4. How does Temperature scaling affect the softmax entropy of vocabulary logits?
  5. How does Top-p (Nucleus) sampling dynamically adapt its candidate pool size across different contexts?

Hands-on exercise

In this lab, you will build and test a complete MiniatureGPT causal language model in PyTorch: implement causal lower-triangular masking, construct the autoregressive decoder pipeline with Pre-LN residual streams, implement the temperature and top-k token generation rollout loop, and verify step-by-step autoregressive generation.

Expected output

[GPT Autoregressive Foundation Models Verification Suite]
Testing MiniatureGPT Forward Pass:
  Input Prompt Shape: torch.Size([2, 5]) (Batch=2, Prompt=5 tokens)
  Vocabulary Logits Output: torch.Size([2, 5, 200]) [CAUSAL SHAPES MATCH]
Testing Autoregressive Generation Rollout:
  Prompt Length: 5 tokens -> Generated Length: 15 tokens (+10 tokens generated)
  Autoregressive Loop Step Latency: Verified constant per step
Testing Temperature & Top-K Sampling:
  Temperature T=0.7, Top-K=10 filtered probability distribution verified.
Test Suite: 3 passed in 0.44s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Top-p (Nucleus) Sampling in pure PyTorch and verify cumulative probability thresholding.
  2. Implement an explicit KV-Cache module that stores past Key/Value tensors in a persistent buffer.

Extension challenge

Implement Speculative Decoding in Pure PyTorch:

Quiz

Q1. What is the mathematical purpose of the Causal Attention Mask (lower-triangular mask) in GPT decoder models?

  1. Future positions are set to -inf
  2. It masks out punctuation characters
  3. It forces the weights to sum to zero
  4. It converts text to uppercase
Show answer

Answer: A. Future positions are set to -inf

Causal masking ensures that during training and generation, the model cannot cheat by looking ahead at future tokens.

Q2. Why is the Key-Value (KV) Cache critical for low-latency autoregressive generation in Large Language Models?

  1. KV caching encrypts the prompts for security
  2. Caching K and V makes each new token O(1) instead of O(N)
  3. KV caching makes the vocabulary 10 times larger
  4. KV caching converts floating-point numbers to integers
Show answer

Answer: B. Caching K and V makes each new token O(1) instead of O(N)

KV caching stores previously computed key and value states, eliminating redundant recomputation during step-by-step rollout.

Q3. What was the fundamental paradigm shift introduced by Chinchilla (Hoffmann et al., 2022) over Kaplan et al. (2020)?

  1. That Transformers should be replaced with recurrent networks
  2. That neural networks cannot scale beyond 1 billion parameters
  3. Parameters and tokens should scale together, ~20 per param
  4. That models should only be trained on Wikipedia
Show answer

Answer: C. Parameters and tokens should scale together, ~20 per param

Chinchilla demonstrated that training smaller models on significantly more data yields better performance and far lower inference serving costs.

Q4. How does Temperature scaling (T) affect next-token probability distribution sampling?

  1. T doubles the model hidden dimension
  2. T increases the physical clock frequency of the GPU
  3. T deletes rare vocabulary words
  4. Dividing logits by T sharpens or flattens it
Show answer

Answer: D. Dividing logits by T sharpens or flattens it

Temperature scales logit magnitudes, controlling the sharpness or flatness of the resulting categorical probability distribution.

Q5. What is Top-p (Nucleus) sampling (Holtzman et al., 2019)?

  1. Sample from the smallest set whose mass exceeds p
  2. Selecting always the top p tokens alphabetically
  3. Selecting only tokens containing the letter p
  4. Padding the input sequence to length p
Show answer

Answer: A. Sample from the smallest set whose mass exceeds p

Nucleus sampling dynamically adjusts the candidate pool size based on the model confidence distribution.

Glossary

Autoregressive Language Model
A model that predicts the next token in a sequence conditioned on all previously generated tokens: P(w_1, ..., w_N) = prod P(w_t | w_<t).
Causal Mask
An upper-triangular negative infinity attention mask that prevents information flow from future token positions.
KV Cache (Key-Value Cache)
A GPU memory buffer storing computed Key and Value tensors across generation steps to enable O(1) single-token rollout.
Chinchilla Optimal Frontier
The empirical compute-optimal scaling balance dictating ~20 training tokens per model parameter.
Temperature
A hyperparameter scaling logit magnitudes prior to softmax to modulate output entropy and randomness.
Top-p (Nucleus) Sampling
A decoding strategy restricting token sampling to the minimal cumulative probability mass exceeding threshold p.
Top-k Sampling
A decoding strategy restricting token sampling strictly to the k highest-probability candidates.
In-Context Learning
The emergent ability of LLMs to execute new tasks from natural language prompts without gradient updates.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.