Deep Learning › Transformers › Day 229
Hands-on lab — Day 229: Decoder Models: The GPT Family
- ← Back to the Day 229 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-229-decoder-models-the-gpt-family/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/decoder_models_the_gpt_family_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/decoder_models_the_gpt_family_lib.py examples/test_decoder_models_the_gpt_family_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/decoder_models_the_gpt_family_lib.py starter/test_decoder_models_the_gpt_family_lib.py tests/run_tests.sh tests/test_decoder_models_the_gpt_family_lib.py troubleshooting.md
Lab README
Lab: Day 229 -- Decoder Models: The GPT Family
Lesson
Day number: 229 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: GPT Decoder Architecture and Autoregressive Generation.
Purpose
Build and test the MiniatureGPT causal language model architecture in PyTorch. Implement causal lower-triangular masking, construct the autoregressive decoder pipeline with Pre-LN residual streams, implement the temperature and top-k token generation rollout loop, and test autoregressive token generation.
Learning objectives
- Implement the GPT causal autoregressive Transformer decoder architecture.
- Construct the Causal Attention Mask to prevent future token information leakage.
- Implement the autoregressive generation loop with temperature-scaled multinomial sampling.
- Verify dimensional compatibility and generation sequence rollout length.
Prerequisites
- Day 228 (Encoder Models: BERT and Friends).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/decoder_models_the_gpt_family_lib.py: Student scaffold file.examples/decoder_models_the_gpt_family_lib.py: Complete reference implementation.tests/test_decoder_models_the_gpt_family_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/decoder_models_the_gpt_family_lib.py
What the commands do
- Executes forward passes through the GPT causal model.
- Evaluates autoregressive generation rollouts.
- Runs unit test assertions.
Expected output
GPT Demo: Prompt Len = 4, Generated Len = 8
Validation steps
- Verify
MiniatureGPT.forwardoutputs a tensor of shape(batch_size, seq_len, vocab_size). - Confirm
MiniatureGPT.generatereturns a tensor of shape(batch_size, prompt_len + max_new_tokens). - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Causal mask dimension error: Ensure the square subsequent mask matches
seq_len = x.size(1)on the matching device.
Security notes
All neural computations execute in local process memory.
Extension exercises
- Implement Top-p (Nucleus) sampling.
- Implement an explicit KV-Cache module.
Navigation
- Lesson title: Decoder Models: The GPT Family
- Day number: 229 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-229-decoder-models-the-gpt-family
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-229-decoder-models-the-gpt-familywhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 229
- `Prompt Len`: Length of initial input prompt in tokens.
- `Generated Len`: Total sequence length after autoregressive rollout.
examples-run.txt
GPT Demo: Prompt Len = 4, Generated Len = 8
measured-values.txt
Prompt Len: 4
Generated Len: 8
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_decoder_models_the_gpt_family_lib.py::test_miniature_gpt_forward PASSED [ 50%]
tests/test_decoder_models_the_gpt_family_lib.py::test_autoregressive_generation PASSED [100%]
============================== 2 passed in 0.44s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/decoder_models_the_gpt_family_lib.py (2855 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional
class GPTDecoderLayer(nn.Module):
def __init__(self, d_model: int, num_heads: int):
super().__init__()
self.norm1 = nn.LayerNorm(d_model)
self.self_attn = nn.MultiheadAttention(d_model, num_heads, batch_first=True)
self.norm2 = nn.LayerNorm(d_model)
self.ffn = nn.Sequential(
nn.Linear(d_model, 4 * d_model),
nn.GELU(),
nn.Linear(4 * d_model, d_model)
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
seq_len = x.size(1)
causal_mask = nn.Transformer.generate_square_subsequent_mask(seq_len).to(x.device)
normed = self.norm1(x)
attn_out, _ = self.self_attn(normed, normed, normed, attn_mask=causal_mask, is_causal=True)
x = x + attn_out
x = x + self.ffn(self.norm2(x))
return x
class MiniatureGPT(nn.Module):
def __init__(self, vocab_size: int = 200, d_model: int = 32,
num_layers: int = 2, num_heads: int = 2):
super().__init__()
self.token_embeddings = nn.Embedding(vocab_size, d_model)
self.pos_embeddings = nn.Embedding(128, d_model)
self.layers = nn.ModuleList([
GPTDecoderLayer(d_model, num_heads) for _ in range(num_layers)
])
self.final_norm = nn.LayerNorm(d_model)
self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
self.lm_head.weight = self.token_embeddings.weight
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
seq_len = input_ids.size(1)
pos = torch.arange(seq_len, device=input_ids.device).unsqueeze(0)
x = self.token_embeddings(input_ids) + self.pos_embeddings(pos)
for layer in self.layers:
x = layer(x)
x = self.final_norm(x)
logits = self.lm_head(x)
return logits
@torch.no_grad()
def generate(self, prompt_ids: torch.Tensor, max_new_tokens: int = 5,
temperature: float = 1.0) -> torch.Tensor:
current_ids = prompt_ids
for _ in range(max_new_tokens):
logits = self.forward(current_ids)[:, -1, :]
logits = logits / temperature
probs = F.softmax(logits, dim=-1)
next_token = torch.multinomial(probs, num_samples=1)
current_ids = torch.cat([current_ids, next_token], dim=1)
return current_ids
def run_gpt_demo():
torch.manual_seed(42)
model = MiniatureGPT(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
prompt = torch.randint(1, 100, (1, 4))
gen = model.generate(prompt, max_new_tokens=4, temperature=0.8)
print(f"GPT Demo: Prompt Len = {prompt.size(1)}, Generated Len = {gen.size(1)}")
return gen
if __name__ == "__main__":
run_gpt_demo()
examples/test_decoder_models_the_gpt_family_lib.py (635 bytes)
import pytest
import torch
from examples.decoder_models_the_gpt_family_lib import MiniatureGPT
def test_miniature_gpt_forward():
torch.manual_seed(42)
model = MiniatureGPT(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
x = torch.randint(0, 100, (2, 6))
logits = model(x)
assert logits.shape == (2, 6, 100)
def test_autoregressive_generation():
torch.manual_seed(42)
model = MiniatureGPT(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
prompt = torch.randint(0, 100, (1, 3))
out = model.generate(prompt, max_new_tokens=4)
assert out.shape == (1, 7) # 3 prompt + 4 new = 7 tokens
metadata.yml (442 bytes)
lesson_id: D229
day: 229
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/decoder_models_the_gpt_family_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/decoder_models_the_gpt_family_lib.py (1112 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional
class GPTDecoderLayer(nn.Module):
def __init__(self, d_model: int, num_heads: int):
super().__init__()
# TODO: Initialize Pre-LN layers, causal MultiheadAttention, and FFN
pass
def forward(self, x: torch.Tensor) -> torch.Tensor:
# TODO: Implement causal transformer block with residual streams
pass
class MiniatureGPT(nn.Module):
def __init__(self, vocab_size: int = 200, d_model: int = 32,
num_layers: int = 2, num_heads: int = 2):
super().__init__()
# TODO: Initialize embeddings, decoder stack, and tied language model head
pass
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
# TODO: Pass input through causal decoder and compute vocabulary logits
pass
@torch.no_grad()
def generate(self, prompt_ids: torch.Tensor, max_new_tokens: int = 5,
temperature: float = 1.0) -> torch.Tensor:
# TODO: Autoregressive token-by-token generation loop
pass
starter/test_decoder_models_the_gpt_family_lib.py (635 bytes)
import pytest
import torch
from examples.decoder_models_the_gpt_family_lib import MiniatureGPT
def test_miniature_gpt_forward():
torch.manual_seed(42)
model = MiniatureGPT(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
x = torch.randint(0, 100, (2, 6))
logits = model(x)
assert logits.shape == (2, 6, 100)
def test_autoregressive_generation():
torch.manual_seed(42)
model = MiniatureGPT(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
prompt = torch.randint(0, 100, (1, 3))
out = model.generate(prompt, max_new_tokens=4)
assert out.shape == (1, 7) # 3 prompt + 4 new = 7 tokens
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 229 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_decoder_models_the_gpt_family_lib.py (635 bytes)
import pytest
import torch
from examples.decoder_models_the_gpt_family_lib import MiniatureGPT
def test_miniature_gpt_forward():
torch.manual_seed(42)
model = MiniatureGPT(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
x = torch.randint(0, 100, (2, 6))
logits = model(x)
assert logits.shape == (2, 6, 100)
def test_autoregressive_generation():
torch.manual_seed(42)
model = MiniatureGPT(vocab_size=100, d_model=16, num_layers=2, num_heads=2)
prompt = torch.randint(0, 100, (1, 3))
out = model.generate(prompt, max_new_tokens=4)
assert out.shape == (1, 7) # 3 prompt + 4 new = 7 tokens
Troubleshooting
Troubleshooting: Day 229 - Decoder Models: The GPT Family
Common Issues
- Causal mask device mismatch:
- Cause: Mask created on CPU while model runs on CUDA/MPS.
- Fix: Use
causal_mask.to(x.device).
Security notes
Security & Privacy: Day 229 - Decoder Models: The GPT Family
Security Guidance
- All generative decoder model computations execute locally in memory.