Deep Learning › Transformers › Day 227
Hands-on lab — Day 227: The Transformer Architecture
- ← Back to the Day 227 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-227-the-transformer-architecture/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/the_transformer_architecture_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/test_the_transformer_architecture_lib.py examples/the_transformer_architecture_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/test_the_transformer_architecture_lib.py starter/the_transformer_architecture_lib.py tests/run_tests.sh tests/test_the_transformer_architecture_lib.py troubleshooting.md
Lab README
Lab: Day 227 -- The Transformer Architecture
Lesson
Day number: 227 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Full Transformer Architecture and Pre-LN Blocks.
Purpose
Build and test the complete TransformerEncoderLayer featuring Pre-LN Layer Normalization, Multi-Head Self-Attention, Position-Wise Feedforward Networks (FFN) with 4x dimension expansion and GELU activations, and unscaled identity residual streams in PyTorch.
Learning objectives
- Implement the Pre-LN Transformer block architecture.
- Construct the Position-Wise Feedforward Network with GELU activation and 4x hidden expansion.
- Verify uninhibited identity residual gradient flow across stacked layers.
- Verify dimensional consistency across multi-layer Transformer pipelines.
Prerequisites
- Day 226 (Self-Attention, Step by Step).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/the_transformer_architecture_lib.py: Student scaffold file.examples/the_transformer_architecture_lib.py: Complete reference implementation.tests/test_the_transformer_architecture_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/the_transformer_architecture_lib.py
What the commands do
- Executes forward passes through the Pre-LN Transformer encoder layer.
- Tests residual gradient flow backpropagation.
- Runs unit test assertions.
Expected output
Transformer Block Demo: Output Shape = torch.Size([2, 6, 32])
Validation steps
- Verify
TransformerEncoderLayer.forwardoutputs a tensor matching(batch_size, seq_len, d_model). - Confirm input tensor gradients are non-zero after backpropagation.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Residual connection dimension error: Ensure
d_modelremains constant across all attention projections and FFN output layers.
Security notes
All neural computations execute in local process memory.
Extension exercises
- Implement a SwiGLU gated feedforward layer.
- Implement RMSNorm as a replacement for LayerNorm.
Navigation
- Lesson title: The Transformer Architecture
- Day number: 227 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-227-the-transformer-architecture
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-227-the-transformer-architecturewhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 227
- `Output Shape`: Tensor shape of the transformed representation vectors.
examples-run.txt
Transformer Block Demo: Output Shape = torch.Size([2, 6, 32])
measured-values.txt
Output Shape: torch.Size([2, 6, 32])
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_the_transformer_architecture_lib.py::test_transformer_encoder_layer_shapes PASSED [ 50%]
tests/test_the_transformer_architecture_lib.py::test_residual_gradient_flow PASSED [100%]
============================== 2 passed in 0.40s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/test_the_transformer_architecture_lib.py (647 bytes)
import pytest
import torch
from examples.the_transformer_architecture_lib import TransformerEncoderLayer
def test_transformer_encoder_layer_shapes():
torch.manual_seed(42)
layer = TransformerEncoderLayer(d_model=32, num_heads=4, d_ffn=128)
x = torch.randn(2, 5, 32)
out = layer(x)
assert out.shape == (2, 5, 32)
def test_residual_gradient_flow():
torch.manual_seed(42)
layer = TransformerEncoderLayer(d_model=16, num_heads=2, d_ffn=64)
x = torch.randn(2, 4, 16, requires_grad=True)
out = layer(x)
loss = out.sum()
loss.backward()
assert x.grad is not None
assert (x.grad.abs() > 0).all()
examples/the_transformer_architecture_lib.py (1521 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional
class TransformerEncoderLayer(nn.Module):
def __init__(self, d_model: int, num_heads: int, d_ffn: Optional[int] = None, dropout: float = 0.1):
super().__init__()
if d_ffn is None:
d_ffn = 4 * d_model
self.norm1 = nn.LayerNorm(d_model)
self.self_attn = nn.MultiheadAttention(embed_dim=d_model, num_heads=num_heads, dropout=dropout, batch_first=True)
self.dropout1 = nn.Dropout(dropout)
self.norm2 = nn.LayerNorm(d_model)
self.ffn = nn.Sequential(
nn.Linear(d_model, d_ffn),
nn.GELU(),
nn.Dropout(dropout),
nn.Linear(d_ffn, d_model),
nn.Dropout(dropout)
)
def forward(self, x: torch.Tensor, mask: Optional[torch.Tensor] = None) -> torch.Tensor:
# Pre-LN Block 1: Attention
normed_x = self.norm1(x)
attn_out, _ = self.self_attn(query=normed_x, key=normed_x, value=normed_x, key_padding_mask=mask)
x = x + self.dropout1(attn_out)
# Pre-LN Block 2: FFN
x = x + self.ffn(self.norm2(x))
return x
def run_transformer_block_demo():
torch.manual_seed(42)
layer = TransformerEncoderLayer(d_model=32, num_heads=4, d_ffn=128)
x = torch.randn(2, 6, 32)
out = layer(x)
print(f"Transformer Block Demo: Output Shape = {out.shape}")
return out
if __name__ == "__main__":
run_transformer_block_demo()
metadata.yml (441 bytes)
lesson_id: D227
day: 227
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/the_transformer_architecture_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_the_transformer_architecture_lib.py (647 bytes)
import pytest
import torch
from examples.the_transformer_architecture_lib import TransformerEncoderLayer
def test_transformer_encoder_layer_shapes():
torch.manual_seed(42)
layer = TransformerEncoderLayer(d_model=32, num_heads=4, d_ffn=128)
x = torch.randn(2, 5, 32)
out = layer(x)
assert out.shape == (2, 5, 32)
def test_residual_gradient_flow():
torch.manual_seed(42)
layer = TransformerEncoderLayer(d_model=16, num_heads=2, d_ffn=64)
x = torch.randn(2, 4, 16, requires_grad=True)
out = layer(x)
loss = out.sum()
loss.backward()
assert x.grad is not None
assert (x.grad.abs() > 0).all()
starter/the_transformer_architecture_lib.py (569 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional
class TransformerEncoderLayer(nn.Module):
def __init__(self, d_model: int, num_heads: int, d_ffn: Optional[int] = None, dropout: float = 0.1):
super().__init__()
# TODO: Initialize Pre-LN norms, MultiheadAttention, and FFN with 4x expansion
pass
def forward(self, x: torch.Tensor, mask: Optional[torch.Tensor] = None) -> torch.Tensor:
# TODO: Implement Pre-LN attention block and Pre-LN FFN block with residual streams
pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 227 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_the_transformer_architecture_lib.py (647 bytes)
import pytest
import torch
from examples.the_transformer_architecture_lib import TransformerEncoderLayer
def test_transformer_encoder_layer_shapes():
torch.manual_seed(42)
layer = TransformerEncoderLayer(d_model=32, num_heads=4, d_ffn=128)
x = torch.randn(2, 5, 32)
out = layer(x)
assert out.shape == (2, 5, 32)
def test_residual_gradient_flow():
torch.manual_seed(42)
layer = TransformerEncoderLayer(d_model=16, num_heads=2, d_ffn=64)
x = torch.randn(2, 4, 16, requires_grad=True)
out = layer(x)
loss = out.sum()
loss.backward()
assert x.grad is not None
assert (x.grad.abs() > 0).all()
Troubleshooting
Troubleshooting: Day 227 - The Transformer Architecture
Common Issues
- Vanishing gradients in deep stacks:
- Cause: LayerNorm applied to residual sum instead of input branch (Post-LN).
- Fix: Use Pre-LN
x = x + SubLayer(LayerNorm(x)).
Security notes
Security & Privacy: Day 227 - The Transformer Architecture
Security Guidance
- All transformer block operations execute in local memory.