Deep Learning › Transformers › Day 226
Hands-on lab — Day 226: Self-Attention, Step by Step
- ← Back to the Day 226 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-226-self-attention-step-by-step/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/self_attention_step_by_step_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/self_attention_step_by_step_lib.py examples/test_self_attention_step_by_step_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/self_attention_step_by_step_lib.py starter/test_self_attention_step_by_step_lib.py tests/run_tests.sh tests/test_self_attention_step_by_step_lib.py troubleshooting.md
Lab README
Lab: Day 226 -- Self-Attention, Step by Step
Lesson
Day number: 226 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Multi-Head Self-Attention and Positional Encodings.
Purpose
Build and test the Multi-Head Attention module and Sinusoidal Positional Encoding generator in PyTorch. Implement subspace projections, parallel scaled dot-product attention across multiple heads, tensor recombination, and output linear transformations.
Learning objectives
- Implement Multi-Head Attention with parallel subspace splitting.
- Generate sinusoidal positional encoding matrices across arbitrary sequence lengths.
- Perform head splitting, transposition, and contiguous tensor reconstruction.
- Verify dimensional compatibility and attention probability normalization.
Prerequisites
- Day 225 (“Attention Is All You Need”).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/self_attention_step_by_step_lib.py: Student scaffold file.examples/self_attention_step_by_step_lib.py: Complete reference implementation.tests/test_self_attention_step_by_step_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/self_attention_step_by_step_lib.py
What the commands do
- Evaluates MultiHeadAttention forward passes over positional-encoded input tensors.
- Verifies output tensor shapes and multi-head probability distributions.
- Runs unit test assertions.
Expected output
MHA Demo: Output Shape = torch.Size([2, 6, 32]), Weights Shape = torch.Size([2, 4, 6, 6])
Validation steps
- Verify
MultiHeadAttention.forwardoutputs a tensor matching(batch_size, seq_len, d_model). - Confirm
attn_weightshas shape(batch_size, num_heads, seq_len, seq_len). - Ensure sinusoidal positional encodings lie strictly in
[-1.0, 1.0]. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Non-contiguous error on view: Call
.contiguous()before calling.view(batch_size, -1, self.d_model).
Security notes
All attention calculations execute locally in process memory.
Extension exercises
- Implement Rotary Position Embeddings (RoPE) in pure PyTorch.
- Implement Grouped-Query Attention (GQA).
Navigation
- Lesson title: Self-Attention, Step by Step
- Day number: 226 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-226-self-attention-step-by-step
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-226-self-attention-step-by-stepwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 226
- `Output Shape`: Tensor shape of the multi-head attention output.
- `Weights Shape`: Tensor shape of the multi-head attention probability distributions.
examples-run.txt
MHA Demo: Output Shape = torch.Size([2, 6, 32]), Weights Shape = torch.Size([2, 4, 6, 6])
measured-values.txt
Output Shape: torch.Size([2, 6, 32])
Weights Shape: torch.Size([2, 4, 6, 6])
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_self_attention_step_by_step_lib.py::test_multi_head_attention_shapes PASSED [ 50%]
tests/test_self_attention_step_by_step_lib.py::test_sinusoidal_positional_encoding PASSED [100%]
============================== 2 passed in 0.41s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/self_attention_step_by_step_lib.py (2496 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from typing import Tuple, Optional
class MultiHeadAttention(nn.Module):
def __init__(self, d_model: int, num_heads: int, dropout: float = 0.1):
super().__init__()
assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model, bias=False)
self.W_k = nn.Linear(d_model, d_model, bias=False)
self.W_v = nn.Linear(d_model, d_model, bias=False)
self.W_o = nn.Linear(d_model, d_model, bias=False)
self.dropout = nn.Dropout(p=dropout)
def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
batch_size = q.size(0)
Q = self.W_q(q).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
K = self.W_k(k).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
V = self.W_v(v).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
attn_weights = F.softmax(scores, dim=-1)
attn_weights = self.dropout(attn_weights)
context = torch.matmul(attn_weights, V)
context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)
output = self.W_o(context)
return output, attn_weights
def get_sinusoidal_positional_encoding(seq_len: int, d_model: int) -> torch.Tensor:
pe = torch.zeros(seq_len, d_model)
position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
return pe.unsqueeze(0)
def run_mha_demo():
torch.manual_seed(42)
mha = MultiHeadAttention(d_model=32, num_heads=4)
x = torch.randn(2, 6, 32)
pe = get_sinusoidal_positional_encoding(seq_len=6, d_model=32)
x_pos = x + pe
out, weights = mha(x_pos, x_pos, x_pos)
print(f"MHA Demo: Output Shape = {out.shape}, Weights Shape = {weights.shape}")
return out, weights
if __name__ == "__main__":
run_mha_demo()
examples/test_self_attention_step_by_step_lib.py (641 bytes)
import pytest
import torch
from examples.self_attention_step_by_step_lib import MultiHeadAttention, get_sinusoidal_positional_encoding
def test_multi_head_attention_shapes():
torch.manual_seed(42)
mha = MultiHeadAttention(d_model=32, num_heads=4)
x = torch.randn(2, 5, 32)
out, weights = mha(x, x, x)
assert out.shape == (2, 5, 32)
assert weights.shape == (2, 4, 5, 5)
def test_sinusoidal_positional_encoding():
pe = get_sinusoidal_positional_encoding(seq_len=8, d_model=16)
assert pe.shape == (1, 8, 16)
# Values should lie between -1.0 and 1.0
assert (pe >= -1.0).all() and (pe <= 1.0).all()
metadata.yml (440 bytes)
lesson_id: D226
day: 226
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/self_attention_step_by_step_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/self_attention_step_by_step_lib.py (785 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from typing import Tuple, Optional
class MultiHeadAttention(nn.Module):
def __init__(self, d_model: int, num_heads: int, dropout: float = 0.1):
super().__init__()
# TODO: Initialize projections for W_q, W_k, W_v, W_o
pass
def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
# TODO: Implement multi-head linear projections, scaled attention, and concatenation
pass
def get_sinusoidal_positional_encoding(seq_len: int, d_model: int) -> torch.Tensor:
# TODO: Generate sinusoidal positional encoding matrix of shape (1, seq_len, d_model)
pass
starter/test_self_attention_step_by_step_lib.py (641 bytes)
import pytest
import torch
from examples.self_attention_step_by_step_lib import MultiHeadAttention, get_sinusoidal_positional_encoding
def test_multi_head_attention_shapes():
torch.manual_seed(42)
mha = MultiHeadAttention(d_model=32, num_heads=4)
x = torch.randn(2, 5, 32)
out, weights = mha(x, x, x)
assert out.shape == (2, 5, 32)
assert weights.shape == (2, 4, 5, 5)
def test_sinusoidal_positional_encoding():
pe = get_sinusoidal_positional_encoding(seq_len=8, d_model=16)
assert pe.shape == (1, 8, 16)
# Values should lie between -1.0 and 1.0
assert (pe >= -1.0).all() and (pe <= 1.0).all()
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 226 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_self_attention_step_by_step_lib.py (641 bytes)
import pytest
import torch
from examples.self_attention_step_by_step_lib import MultiHeadAttention, get_sinusoidal_positional_encoding
def test_multi_head_attention_shapes():
torch.manual_seed(42)
mha = MultiHeadAttention(d_model=32, num_heads=4)
x = torch.randn(2, 5, 32)
out, weights = mha(x, x, x)
assert out.shape == (2, 5, 32)
assert weights.shape == (2, 4, 5, 5)
def test_sinusoidal_positional_encoding():
pe = get_sinusoidal_positional_encoding(seq_len=8, d_model=16)
assert pe.shape == (1, 8, 16)
# Values should lie between -1.0 and 1.0
assert (pe >= -1.0).all() and (pe <= 1.0).all()
Troubleshooting
Troubleshooting: Day 226 - Self-Attention, Step by Step
Common Issues
- Stride mismatch error in tensor reshape:
- Cause: Reshaping transposed tensor directly.
- Fix: Use
.contiguous().view(...).
Security notes
Security & Privacy: Day 226 - Self-Attention, Step by Step
Security Guidance
- All multi-head attention operations execute in local memory.