Deep Learning › Transformers › Day 226

Hands-on lab — Day 226: Self-Attention, Step by Step

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/self_attention_step_by_step_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/self_attention_step_by_step_lib.py
examples/test_self_attention_step_by_step_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/self_attention_step_by_step_lib.py
starter/test_self_attention_step_by_step_lib.py
tests/run_tests.sh
tests/test_self_attention_step_by_step_lib.py
troubleshooting.md

Lab README

Lab: Day 226 -- Self-Attention, Step by Step

Lesson

Day number: 226 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Multi-Head Self-Attention and Positional Encodings.

Purpose

Build and test the Multi-Head Attention module and Sinusoidal Positional Encoding generator in PyTorch. Implement subspace projections, parallel scaled dot-product attention across multiple heads, tensor recombination, and output linear transformations.

Learning objectives

  • Implement Multi-Head Attention with parallel subspace splitting.
  • Generate sinusoidal positional encoding matrices across arbitrary sequence lengths.
  • Perform head splitting, transposition, and contiguous tensor reconstruction.
  • Verify dimensional compatibility and attention probability normalization.

Prerequisites

  • Day 225 (“Attention Is All You Need”).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/self_attention_step_by_step_lib.py: Student scaffold file.
  • examples/self_attention_step_by_step_lib.py: Complete reference implementation.
  • tests/test_self_attention_step_by_step_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/self_attention_step_by_step_lib.py

What the commands do

  • Evaluates MultiHeadAttention forward passes over positional-encoded input tensors.
  • Verifies output tensor shapes and multi-head probability distributions.
  • Runs unit test assertions.

Expected output

MHA Demo: Output Shape = torch.Size([2, 6, 32]), Weights Shape = torch.Size([2, 4, 6, 6])

Validation steps

  1. Verify MultiHeadAttention.forward outputs a tensor matching (batch_size, seq_len, d_model).
  2. Confirm attn_weights has shape (batch_size, num_heads, seq_len, seq_len).
  3. Ensure sinusoidal positional encodings lie strictly in [-1.0, 1.0].
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Non-contiguous error on view: Call .contiguous() before calling .view(batch_size, -1, self.d_model).

Security notes

All attention calculations execute locally in process memory.

Extension exercises

  1. Implement Rotary Position Embeddings (RoPE) in pure PyTorch.
  2. Implement Grouped-Query Attention (GQA).

Expected output

FIELDS.md

# Expected Output Fields: Day 226

- `Output Shape`: Tensor shape of the multi-head attention output.
- `Weights Shape`: Tensor shape of the multi-head attention probability distributions.

examples-run.txt

MHA Demo: Output Shape = torch.Size([2, 6, 32]), Weights Shape = torch.Size([2, 4, 6, 6])

measured-values.txt

Output Shape: torch.Size([2, 6, 32])
Weights Shape: torch.Size([2, 4, 6, 6])

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_self_attention_step_by_step_lib.py::test_multi_head_attention_shapes PASSED [ 50%]
tests/test_self_attention_step_by_step_lib.py::test_sinusoidal_positional_encoding PASSED [100%]

============================== 2 passed in 0.41s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/self_attention_step_by_step_lib.py (2496 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from typing import Tuple, Optional

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, num_heads: int, dropout: float = 0.1):
        super().__init__()
        assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        self.W_q = nn.Linear(d_model, d_model, bias=False)
        self.W_k = nn.Linear(d_model, d_model, bias=False)
        self.W_v = nn.Linear(d_model, d_model, bias=False)
        self.W_o = nn.Linear(d_model, d_model, bias=False)
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
                mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
        batch_size = q.size(0)

        Q = self.W_q(q).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(k).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(v).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)

        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)

        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)

        attn_weights = F.softmax(scores, dim=-1)
        attn_weights = self.dropout(attn_weights)

        context = torch.matmul(attn_weights, V)
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)
        output = self.W_o(context)

        return output, attn_weights

def get_sinusoidal_positional_encoding(seq_len: int, d_model: int) -> torch.Tensor:
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe.unsqueeze(0)

def run_mha_demo():
    torch.manual_seed(42)
    mha = MultiHeadAttention(d_model=32, num_heads=4)
    x = torch.randn(2, 6, 32)
    pe = get_sinusoidal_positional_encoding(seq_len=6, d_model=32)
    x_pos = x + pe

    out, weights = mha(x_pos, x_pos, x_pos)
    print(f"MHA Demo: Output Shape = {out.shape}, Weights Shape = {weights.shape}")
    return out, weights

if __name__ == "__main__":
    run_mha_demo()
examples/test_self_attention_step_by_step_lib.py (641 bytes)
import pytest
import torch
from examples.self_attention_step_by_step_lib import MultiHeadAttention, get_sinusoidal_positional_encoding

def test_multi_head_attention_shapes():
    torch.manual_seed(42)
    mha = MultiHeadAttention(d_model=32, num_heads=4)
    x = torch.randn(2, 5, 32)
    out, weights = mha(x, x, x)
    
    assert out.shape == (2, 5, 32)
    assert weights.shape == (2, 4, 5, 5)

def test_sinusoidal_positional_encoding():
    pe = get_sinusoidal_positional_encoding(seq_len=8, d_model=16)
    assert pe.shape == (1, 8, 16)
    # Values should lie between -1.0 and 1.0
    assert (pe >= -1.0).all() and (pe <= 1.0).all()
metadata.yml (440 bytes)
lesson_id: D226
day: 226
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/self_attention_step_by_step_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/self_attention_step_by_step_lib.py (785 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from typing import Tuple, Optional

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, num_heads: int, dropout: float = 0.1):
        super().__init__()
        # TODO: Initialize projections for W_q, W_k, W_v, W_o
        pass

    def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
                mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
        # TODO: Implement multi-head linear projections, scaled attention, and concatenation
        pass

def get_sinusoidal_positional_encoding(seq_len: int, d_model: int) -> torch.Tensor:
    # TODO: Generate sinusoidal positional encoding matrix of shape (1, seq_len, d_model)
    pass
starter/test_self_attention_step_by_step_lib.py (641 bytes)
import pytest
import torch
from examples.self_attention_step_by_step_lib import MultiHeadAttention, get_sinusoidal_positional_encoding

def test_multi_head_attention_shapes():
    torch.manual_seed(42)
    mha = MultiHeadAttention(d_model=32, num_heads=4)
    x = torch.randn(2, 5, 32)
    out, weights = mha(x, x, x)
    
    assert out.shape == (2, 5, 32)
    assert weights.shape == (2, 4, 5, 5)

def test_sinusoidal_positional_encoding():
    pe = get_sinusoidal_positional_encoding(seq_len=8, d_model=16)
    assert pe.shape == (1, 8, 16)
    # Values should lie between -1.0 and 1.0
    assert (pe >= -1.0).all() and (pe <= 1.0).all()
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 226 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_self_attention_step_by_step_lib.py (641 bytes)
import pytest
import torch
from examples.self_attention_step_by_step_lib import MultiHeadAttention, get_sinusoidal_positional_encoding

def test_multi_head_attention_shapes():
    torch.manual_seed(42)
    mha = MultiHeadAttention(d_model=32, num_heads=4)
    x = torch.randn(2, 5, 32)
    out, weights = mha(x, x, x)
    
    assert out.shape == (2, 5, 32)
    assert weights.shape == (2, 4, 5, 5)

def test_sinusoidal_positional_encoding():
    pe = get_sinusoidal_positional_encoding(seq_len=8, d_model=16)
    assert pe.shape == (1, 8, 16)
    # Values should lie between -1.0 and 1.0
    assert (pe >= -1.0).all() and (pe <= 1.0).all()

Troubleshooting

Troubleshooting: Day 226 - Self-Attention, Step by Step

Common Issues

  1. Stride mismatch error in tensor reshape:
    • Cause: Reshaping transposed tensor directly.
    • Fix: Use .contiguous().view(...).

Security notes

Security & Privacy: Day 226 - Self-Attention, Step by Step

Security Guidance

  • All multi-head attention operations execute in local memory.