Deep Learning › Transformers › Day 225

Hands-on lab — Day 225: “Attention Is All You Need”

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/attention_is_all_you_need_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/attention_is_all_you_need_lib.py
examples/test_attention_is_all_you_need_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/attention_is_all_you_need_lib.py
starter/test_attention_is_all_you_need_lib.py
tests/run_tests.sh
tests/test_attention_is_all_you_need_lib.py
troubleshooting.md

Lab README

Lab: Day 225 -- “Attention Is All You Need”

Lesson

Day number: 225 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Scaled Dot-Product Attention from First Principles.

Purpose

Build and test the foundational Scaled Dot-Product Attention mechanism from first principles in PyTorch. Implement batched matrix multiplication, 1/sqrt(d_k) variance scaling, causal and padding masking, softmax normalization, and value aggregation.

Learning objectives

  • Implement vectorized Scaled Dot-Product Attention in PyTorch.
  • Apply 1/sqrt(d_k) scaling to stabilize dot-product variance to 1.0.
  • Apply causal lower-triangular masks to prevent look-ahead information leakage.
  • Verify softmax probability distributions and output dimensions across multi-dimensional batch tensors.

Prerequisites

  • Day 224 (A Sentiment Analysis Project).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/attention_is_all_you_need_lib.py: Student scaffold file.
  • examples/attention_is_all_you_need_lib.py: Complete reference implementation.
  • tests/test_attention_is_all_you_need_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/attention_is_all_you_need_lib.py

What the commands do

  • Executes Scaled Dot-Product Attention over 4D Query, Key, and Value tensors.
  • Verifies causal masking and probability distribution rows.
  • Runs unit test assertions.

Expected output

Scaled Attention Demo: Output Shape = torch.Size([2, 4, 6, 16]), Weights Shape = torch.Size([2, 4, 6, 6])

Validation steps

  1. Verify ScaledDotProductAttention.forward outputs a tensor matching (..., Seq_Q, d_v).
  2. Confirm attn_weights sum to 1.0 along the last dimension.
  3. Ensure upper-triangular masked positions evaluate to 0.0.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Matrix dimension mismatch: Use k.transpose(-2, -1) to transpose only the last two dimensions of key tensor $K$.

Security notes

All attention matrix multiplications execute in local memory.

Extension exercises

  1. Implement Multi-Query Attention (MQA) sharing Key/Value heads.
  2. Implement an online tiled attention kernel.
  • Lesson title: “Attention Is All You Need”
  • Day number: 225 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-225-attention-is-all-you-need
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-225-attention-is-all-you-need when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 225

- `Output Shape`: Tensor shape of the aggregated value representations.
- `Weights Shape`: Tensor shape of the calculated attention probability matrix.

examples-run.txt

Scaled Attention Demo: Output Shape = torch.Size([2, 4, 6, 16]), Weights Shape = torch.Size([2, 4, 6, 6])

measured-values.txt

Output Shape: torch.Size([2, 4, 6, 16])
Weights Shape: torch.Size([2, 4, 6, 6])

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_attention_is_all_you_need_lib.py::test_scaled_dot_product_attention_shapes PASSED [ 50%]
tests/test_attention_is_all_you_need_lib.py::test_causal_masking PASSED                       [100%]

============================== 2 passed in 0.39s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/attention_is_all_you_need_lib.py (1330 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from typing import Tuple, Optional

class ScaledDotProductAttention(nn.Module):
    def __init__(self, dropout: float = 0.0):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
                mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
        # q: (..., Seq_Q, d_k)
        # k: (..., Seq_K, d_k)
        # v: (..., Seq_K, d_v)
        d_k = q.size(-1)
        scale = 1.0 / math.sqrt(d_k)

        scores = torch.matmul(q, k.transpose(-2, -1)) * scale

        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)

        attn_weights = F.softmax(scores, dim=-1)
        attn_weights = self.dropout(attn_weights)

        output = torch.matmul(attn_weights, v)
        return output, attn_weights

def run_attention_demo():
    torch.manual_seed(42)
    attn = ScaledDotProductAttention()
    q = torch.randn(2, 4, 6, 16)
    k = torch.randn(2, 4, 6, 16)
    v = torch.randn(2, 4, 6, 16)

    out, weights = attn(q, k, v)
    print(f"Scaled Attention Demo: Output Shape = {out.shape}, Weights Shape = {weights.shape}")
    return out, weights

if __name__ == "__main__":
    run_attention_demo()
examples/test_attention_is_all_you_need_lib.py (1149 bytes)
import pytest
import torch
from examples.attention_is_all_you_need_lib import ScaledDotProductAttention

def test_scaled_dot_product_attention_shapes():
    torch.manual_seed(42)
    attn = ScaledDotProductAttention()
    q = torch.randn(2, 4, 5, 16)
    k = torch.randn(2, 4, 5, 16)
    v = torch.randn(2, 4, 5, 16)

    out, weights = attn(q, k, v)
    assert out.shape == (2, 4, 5, 16)
    assert weights.shape == (2, 4, 5, 5)
    
    # Softmax rows must sum to 1.0
    row_sums = weights.sum(dim=-1)
    for val in row_sums.flatten():
        assert pytest.approx(val.item(), 1e-5) == 1.0

def test_causal_masking():
    torch.manual_seed(42)
    attn = ScaledDotProductAttention()
    seq_len = 4
    q = torch.randn(1, 1, seq_len, 8)
    k = torch.randn(1, 1, seq_len, 8)
    v = torch.randn(1, 1, seq_len, 8)

    # Causal lower-triangular mask
    mask = torch.tril(torch.ones(seq_len, seq_len)).unsqueeze(0).unsqueeze(0)
    out, weights = attn(q, k, v, mask=mask)

    # Upper triangular values must be strictly 0.0
    for i in range(seq_len):
        for j in range(i + 1, seq_len):
            assert weights[0, 0, i, j].item() < 1e-5
metadata.yml (438 bytes)
lesson_id: D225
day: 225
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/attention_is_all_you_need_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/attention_is_all_you_need_lib.py (534 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from typing import Tuple, Optional

class ScaledDotProductAttention(nn.Module):
    def __init__(self, dropout: float = 0.0):
        super().__init__()
        # TODO: Initialize dropout
        pass

    def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
                mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
        # TODO: Implement Q @ K.T / sqrt(d_k), apply mask, softmax, and @ V
        pass
starter/test_attention_is_all_you_need_lib.py (1149 bytes)
import pytest
import torch
from examples.attention_is_all_you_need_lib import ScaledDotProductAttention

def test_scaled_dot_product_attention_shapes():
    torch.manual_seed(42)
    attn = ScaledDotProductAttention()
    q = torch.randn(2, 4, 5, 16)
    k = torch.randn(2, 4, 5, 16)
    v = torch.randn(2, 4, 5, 16)

    out, weights = attn(q, k, v)
    assert out.shape == (2, 4, 5, 16)
    assert weights.shape == (2, 4, 5, 5)
    
    # Softmax rows must sum to 1.0
    row_sums = weights.sum(dim=-1)
    for val in row_sums.flatten():
        assert pytest.approx(val.item(), 1e-5) == 1.0

def test_causal_masking():
    torch.manual_seed(42)
    attn = ScaledDotProductAttention()
    seq_len = 4
    q = torch.randn(1, 1, seq_len, 8)
    k = torch.randn(1, 1, seq_len, 8)
    v = torch.randn(1, 1, seq_len, 8)

    # Causal lower-triangular mask
    mask = torch.tril(torch.ones(seq_len, seq_len)).unsqueeze(0).unsqueeze(0)
    out, weights = attn(q, k, v, mask=mask)

    # Upper triangular values must be strictly 0.0
    for i in range(seq_len):
        for j in range(i + 1, seq_len):
            assert weights[0, 0, i, j].item() < 1e-5
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 225 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_attention_is_all_you_need_lib.py (1149 bytes)
import pytest
import torch
from examples.attention_is_all_you_need_lib import ScaledDotProductAttention

def test_scaled_dot_product_attention_shapes():
    torch.manual_seed(42)
    attn = ScaledDotProductAttention()
    q = torch.randn(2, 4, 5, 16)
    k = torch.randn(2, 4, 5, 16)
    v = torch.randn(2, 4, 5, 16)

    out, weights = attn(q, k, v)
    assert out.shape == (2, 4, 5, 16)
    assert weights.shape == (2, 4, 5, 5)
    
    # Softmax rows must sum to 1.0
    row_sums = weights.sum(dim=-1)
    for val in row_sums.flatten():
        assert pytest.approx(val.item(), 1e-5) == 1.0

def test_causal_masking():
    torch.manual_seed(42)
    attn = ScaledDotProductAttention()
    seq_len = 4
    q = torch.randn(1, 1, seq_len, 8)
    k = torch.randn(1, 1, seq_len, 8)
    v = torch.randn(1, 1, seq_len, 8)

    # Causal lower-triangular mask
    mask = torch.tril(torch.ones(seq_len, seq_len)).unsqueeze(0).unsqueeze(0)
    out, weights = attn(q, k, v, mask=mask)

    # Upper triangular values must be strictly 0.0
    for i in range(seq_len):
        for j in range(i + 1, seq_len):
            assert weights[0, 0, i, j].item() < 1e-5

Troubleshooting

Troubleshooting: Day 225 - “Attention Is All You Need”

Common Issues

  1. Dimension error in batched matmul:
    • Cause: Calling .T on 4D tensor.
    • Fix: Use .transpose(-2, -1).

Security notes

Security & Privacy: Day 225 - “Attention Is All You Need”

Security Guidance

  • All attention matrix multiplications execute locally.