Deep Learning › Sequences and Text › Day 222

Hands-on lab — Day 222: Sequence-to-Sequence and Early Attention

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/sequence_to_sequence_and_early_attention_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/sequence_to_sequence_and_early_attention_lib.py
examples/test_sequence_to_sequence_and_early_attention_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/sequence_to_sequence_and_early_attention_lib.py
starter/test_sequence_to_sequence_and_early_attention_lib.py
tests/run_tests.sh
tests/test_sequence_to_sequence_and_early_attention_lib.py
troubleshooting.md

Lab README

Lab: Day 222 -- Sequence-to-Sequence and Early Attention

Lesson

Day number: 222 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Sequence-to-Sequence Models and Early Additive Attention.

Purpose

Build and test the Bahdanau Additive Attention mechanism from first principles in PyTorch. Implement query-key alignment scoring, compute dynamic context vectors over encoder hidden states, apply padding masks, and verify attention probability distributions.

Learning objectives

  • Implement Bahdanau Additive Attention linear projections and tanh scoring.
  • Apply softmax normalization and batch matrix multiplication for dynamic context vector creation.
  • Enforce padding masks to eliminate attention leakage onto padding tokens.
  • Validate attention alignment properties across variable-length sequence batches.

Prerequisites

  • Day 221 (LSTMs and GRUs).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/sequence_to_sequence_and_early_attention_lib.py: Student scaffold file.
  • examples/sequence_to_sequence_and_early_attention_lib.py: Complete reference implementation.
  • tests/test_sequence_to_sequence_and_early_attention_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/sequence_to_sequence_and_early_attention_lib.py

What the commands do

  • Evaluates Bahdanau attention forward pass over query and key tensors.
  • Applies padding masks and computes dynamic context vectors.
  • Runs unit test assertions.

Expected output

Attention Demo: Context Shape = torch.Size([2, 8]), Weight Sums = [1.0, 1.0]

Validation steps

  1. Verify BahdanauAttention.forward outputs a context vector of shape (batch_size, enc_dim).
  2. Confirm attn_weights sum to 1.0 along the sequence dimension.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Dimensions mismatch in bmm: Ensure query is unsqueezed along dimension 1 and weights are unsqueezed along dimension 1 before torch.bmm.

Security notes

All attention matrix operations execute in local process memory.

Extension exercises

  1. Implement Luong multiplicative dot-product attention scoring.
  2. Build an attention heatmap visualizer using Matplotlib.

Expected output

FIELDS.md

# Expected Output Fields: Day 222

- `Context Shape`: Shape of the calculated dynamic context vector.
- `Weight Sums`: Sum of attention probability weights across sequence length.

examples-run.txt

Attention Demo: Context Shape = torch.Size([2, 8]), Weight Sums = [1.0, 1.0]

measured-values.txt

Context Shape: torch.Size([2, 8])
Weight Sums: [1.0, 1.0]

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_sequence_to_sequence_and_early_attention_lib.py::test_attention_dimensions_and_sum PASSED [ 50%]
tests/test_sequence_to_sequence_and_early_attention_lib.py::test_attention_masking PASSED            [100%]

============================== 2 passed in 0.36s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/sequence_to_sequence_and_early_attention_lib.py (1686 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple, Optional

class BahdanauAttention(nn.Module):
    def __init__(self, enc_dim: int, dec_dim: int, attn_dim: int):
        super().__init__()
        self.W_query = nn.Linear(dec_dim, attn_dim, bias=False)
        self.U_keys = nn.Linear(enc_dim, attn_dim, bias=False)
        self.v_att = nn.Linear(attn_dim, 1, bias=False)

    def forward(self, query: torch.Tensor, keys: torch.Tensor,
                mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
        # query: (Batch, dec_dim)
        # keys: (Batch, seq_len, enc_dim)
        proj_query = self.W_query(query).unsqueeze(1) # (Batch, 1, attn_dim)
        proj_keys = self.U_keys(keys) # (Batch, seq_len, attn_dim)

        energy = self.v_att(torch.tanh(proj_query + proj_keys)).squeeze(2) # (Batch, seq_len)

        if mask is not None:
            energy = energy.masked_fill(mask == 0, -1e9)

        attn_weights = F.softmax(energy, dim=1) # (Batch, seq_len)
        context = torch.bmm(attn_weights.unsqueeze(1), keys).squeeze(1) # (Batch, enc_dim)

        return context, attn_weights

def run_attention_demo():
    torch.manual_seed(42)
    attn = BahdanauAttention(enc_dim=8, dec_dim=16, attn_dim=12)
    
    query = torch.randn(2, 16)
    keys = torch.randn(2, 5, 8)
    mask = torch.tensor([[1, 1, 1, 0, 0], [1, 1, 1, 1, 0]])

    context, weights = attn(query, keys, mask=mask)
    weight_sum = weights.sum(dim=1).tolist()

    print(f"Attention Demo: Context Shape = {context.shape}, Weight Sums = {weight_sum}")
    return context, weights

if __name__ == "__main__":
    run_attention_demo()
examples/test_sequence_to_sequence_and_early_attention_lib.py (1005 bytes)
import pytest
import torch
from examples.sequence_to_sequence_and_early_attention_lib import BahdanauAttention

def test_attention_dimensions_and_sum():
    torch.manual_seed(42)
    attn = BahdanauAttention(enc_dim=10, dec_dim=20, attn_dim=15)
    query = torch.randn(4, 20)
    keys = torch.randn(4, 7, 10)

    context, weights = attn(query, keys)
    assert context.shape == (4, 10)
    assert weights.shape == (4, 7)
    
    # Softmax weights must sum to 1.0
    for s in weights.sum(dim=1):
        assert pytest.approx(s.item(), 1e-5) == 1.0

def test_attention_masking():
    torch.manual_seed(42)
    attn = BahdanauAttention(enc_dim=6, dec_dim=12, attn_dim=8)
    query = torch.randn(2, 12)
    keys = torch.randn(2, 4, 6)
    mask = torch.tensor([[1, 1, 0, 0], [1, 1, 1, 0]])

    _, weights = attn(query, keys, mask=mask)
    # Masked positions should have near-zero weights
    assert weights[0, 2].item() < 1e-4
    assert weights[0, 3].item() < 1e-4
    assert weights[1, 3].item() < 1e-4
metadata.yml (453 bytes)
lesson_id: D222
day: 222
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/sequence_to_sequence_and_early_attention_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/sequence_to_sequence_and_early_attention_lib.py (578 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple, Optional

class BahdanauAttention(nn.Module):
    def __init__(self, enc_dim: int, dec_dim: int, attn_dim: int):
        super().__init__()
        # TODO: Initialize query projection, keys projection, and v_att vector
        pass

    def forward(self, query: torch.Tensor, keys: torch.Tensor,
                mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
        # TODO: Compute additive energy, softmax weights, and dynamic context vector
        pass
starter/test_sequence_to_sequence_and_early_attention_lib.py (1005 bytes)
import pytest
import torch
from examples.sequence_to_sequence_and_early_attention_lib import BahdanauAttention

def test_attention_dimensions_and_sum():
    torch.manual_seed(42)
    attn = BahdanauAttention(enc_dim=10, dec_dim=20, attn_dim=15)
    query = torch.randn(4, 20)
    keys = torch.randn(4, 7, 10)

    context, weights = attn(query, keys)
    assert context.shape == (4, 10)
    assert weights.shape == (4, 7)
    
    # Softmax weights must sum to 1.0
    for s in weights.sum(dim=1):
        assert pytest.approx(s.item(), 1e-5) == 1.0

def test_attention_masking():
    torch.manual_seed(42)
    attn = BahdanauAttention(enc_dim=6, dec_dim=12, attn_dim=8)
    query = torch.randn(2, 12)
    keys = torch.randn(2, 4, 6)
    mask = torch.tensor([[1, 1, 0, 0], [1, 1, 1, 0]])

    _, weights = attn(query, keys, mask=mask)
    # Masked positions should have near-zero weights
    assert weights[0, 2].item() < 1e-4
    assert weights[0, 3].item() < 1e-4
    assert weights[1, 3].item() < 1e-4
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 222 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_sequence_to_sequence_and_early_attention_lib.py (1005 bytes)
import pytest
import torch
from examples.sequence_to_sequence_and_early_attention_lib import BahdanauAttention

def test_attention_dimensions_and_sum():
    torch.manual_seed(42)
    attn = BahdanauAttention(enc_dim=10, dec_dim=20, attn_dim=15)
    query = torch.randn(4, 20)
    keys = torch.randn(4, 7, 10)

    context, weights = attn(query, keys)
    assert context.shape == (4, 10)
    assert weights.shape == (4, 7)
    
    # Softmax weights must sum to 1.0
    for s in weights.sum(dim=1):
        assert pytest.approx(s.item(), 1e-5) == 1.0

def test_attention_masking():
    torch.manual_seed(42)
    attn = BahdanauAttention(enc_dim=6, dec_dim=12, attn_dim=8)
    query = torch.randn(2, 12)
    keys = torch.randn(2, 4, 6)
    mask = torch.tensor([[1, 1, 0, 0], [1, 1, 1, 0]])

    _, weights = attn(query, keys, mask=mask)
    # Masked positions should have near-zero weights
    assert weights[0, 2].item() < 1e-4
    assert weights[0, 3].item() < 1e-4
    assert weights[1, 3].item() < 1e-4

Troubleshooting

Troubleshooting: Day 222 - Sequence-to-Sequence and Early Attention

Common Issues

  1. Masked positions receiving non-zero attention:
    • Cause: Mask value used is not negative enough before softmax.
    • Fix: Use energy.masked_fill(mask == 0, -1e9).

Security notes

Security & Privacy: Day 222 - Sequence-to-Sequence and Early Attention

Security Guidance

  • All attention computations execute locally in memory buffers.