Deep Learning › Sequences and Text › Day 222
Hands-on lab — Day 222: Sequence-to-Sequence and Early Attention
- ← Back to the Day 222 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-222-sequence-to-sequence-and-early-attention/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/sequence_to_sequence_and_early_attention_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/sequence_to_sequence_and_early_attention_lib.py examples/test_sequence_to_sequence_and_early_attention_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/sequence_to_sequence_and_early_attention_lib.py starter/test_sequence_to_sequence_and_early_attention_lib.py tests/run_tests.sh tests/test_sequence_to_sequence_and_early_attention_lib.py troubleshooting.md
Lab README
Lab: Day 222 -- Sequence-to-Sequence and Early Attention
Lesson
Day number: 222 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Sequence-to-Sequence Models and Early Additive Attention.
Purpose
Build and test the Bahdanau Additive Attention mechanism from first principles in PyTorch. Implement query-key alignment scoring, compute dynamic context vectors over encoder hidden states, apply padding masks, and verify attention probability distributions.
Learning objectives
- Implement Bahdanau Additive Attention linear projections and tanh scoring.
- Apply softmax normalization and batch matrix multiplication for dynamic context vector creation.
- Enforce padding masks to eliminate attention leakage onto padding tokens.
- Validate attention alignment properties across variable-length sequence batches.
Prerequisites
- Day 221 (LSTMs and GRUs).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/sequence_to_sequence_and_early_attention_lib.py: Student scaffold file.examples/sequence_to_sequence_and_early_attention_lib.py: Complete reference implementation.tests/test_sequence_to_sequence_and_early_attention_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/sequence_to_sequence_and_early_attention_lib.py
What the commands do
- Evaluates Bahdanau attention forward pass over query and key tensors.
- Applies padding masks and computes dynamic context vectors.
- Runs unit test assertions.
Expected output
Attention Demo: Context Shape = torch.Size([2, 8]), Weight Sums = [1.0, 1.0]
Validation steps
- Verify
BahdanauAttention.forwardoutputs a context vector of shape(batch_size, enc_dim). - Confirm
attn_weightssum to 1.0 along the sequence dimension. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Dimensions mismatch in bmm: Ensure query is unsqueezed along dimension 1 and weights are unsqueezed along dimension 1 before
torch.bmm.
Security notes
All attention matrix operations execute in local process memory.
Extension exercises
- Implement Luong multiplicative dot-product attention scoring.
- Build an attention heatmap visualizer using Matplotlib.
Navigation
- Lesson title: Sequence-to-Sequence and Early Attention
- Day number: 222 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-222-sequence-to-sequence-and-early-attention
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-222-sequence-to-sequence-and-early-attentionwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 222
- `Context Shape`: Shape of the calculated dynamic context vector.
- `Weight Sums`: Sum of attention probability weights across sequence length.
examples-run.txt
Attention Demo: Context Shape = torch.Size([2, 8]), Weight Sums = [1.0, 1.0]
measured-values.txt
Context Shape: torch.Size([2, 8])
Weight Sums: [1.0, 1.0]
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_sequence_to_sequence_and_early_attention_lib.py::test_attention_dimensions_and_sum PASSED [ 50%]
tests/test_sequence_to_sequence_and_early_attention_lib.py::test_attention_masking PASSED [100%]
============================== 2 passed in 0.36s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/sequence_to_sequence_and_early_attention_lib.py (1686 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple, Optional
class BahdanauAttention(nn.Module):
def __init__(self, enc_dim: int, dec_dim: int, attn_dim: int):
super().__init__()
self.W_query = nn.Linear(dec_dim, attn_dim, bias=False)
self.U_keys = nn.Linear(enc_dim, attn_dim, bias=False)
self.v_att = nn.Linear(attn_dim, 1, bias=False)
def forward(self, query: torch.Tensor, keys: torch.Tensor,
mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
# query: (Batch, dec_dim)
# keys: (Batch, seq_len, enc_dim)
proj_query = self.W_query(query).unsqueeze(1) # (Batch, 1, attn_dim)
proj_keys = self.U_keys(keys) # (Batch, seq_len, attn_dim)
energy = self.v_att(torch.tanh(proj_query + proj_keys)).squeeze(2) # (Batch, seq_len)
if mask is not None:
energy = energy.masked_fill(mask == 0, -1e9)
attn_weights = F.softmax(energy, dim=1) # (Batch, seq_len)
context = torch.bmm(attn_weights.unsqueeze(1), keys).squeeze(1) # (Batch, enc_dim)
return context, attn_weights
def run_attention_demo():
torch.manual_seed(42)
attn = BahdanauAttention(enc_dim=8, dec_dim=16, attn_dim=12)
query = torch.randn(2, 16)
keys = torch.randn(2, 5, 8)
mask = torch.tensor([[1, 1, 1, 0, 0], [1, 1, 1, 1, 0]])
context, weights = attn(query, keys, mask=mask)
weight_sum = weights.sum(dim=1).tolist()
print(f"Attention Demo: Context Shape = {context.shape}, Weight Sums = {weight_sum}")
return context, weights
if __name__ == "__main__":
run_attention_demo()
examples/test_sequence_to_sequence_and_early_attention_lib.py (1005 bytes)
import pytest
import torch
from examples.sequence_to_sequence_and_early_attention_lib import BahdanauAttention
def test_attention_dimensions_and_sum():
torch.manual_seed(42)
attn = BahdanauAttention(enc_dim=10, dec_dim=20, attn_dim=15)
query = torch.randn(4, 20)
keys = torch.randn(4, 7, 10)
context, weights = attn(query, keys)
assert context.shape == (4, 10)
assert weights.shape == (4, 7)
# Softmax weights must sum to 1.0
for s in weights.sum(dim=1):
assert pytest.approx(s.item(), 1e-5) == 1.0
def test_attention_masking():
torch.manual_seed(42)
attn = BahdanauAttention(enc_dim=6, dec_dim=12, attn_dim=8)
query = torch.randn(2, 12)
keys = torch.randn(2, 4, 6)
mask = torch.tensor([[1, 1, 0, 0], [1, 1, 1, 0]])
_, weights = attn(query, keys, mask=mask)
# Masked positions should have near-zero weights
assert weights[0, 2].item() < 1e-4
assert weights[0, 3].item() < 1e-4
assert weights[1, 3].item() < 1e-4
metadata.yml (453 bytes)
lesson_id: D222
day: 222
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/sequence_to_sequence_and_early_attention_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/sequence_to_sequence_and_early_attention_lib.py (578 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple, Optional
class BahdanauAttention(nn.Module):
def __init__(self, enc_dim: int, dec_dim: int, attn_dim: int):
super().__init__()
# TODO: Initialize query projection, keys projection, and v_att vector
pass
def forward(self, query: torch.Tensor, keys: torch.Tensor,
mask: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
# TODO: Compute additive energy, softmax weights, and dynamic context vector
pass
starter/test_sequence_to_sequence_and_early_attention_lib.py (1005 bytes)
import pytest
import torch
from examples.sequence_to_sequence_and_early_attention_lib import BahdanauAttention
def test_attention_dimensions_and_sum():
torch.manual_seed(42)
attn = BahdanauAttention(enc_dim=10, dec_dim=20, attn_dim=15)
query = torch.randn(4, 20)
keys = torch.randn(4, 7, 10)
context, weights = attn(query, keys)
assert context.shape == (4, 10)
assert weights.shape == (4, 7)
# Softmax weights must sum to 1.0
for s in weights.sum(dim=1):
assert pytest.approx(s.item(), 1e-5) == 1.0
def test_attention_masking():
torch.manual_seed(42)
attn = BahdanauAttention(enc_dim=6, dec_dim=12, attn_dim=8)
query = torch.randn(2, 12)
keys = torch.randn(2, 4, 6)
mask = torch.tensor([[1, 1, 0, 0], [1, 1, 1, 0]])
_, weights = attn(query, keys, mask=mask)
# Masked positions should have near-zero weights
assert weights[0, 2].item() < 1e-4
assert weights[0, 3].item() < 1e-4
assert weights[1, 3].item() < 1e-4
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 222 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_sequence_to_sequence_and_early_attention_lib.py (1005 bytes)
import pytest
import torch
from examples.sequence_to_sequence_and_early_attention_lib import BahdanauAttention
def test_attention_dimensions_and_sum():
torch.manual_seed(42)
attn = BahdanauAttention(enc_dim=10, dec_dim=20, attn_dim=15)
query = torch.randn(4, 20)
keys = torch.randn(4, 7, 10)
context, weights = attn(query, keys)
assert context.shape == (4, 10)
assert weights.shape == (4, 7)
# Softmax weights must sum to 1.0
for s in weights.sum(dim=1):
assert pytest.approx(s.item(), 1e-5) == 1.0
def test_attention_masking():
torch.manual_seed(42)
attn = BahdanauAttention(enc_dim=6, dec_dim=12, attn_dim=8)
query = torch.randn(2, 12)
keys = torch.randn(2, 4, 6)
mask = torch.tensor([[1, 1, 0, 0], [1, 1, 1, 0]])
_, weights = attn(query, keys, mask=mask)
# Masked positions should have near-zero weights
assert weights[0, 2].item() < 1e-4
assert weights[0, 3].item() < 1e-4
assert weights[1, 3].item() < 1e-4
Troubleshooting
Troubleshooting: Day 222 - Sequence-to-Sequence and Early Attention
Common Issues
- Masked positions receiving non-zero attention:
- Cause: Mask value used is not negative enough before softmax.
- Fix: Use
energy.masked_fill(mask == 0, -1e9).
Security notes
Security & Privacy: Day 222 - Sequence-to-Sequence and Early Attention
Security Guidance
- All attention computations execute locally in memory buffers.