Deep Learning βΊ Sequences and Text βΊ Day 222
Day 222: Sequence-to-Sequence and Early Attention
Master the architectural bridge to modern Transformers: understand classic Seq2Seq encoder-decoder networks, diagnose the fixed-size context vector bottleneck, derive Bahdanau Additive Attention and Luong Multiplicative Attention, visualize soft alignment matrices, and build an Attention-augmented Seq2Seq model in PyTorch.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-222-sequence-to-sequence-and-early-attention
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/deep-learning/day-222-sequence-to-sequence-and-early-attention - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Derive the classical Seq2Seq encoder-decoder architecture and explain its fixed-context bottleneck.
- Derive the Bahdanau Additive Attention mathematical formulation: score, softmax weights, and dynamic context vector.
- Compare Bahdanau Additive Attention with Luong Multiplicative (Dot-Product) Attention.
- Implement Teacher Forcing scheduled sampling vs greedy autoregressive inference loops.
- Extract and visualize cross-attention alignment heatmaps demonstrating word-to-word translation correspondences.
Prerequisites
Why this matters
In 2014, deep learning achieved a massive milestone with the introduction of the Sequence-to-Sequence (Seq2Seq) framework (Sutskever et al. & Cho et al.). For the first time, a single end-to-end neural network could map an arbitrary-length input sequence to an arbitrary-length output sequence (such as translating English text into French, or converting audio into transcripts).
However, early Seq2Seq models suffered from a catastrophic architectural flaw known as the Information Bottleneck:
- The Encoder processed a 50-word sentence and compressed the entire meaning into a single, fixed-size vector: $c = h_T \in R^(512)$.
- The Decoder was then forced to reconstruct the entire output translation solely from this single vector.
- As sentence length exceeded 15 to 20 words, translation quality collapsed because a fixed-size vector cannot hold infinite information.
In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced the single most transformative concept in modern artificial intelligence: The Attention Mechanism.
Instead of compressing the input into a single static bottleneck vector, Attention allows the Decoder to look back across all Encoder hidden states at every step of generation, dynamically focusing on the exact source words relevant to the current word being produced.
Today, you will master the foundational bridge from recurrent networks to modern Transformers: Seq2Seq architectures, Bahdanau Additive Attention, Luong Multiplicative Attention, and Attention Alignment Visualization.
The idea in plain language
Imagine translating an entire English Wikipedia article into Spanish:
- Classic Seq2Seq (The Memory Memorizer): You are forced to read all 1,000 words of the English article. Then, the article is locked in a vault, and you must write the entire Spanish translation strictly from your memory. By paragraph 3, you forget names, dates, and subtle details.
- Attention-Augmented Seq2Seq (The Open-Book Translator): You keep the original English text open on your desk in front of you.
- When writing the Spanish word for
"refrigerator", your eyes dart directly to word #42 in the English text ("refrigerator"). - When writing the verb, your eyes focus on the English subject and tense.
- At every step of writing, you dynamically shine a mental spotlight (Attention) on the exact source words you need.
- When writing the Spanish word for
Historical background
- 2014 (Sutskever et al. & Cho et al.): Introduced the RNN Encoder-Decoder Seq2Seq framework, demonstrating that reversing source sentence order improved optimization.
- 2014 (Bahdanau et al. & Additive Attention): Published βNeural Machine Translation by Jointly Learning to Align and Translateβ, introducing soft additive attention and eliminating the fixed-vector bottleneck.
- 2015 (Luong et al. & Multiplicative Attention): Simplified attention scoring into fast dot-product matrix multiplications and introduced global vs local attention mechanisms.
- 2017 (Vaswani et al. & βAttention Is All You Needβ): Discovered that recurrence could be eliminated entirely, creating the Transformer based 100% on self-attention.
What it is β and what it is not
What Attention IS:
- A Dynamic Weighted Memory Lookup: Given a Decoder query vector $s_(t-1)$ and a set of Encoder key/value vectors $h_1, \dots, h_T$, Attention computes a softmax alignment probability distribution $\alpha_(t, i)$ and returns a dynamic weighted context vector $c_t = \sum_i \alpha_(t, i) h_i$.
- A Fully Differentiable Soft Indexing Engine: Unlike hard indexing (which selects a single discrete integer token), soft attention assigns fractional weights ($0.0 \le \alpha_i \le 1.0$) that backpropagate gradients smoothly.
What it is NOT:
- Not Self-Attention (Yet): Early attention (Bahdanau/Luong) is Cross-Attention: it connects a Decoder query to Encoder memories. (Self-Attention connects tokens within the same sequence to each other, which we study in Week 33).
- Not a Static Weight: Attention weights change dynamically at every single decoding step $t$.
Why it was created and what problems it solves
Attention solved three critical bottlenecks in sequential deep learning:
- Eliminating the Information Bottleneck: Allowing Decoders to access raw representations of every source token directly, maintaining high accuracy on 100+ word sentences.
- Direct Gradient Shortcuts: In BPTT, gradients had to travel from Decoder step $t$ all the way back through $h_T, \dots, h_1$. Attention creates direct $O(1)$ gradient highways connecting decoder outputs directly to any encoder time step.
- Model Interpretability: Visualizing the 2D attention alignment matrix $\alpha_(t, i)$ provides human-interpretable heatmaps showing exactly which source words the model used to make each decision.
How it works
Let us dissect the mathematical formulation of Bahdanau Additive Attention and Luong Multiplicative Attention.
1. Bahdanau Additive Attention Formulation
At decoder time step $t$, given current decoder state $s_(t-1) \in R^(d_d)$ and all encoder hidden states $h_i \in R^(d_e)$ for $i \in [1, T]$:
-
Compute Alignment Energy Scores ($e_(t, i)$): Pass query $s_(t-1)$ and memory $h_i$ through a single-hidden-layer feedforward network with $\tanh$ activation:
e_{t, i} = v_a^\top \tanh(W_a * s_{t-1} + U_a * h_i + b_a)where $W_a \in R^(d_a \times d_d)$, $U_a \in R^(d_a \times d_e)$, and $v_a \in R^(d_a)$.
-
Softmax Normalization (Attention Weights $\alpha_(t, i)$):
\alpha_{t, i} = \frac{\exp(e_{t, i})}{\sum_{j=1}^T \exp(e_{t, j})} -
Compute Dynamic Context Vector ($c_t$):
c_t = \sum_{i=1}^T \alpha_{t, i} * h_i -
Update Decoder State ($s_t$) and Output Logits:
s_t = \text{RNNCell}([y_{t-1}, c_t], s_{t-1}) \hat{y}_t = \text{Softmax}(W_s * s_t + W_c * c_t + b)
2. Luong Multiplicative (Dot-Product) Attention Formulation
In 2015, Luong et al. simplified the alignment scoring function:
-
Dot-Product Attention (Equal Dimensions $d_d = d_e$):
\text{score}(s_t, h_i) = s_t^\top h_i -
General (Bilinear) Attention:
\text{score}(s_t, h_i) = s_t^\top W_a h_i -
Scaled Dot-Product Attention: Dividing by $\sqrt(d)$ prevents large vector dimensions from pushing softmax into saturation regions with near-zero gradients (the foundation of modern Transformers).
3. Comparison: Bahdanau vs Luong Attention
| Dimension | Bahdanau (Additive) | Luong (Multiplicative) |
|---|---|---|
| Alignment Formula | $v_a^\top \tanh(W_a s_(t-1) + U_a h_i)$ | $s_t^\top W_a h_i$ |
| Decoder State Used | Previous state $s_(t-1)$ | Current state $s_t$ |
| Computational Speed | Slower (MLP forward pass) | Faster (Matrix Multiplication) |
| Memory Efficiency | $O(d_a)$ intermediate tensors | Direct BLAS GEMM matrix operations |
4. Teacher Forcing vs Autoregressive Inference
-
Training Mode (Teacher Forcing): At step $t$, the decoder input is the true target token $y_(t-1)^*$ from the dataset. All decoding steps can be executed quickly.
-
Inference Mode (Autoregressive Rollout): At step $t$, the ground truth is unknown. The decoder samples or takes $\arg\max$ of its own output $\hat(y)_(t-1)$ and feeds it into step $t$ until the special token
<eos>is produced.
5. Global Attention vs Local Attention (Luong et al., 2015)
Luong et al. explored two distinct computational paradigms for attention:
-
Global Attention: The model attends to all encoder hidden states at every decoder step ($O(N \cdot M)$). Highly accurate, but computationally heavy for long paragraphs.
-
Local (Monotonic & Predictive) Attention: The model first predicts an aligned target center position $p_t \in [1, T]$ in the source sentence, then places a Gaussian window $[p_t - D, p_t + D]$ around that position:
p_t = T * sigma(v_p^\top tanh(W_p s_t)) alpha_{t, i} = a_{t, i} * exp( - (i - p_t)^2 / (2 * sigma^2) ) -
Focuses computation only on a small local window of $2D + 1$ source tokens, reducing inference compute for long documents.
6. Coverage Mechanisms to Prevent Repetition
A well-known defect of early attention models was over-generation and repetition (e.g. translating the same source phrase three times):
-
Coverage Vectors (Tu et al., 2016 / See et al., 2017): Maintains a cumulative sum of all past attention distributions:
c^t = sum_{k=1}^{t-1} alpha^k -
Coverage Loss: Adds a penalty term that discourages the network from attending repeatedly to source words that have already received high cumulative attention:
L_{\text{cov}} = sum_i min(alpha_{t, i}, c_i^t) -
Dramatically reduces repetition in document summarization and neural machine translation.
7. The Bridge to Transformers: Self-Attention
In Seq2Seq Cross-Attention, the Query ($Q$) comes from the Decoder, while Keys ($K$) and Values ($V$) come from the Encoder:
Attention(Q, K, V) = softmax( (Q K^\top) / sqrt(d_k) ) * V
- In 2017, Vaswani et al. asked: What if Queries, Keys, and Values all come from the SAME sentence?
- This insight yielded Self-Attention: tokens inside the same sentence attend directly to each other (e.g.
"The animal didn't cross the street because it was too tired"->"it"attends directly to"animal"), eliminating recurrence entirely.
8. Generation Decoding Strategies: Greedy vs Beam Search
Once logits are produced at decoder step $t$:
-
Greedy Decoding: Picks $\hat(y)_t = \arg\max p_t$. Fast ($O(1)$), but myopic: a single early mistake derails the entire sentence.
-
Beam Search: Maintains a beam of the top $B = 4$ or $B = 8$ highest cumulative probability sequence hypotheses:
\text{Score}(y_1, \dots, y_t) = \sum_{k=1}^t \log P(y_k | y_{<k}, x)Expands all $B \times |V|$ candidates at each step and retains the top $B$ paths, finding globally optimal translations.
- Length Normalization: Without length normalization, beam search inherently favors shorter sentences because multiplying probabilities $< 1$ penalizes longer paths. Dividing total log likelihood by sequence length $|Y|^\alpha$ neutralizes this bias.
An everyday analogy
Think of a courtroom stenographer transcribing a witnessβs testimony:
- Classic Seq2Seq: The stenographer is blindfolded and forced to listen to 10 minutes of complex testimony, then write the entire legal summary from memory.
- Bahdanau Attention: The stenographer has an audio recording interface with a scrub wheel (Attention weights $\alpha_(t, i)$).
- When writing about the defendantβs car, the stenographer rewinds the audio slider directly to the 2-minute mark where the car was described (Dynamic Context Vector $c_t$).
- They transcribe the sentence with 100% accuracy.
Examples in practice
Let us build a Bahdanau Additive Attention Module in PyTorch:
import torch
import torch.nn as nn
import torch.nn.functional as F
class BahdanauAttention(nn.Module):
def __init__(self, enc_dim: int, dec_dim: int, attn_dim: int):
super().__init__()
self.W_query = nn.Linear(dec_dim, attn_dim, bias=False)
self.U_keys = nn.Linear(enc_dim, attn_dim, bias=False)
self.v_att = nn.Linear(attn_dim, 1, bias=False)
def forward(self, query: torch.Tensor, keys: torch.Tensor,
mask: torch.Tensor = None) -> torch.Tensor:
# query shape: (Batch, dec_dim) -> (Batch, 1, attn_dim)
# keys shape: (Batch, Seq_Len, enc_dim) -> (Batch, Seq_Len, attn_dim)
proj_query = self.W_query(query).unsqueeze(1)
proj_keys = self.U_keys(keys)
# Compute additive energy: v_a^T * tanh(W*s + U*h)
energy = self.v_att(torch.tanh(proj_query + proj_keys)).squeeze(2) # (Batch, Seq_Len)
# Mask padding tokens if mask provided
if mask is not None:
energy = energy.masked_fill(mask == 0, -1e9)
# Softmax attention weights alpha
attn_weights = F.softmax(energy, dim=1) # (Batch, Seq_Len)
# Compute dynamic context vector: sum(alpha * keys)
context = torch.bmm(attn_weights.unsqueeze(1), keys).squeeze(1) # (Batch, enc_dim)
return context, attn_weights
Implications: security, privacy, performance, scalability, and cost
- Quadratic Complexity ($O(N \cdot M)$):
- Attention between source sequence of length $N$ and target sequence of length $M$ computes an $N \times M$ matrix. While manageable for sentences ($30 \times 30 = 900$ elements), it becomes expensive for long documents (e.g. $10,000 \times 10,000 = 100,000,000$).
- Attention Heatmap Interpretability in Legal/Healthcare:
- In medical record summarization, attention weights provide clinical audibility: doctors can verify which diagnostic lab result in the patient history triggered a flagged medication alert.
Alternatives: free, open source, and commercial
| Architecture | Attention Style | Computation | Best For |
|---|---|---|---|
| Seq2Seq + Bahdanau | Additive MLP Attention | $O(N \cdot M)$ Sequential | Historical NLP & Interpretability |
| Seq2Seq + Luong | Dot-Product Multiplicative | $O(N \cdot M)$ Fast GEMM | Fast recurrent translation |
| Transformer (Vaswani) | Scaled Multi-Head Self-Attn | $O(N^2)$ Fully Parallel | SOTA LLMs (GPT, Llama, Claude) |
| Linear Attention (Mamba) | State Space / Recurrent | $O(N)$ Linear | Million-token context windows |
Comparison with related concepts
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ATTENTION EVOLUTION TAXONOMY β
ββββββββββββββββββββββ¬ββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββ€
β Mechanism β Query / Key β Formula β Parallelism β
ββββββββββββββββββββββΌββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββ€
β No Attention β Fixed h_T β c = h_T β None β
β Bahdanau Additive β Decoder / Enc β v^T tanh(Ws + Uh)β Recurrent β
β Luong Dot-Product β Decoder / Enc β s^T W h β Recurrent β
β Multi-Head Self-Attβ Within Sequence β Q K^T / sqrt(d) β Fully Para β
ββββββββββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββ
When to use it β and when not to
When to USE Recurrent Seq2Seq with Attention:
- Embedded edge translation and low-memory embedded voice assistants where full Transformer attention caches exceed available SRAM.
When NOT to use:
- Modern large-scale generative AI and foundation models (Transformers with Multi-Head Self-Attention offer superior GPU parallel scaling).
The AI connection
- Attention removed the fixed-size bottleneck. A whole sentence compressed into one vector is a hard limit, and a weighted lookup over every encoder state is not.
- Soft indexing is what makes it trainable. Fractional weights are differentiable; choosing one position discretely is not.
- The alignment weights are interpretable, which is unusual and genuinely useful: they show which source words the model used for each output word.
- Cross-attention became self-attention by pointing the same mechanism at the sequence itself, which is the single step from here to transformers.
- Beam search is a decoding decision, not a model property β the same model produces very different output under greedy and beam decoding.
Knowledge check
- What caused the information bottleneck in early 2014 Seq2Seq models?
- What are the three core mathematical steps of Bahdanau Additive Attention?
- How does Luong Multiplicative Attention differ from Bahdanau Additive Attention?
- What is Teacher Forcing and what problem does it solve during training?
- How does an Attention Alignment matrix provide interpretability for translation models?
Hands-on exercise
In this lab, you will build and test an Attention-augmented Seq2Seq mechanism in PyTorch: implement the Bahdanau Additive Attention module from first principles, compute dynamic context vectors over encoder hidden states, mask padding tokens during softmax normalization, implement Teacher Forcing vs greedy decoding loops, and verify attention weight distributions.
Expected output
[Seq2Seq and Early Attention Verification Suite]
Testing Bahdanau Additive Attention Forward Pass:
Encoder Keys Shape: torch.Size([4, 8, 32]) (Batch=4, Steps=8, Dim=32)
Decoder Query Shape: torch.Size([4, 16])
Context Vector Shape: torch.Size([4, 32])
Attention Weights Sum (Probability Check): 1.0000 [VERIFIED]
Testing Attention Padding Masking:
Padding Positions (Mask=0) Received Near-Zero Attention: alpha <= 1e-6 [MASKED]
Testing Greedy Autoregressive Generation Rollout:
Generated Sequence Tokens: [2, 14, 89, 412, 3] (Terminated at <eos>=3)
Test Suite: 3 passed in 0.36s
Validate your work
Run the automated test runner:
./tests/run_tests.sh
Troubleshooting
- If attention weights do not sum to 1.0, ensure
F.softmaxis applied alongdim=1(across the sequence length dimension). - When masking padding tokens, apply
energy.masked_fill(mask == 0, -1e9)before applying softmax.
Common mistakes
- Applying Softmax on Unmasked Padding: Forgetting to mask
<pad>tokens causes the model to assign attention probability to empty padding slots, diluting the context vector.
Practice assignment
- Implement Luong Scaled Dot-Product Attention and compare forward pass execution speed against Bahdanau Additive Attention.
- Build an Interactive Alignment Heatmap Plotter using
matplotlibthat renders the 2D attention matrix $\alpha_(t, i)$ for a sample sentence.
Extension challenge
Implement Beam Search Decoding with Length Penalty:
- Replace greedy $\arg\max$ decoding with a Beam Search decoder maintaining the top $B = 5$ candidate hypotheses.
- Apply length normalization: $Score(Y) = \frac(\log P(Y))((5 + |Y|)^\alpha / 6^\alpha)$.
Quiz
Q1. What was the fatal "Information Bottleneck" in the original 2014 Sutskever Seq2Seq encoder-decoder architecture that attention solved?
- Backpropagation was impossible
- The decoder could only generate numbers
- The model required 500 GPUs
- The whole sentence had to fit one fixed-length vector
Show answer
Answer: D. The whole sentence had to fit one fixed-length vector
Compressing arbitrary-length documents into a single fixed vector creates a severe information bottleneck, degrading performance on long sequences.
Q2. How does Bahdanau Additive Attention (1914) construct the dynamic context vector c_t at each decoder step t?
- It takes a weighted sum over all encoder states
- It takes the maximum value of the first token
- It averages the entire vocabulary embedding table
- It randomly drops 50% of the encoder states
Show answer
Answer: A. It takes a weighted sum over all encoder states
Attention computes dynamic soft weights over all encoder hidden states, allowing the decoder to look back at relevant source words at every decoding step.
Q3. What is the mathematical difference between Bahdanau Additive Attention and Luong Multiplicative Attention?
- Luong attention uses convolutional layers throughout
- Bahdanau uses an MLP; Luong uses a matrix product
- Bahdanau attention only works in French
- Luong attention does not use softmax
Show answer
Answer: B. Bahdanau uses an MLP; Luong uses a matrix product
Bahdanau uses an additive MLP with tanh, while Luong uses efficient bilinear dot products (s^T W h).
Q4. What is "Teacher Forcing" during Seq2Seq training, and why is it used?
- Freezing the encoder weights
- Training the model with an automated grading script
- Feeding the true previous token instead of the predicted one
- Multiplying the learning rate by 10
Show answer
Answer: C. Feeding the true previous token instead of the predicted one
Teacher forcing feeds ground-truth tokens during training to prevent compounding error cascades during early training iterations.
Q5. Why are Attention Alignment Heatmaps considered one of the earliest forms of interpretable neural network visualizations?
- They display the dataset's folder hierarchy on disk
- They display the GPU temperature
- They show the loss curve
- The attention matrix shows the word alignments
Show answer
Answer: D. The attention matrix shows the word alignments
Visualizing alpha_{t,i} displays a soft alignment matrix showing exact source-to-target semantic correspondences.
Glossary
- Sequence-to-Sequence (Seq2Seq)
- A neural architecture consisting of an Encoder that processes an input sequence and a Decoder that generates an output sequence.
- Information Bottleneck
- The performance degradation caused by compressing an arbitrary-length sequence into a single fixed-size context vector.
- Additive Attention (Bahdanau)
- An alignment scoring function that combines query and key vectors using a feedforward neural layer with tanh activation.
- Multiplicative Attention (Luong)
- An alignment scoring function that calculates the direct dot product or bilinear projection between query and key vectors.
- Attention Weights
- A normalized probability distribution (alpha_t) produced via softmax indicating the relative importance of each source token.
- Dynamic Context Vector
- A weighted sum of all encoder hidden states (c_t = sum_i alpha_{t,i} h_i) computed uniquely at each decoder time step.
- Teacher Forcing
- A training strategy where ground-truth tokens are fed as decoder inputs to stabilize and accelerate convergence.
- Alignment Matrix
- A 2D heatmap matrix mapping source sequence positions to target sequence positions via attention weights.
Sources and further reading
- Sequence to Sequence Learning with Neural Networks β NeurIPS 2014 (accessed 2026-08-29)
- Neural Machine Translation by Jointly Learning to Align and Translate β ICLR 2015 (accessed 2026-08-29)
- Effective Approaches to Attention-based Neural Machine Translation β EMNLP 2015 (accessed 2026-08-29)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.