Deep Learning β€Ί Sequences and Text β€Ί Day 222

Day 222: Sequence-to-Sequence and Early Attention

Day 222 of 365 β€” Sequence-to-Sequence and Early Attention

Master the architectural bridge to modern Transformers: understand classic Seq2Seq encoder-decoder networks, diagnose the fixed-size context vector bottleneck, derive Bahdanau Additive Attention and Luong Multiplicative Attention, visualize soft alignment matrices, and build an Attention-augmented Seq2Seq model in PyTorch.

Course
Deep Learning
Category
Sequences and Text
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-222-sequence-to-sequence-and-early-attention

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-222-sequence-to-sequence-and-early-attention
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

In 2014, deep learning achieved a massive milestone with the introduction of the Sequence-to-Sequence (Seq2Seq) framework (Sutskever et al. & Cho et al.). For the first time, a single end-to-end neural network could map an arbitrary-length input sequence to an arbitrary-length output sequence (such as translating English text into French, or converting audio into transcripts).

However, early Seq2Seq models suffered from a catastrophic architectural flaw known as the Information Bottleneck:

In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced the single most transformative concept in modern artificial intelligence: The Attention Mechanism.

Instead of compressing the input into a single static bottleneck vector, Attention allows the Decoder to look back across all Encoder hidden states at every step of generation, dynamically focusing on the exact source words relevant to the current word being produced.

Today, you will master the foundational bridge from recurrent networks to modern Transformers: Seq2Seq architectures, Bahdanau Additive Attention, Luong Multiplicative Attention, and Attention Alignment Visualization.


The idea in plain language

Diagram: soft attention weights compared with discrete selection

Diagram: the fixed-size context bottleneck in encoder-decoder models

Imagine translating an entire English Wikipedia article into Spanish:


Historical background

Sequence to sequence encoder decoder architecture demonstrating the fixed size context vector information bottleneck

  1. 2014 (Sutskever et al. & Cho et al.): Introduced the RNN Encoder-Decoder Seq2Seq framework, demonstrating that reversing source sentence order improved optimization.
  2. 2014 (Bahdanau et al. & Additive Attention): Published β€œNeural Machine Translation by Jointly Learning to Align and Translate”, introducing soft additive attention and eliminating the fixed-vector bottleneck.
  3. 2015 (Luong et al. & Multiplicative Attention): Simplified attention scoring into fast dot-product matrix multiplications and introduced global vs local attention mechanisms.
  4. 2017 (Vaswani et al. & β€œAttention Is All You Need”): Discovered that recurrence could be eliminated entirely, creating the Transformer based 100% on self-attention.

What it is β€” and what it is not

What Attention IS:

What it is NOT:


Why it was created and what problems it solves

Attention solved three critical bottlenecks in sequential deep learning:

  1. Eliminating the Information Bottleneck: Allowing Decoders to access raw representations of every source token directly, maintaining high accuracy on 100+ word sentences.
  2. Direct Gradient Shortcuts: In BPTT, gradients had to travel from Decoder step $t$ all the way back through $h_T, \dots, h_1$. Attention creates direct $O(1)$ gradient highways connecting decoder outputs directly to any encoder time step.
  3. Model Interpretability: Visualizing the 2D attention alignment matrix $\alpha_(t, i)$ provides human-interpretable heatmaps showing exactly which source words the model used to make each decision.

How it works

Diagram: the four steps of an attention computation

Let us dissect the mathematical formulation of Bahdanau Additive Attention and Luong Multiplicative Attention.


1. Bahdanau Additive Attention Formulation

Bahdanau Additive Attention mechanism showing dynamic soft attention weights connecting decoder query state to all encoder keys

At decoder time step $t$, given current decoder state $s_(t-1) \in R^(d_d)$ and all encoder hidden states $h_i \in R^(d_e)$ for $i \in [1, T]$:

  1. Compute Alignment Energy Scores ($e_(t, i)$): Pass query $s_(t-1)$ and memory $h_i$ through a single-hidden-layer feedforward network with $\tanh$ activation:

    e_{t, i} = v_a^\top \tanh(W_a * s_{t-1} + U_a * h_i + b_a)

    where $W_a \in R^(d_a \times d_d)$, $U_a \in R^(d_a \times d_e)$, and $v_a \in R^(d_a)$.

  2. Softmax Normalization (Attention Weights $\alpha_(t, i)$):

    \alpha_{t, i} = \frac{\exp(e_{t, i})}{\sum_{j=1}^T \exp(e_{t, j})}
  3. Compute Dynamic Context Vector ($c_t$):

    c_t = \sum_{i=1}^T \alpha_{t, i} * h_i
  4. Update Decoder State ($s_t$) and Output Logits:

    s_t = \text{RNNCell}([y_{t-1}, c_t], s_{t-1})
    \hat{y}_t = \text{Softmax}(W_s * s_t + W_c * c_t + b)

2. Luong Multiplicative (Dot-Product) Attention Formulation

In 2015, Luong et al. simplified the alignment scoring function:


3. Comparison: Bahdanau vs Luong Attention

Diagram: additive attention scoring compared with multiplicative scoring

DimensionBahdanau (Additive)Luong (Multiplicative)
Alignment Formula$v_a^\top \tanh(W_a s_(t-1) + U_a h_i)$$s_t^\top W_a h_i$
Decoder State UsedPrevious state $s_(t-1)$Current state $s_t$
Computational SpeedSlower (MLP forward pass)Faster (Matrix Multiplication)
Memory Efficiency$O(d_a)$ intermediate tensorsDirect BLAS GEMM matrix operations

4. Teacher Forcing vs Autoregressive Inference


5. Global Attention vs Local Attention (Luong et al., 2015)

Luong et al. explored two distinct computational paradigms for attention:


6. Coverage Mechanisms to Prevent Repetition

Diagram: how a coverage mechanism prevents repeated attention to the same source positions

A well-known defect of early attention models was over-generation and repetition (e.g. translating the same source phrase three times):


7. The Bridge to Transformers: Self-Attention

Diagram: cross-attention compared with self-attention by where queries and keys originate

In Seq2Seq Cross-Attention, the Query ($Q$) comes from the Decoder, while Keys ($K$) and Values ($V$) come from the Encoder:

Attention(Q, K, V) = softmax( (Q K^\top) / sqrt(d_k) ) * V

Diagram: greedy decoding compared with beam search

Once logits are produced at decoder step $t$:

  1. Greedy Decoding: Picks $\hat(y)_t = \arg\max p_t$. Fast ($O(1)$), but myopic: a single early mistake derails the entire sentence.

  2. Beam Search: Maintains a beam of the top $B = 4$ or $B = 8$ highest cumulative probability sequence hypotheses:

    \text{Score}(y_1, \dots, y_t) = \sum_{k=1}^t \log P(y_k | y_{<k}, x)

    Expands all $B \times |V|$ candidates at each step and retains the top $B$ paths, finding globally optimal translations.

    • Length Normalization: Without length normalization, beam search inherently favors shorter sentences because multiplying probabilities $< 1$ penalizes longer paths. Dividing total log likelihood by sequence length $|Y|^\alpha$ neutralizes this bias.

An everyday analogy

Diagram: translating from memory compared with referring to the source

Think of a courtroom stenographer transcribing a witness’s testimony:


Examples in practice

Let us build a Bahdanau Additive Attention Module in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F

class BahdanauAttention(nn.Module):
    def __init__(self, enc_dim: int, dec_dim: int, attn_dim: int):
        super().__init__()
        self.W_query = nn.Linear(dec_dim, attn_dim, bias=False)
        self.U_keys = nn.Linear(enc_dim, attn_dim, bias=False)
        self.v_att = nn.Linear(attn_dim, 1, bias=False)

    def forward(self, query: torch.Tensor, keys: torch.Tensor,
                mask: torch.Tensor = None) -> torch.Tensor:
        # query shape: (Batch, dec_dim) -> (Batch, 1, attn_dim)
        # keys shape: (Batch, Seq_Len, enc_dim) -> (Batch, Seq_Len, attn_dim)
        
        proj_query = self.W_query(query).unsqueeze(1)
        proj_keys = self.U_keys(keys)

        # Compute additive energy: v_a^T * tanh(W*s + U*h)
        energy = self.v_att(torch.tanh(proj_query + proj_keys)).squeeze(2) # (Batch, Seq_Len)

        # Mask padding tokens if mask provided
        if mask is not None:
            energy = energy.masked_fill(mask == 0, -1e9)

        # Softmax attention weights alpha
        attn_weights = F.softmax(energy, dim=1) # (Batch, Seq_Len)

        # Compute dynamic context vector: sum(alpha * keys)
        context = torch.bmm(attn_weights.unsqueeze(1), keys).squeeze(1) # (Batch, enc_dim)

        return context, attn_weights

Implications: security, privacy, performance, scalability, and cost

Diagram: the interpretability, cost, alignment and architectural implications of attention

  1. Quadratic Complexity ($O(N \cdot M)$):
    • Attention between source sequence of length $N$ and target sequence of length $M$ computes an $N \times M$ matrix. While manageable for sentences ($30 \times 30 = 900$ elements), it becomes expensive for long documents (e.g. $10,000 \times 10,000 = 100,000,000$).
  2. Attention Heatmap Interpretability in Legal/Healthcare:
    • In medical record summarization, attention weights provide clinical audibility: doctors can verify which diagnostic lab result in the patient history triggered a flagged medication alert.

Alternatives: free, open source, and commercial

ArchitectureAttention StyleComputationBest For
Seq2Seq + BahdanauAdditive MLP Attention$O(N \cdot M)$ SequentialHistorical NLP & Interpretability
Seq2Seq + LuongDot-Product Multiplicative$O(N \cdot M)$ Fast GEMMFast recurrent translation
Transformer (Vaswani)Scaled Multi-Head Self-Attn$O(N^2)$ Fully ParallelSOTA LLMs (GPT, Llama, Claude)
Linear Attention (Mamba)State Space / Recurrent$O(N)$ LinearMillion-token context windows

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   ATTENTION EVOLUTION TAXONOMY                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Mechanism          β”‚ Query / Key     β”‚ Formula          β”‚ Parallelism  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ No Attention       β”‚ Fixed h_T       β”‚ c = h_T          β”‚ None         β”‚
β”‚ Bahdanau Additive  β”‚ Decoder / Enc   β”‚ v^T tanh(Ws + Uh)β”‚ Recurrent    β”‚
β”‚ Luong Dot-Product  β”‚ Decoder / Enc   β”‚ s^T W h          β”‚ Recurrent    β”‚
β”‚ Multi-Head Self-Attβ”‚ Within Sequence β”‚ Q K^T / sqrt(d)  β”‚ Fully Para   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: where cross-attention is used in modern architectures

When to USE Recurrent Seq2Seq with Attention:

When NOT to use:


The AI connection

Knowledge check

  1. What caused the information bottleneck in early 2014 Seq2Seq models?
  2. What are the three core mathematical steps of Bahdanau Additive Attention?
  3. How does Luong Multiplicative Attention differ from Bahdanau Additive Attention?
  4. What is Teacher Forcing and what problem does it solve during training?
  5. How does an Attention Alignment matrix provide interpretability for translation models?

Hands-on exercise

In this lab, you will build and test an Attention-augmented Seq2Seq mechanism in PyTorch: implement the Bahdanau Additive Attention module from first principles, compute dynamic context vectors over encoder hidden states, mask padding tokens during softmax normalization, implement Teacher Forcing vs greedy decoding loops, and verify attention weight distributions.

Expected output

[Seq2Seq and Early Attention Verification Suite]
Testing Bahdanau Additive Attention Forward Pass:
  Encoder Keys Shape: torch.Size([4, 8, 32]) (Batch=4, Steps=8, Dim=32)
  Decoder Query Shape: torch.Size([4, 16])
  Context Vector Shape: torch.Size([4, 32])
  Attention Weights Sum (Probability Check): 1.0000 [VERIFIED]
Testing Attention Padding Masking:
  Padding Positions (Mask=0) Received Near-Zero Attention: alpha <= 1e-6 [MASKED]
Testing Greedy Autoregressive Generation Rollout:
  Generated Sequence Tokens: [2, 14, 89, 412, 3] (Terminated at <eos>=3)
Test Suite: 3 passed in 0.36s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Luong Scaled Dot-Product Attention and compare forward pass execution speed against Bahdanau Additive Attention.
  2. Build an Interactive Alignment Heatmap Plotter using matplotlib that renders the 2D attention matrix $\alpha_(t, i)$ for a sample sentence.

Extension challenge

Implement Beam Search Decoding with Length Penalty:

Quiz

Q1. What was the fatal "Information Bottleneck" in the original 2014 Sutskever Seq2Seq encoder-decoder architecture that attention solved?

  1. Backpropagation was impossible
  2. The decoder could only generate numbers
  3. The model required 500 GPUs
  4. The whole sentence had to fit one fixed-length vector
Show answer

Answer: D. The whole sentence had to fit one fixed-length vector

Compressing arbitrary-length documents into a single fixed vector creates a severe information bottleneck, degrading performance on long sequences.

Q2. How does Bahdanau Additive Attention (1914) construct the dynamic context vector c_t at each decoder step t?

  1. It takes a weighted sum over all encoder states
  2. It takes the maximum value of the first token
  3. It averages the entire vocabulary embedding table
  4. It randomly drops 50% of the encoder states
Show answer

Answer: A. It takes a weighted sum over all encoder states

Attention computes dynamic soft weights over all encoder hidden states, allowing the decoder to look back at relevant source words at every decoding step.

Q3. What is the mathematical difference between Bahdanau Additive Attention and Luong Multiplicative Attention?

  1. Luong attention uses convolutional layers throughout
  2. Bahdanau uses an MLP; Luong uses a matrix product
  3. Bahdanau attention only works in French
  4. Luong attention does not use softmax
Show answer

Answer: B. Bahdanau uses an MLP; Luong uses a matrix product

Bahdanau uses an additive MLP with tanh, while Luong uses efficient bilinear dot products (s^T W h).

Q4. What is "Teacher Forcing" during Seq2Seq training, and why is it used?

  1. Freezing the encoder weights
  2. Training the model with an automated grading script
  3. Feeding the true previous token instead of the predicted one
  4. Multiplying the learning rate by 10
Show answer

Answer: C. Feeding the true previous token instead of the predicted one

Teacher forcing feeds ground-truth tokens during training to prevent compounding error cascades during early training iterations.

Q5. Why are Attention Alignment Heatmaps considered one of the earliest forms of interpretable neural network visualizations?

  1. They display the dataset's folder hierarchy on disk
  2. They display the GPU temperature
  3. They show the loss curve
  4. The attention matrix shows the word alignments
Show answer

Answer: D. The attention matrix shows the word alignments

Visualizing alpha_{t,i} displays a soft alignment matrix showing exact source-to-target semantic correspondences.

Glossary

Sequence-to-Sequence (Seq2Seq)
A neural architecture consisting of an Encoder that processes an input sequence and a Decoder that generates an output sequence.
Information Bottleneck
The performance degradation caused by compressing an arbitrary-length sequence into a single fixed-size context vector.
Additive Attention (Bahdanau)
An alignment scoring function that combines query and key vectors using a feedforward neural layer with tanh activation.
Multiplicative Attention (Luong)
An alignment scoring function that calculates the direct dot product or bilinear projection between query and key vectors.
Attention Weights
A normalized probability distribution (alpha_t) produced via softmax indicating the relative importance of each source token.
Dynamic Context Vector
A weighted sum of all encoder hidden states (c_t = sum_i alpha_{t,i} h_i) computed uniquely at each decoder time step.
Teacher Forcing
A training strategy where ground-truth tokens are fed as decoder inputs to stabilize and accelerate convergence.
Alignment Matrix
A 2D heatmap matrix mapping source sequence positions to target sequence positions via attention weights.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.