Deep Learning β€Ί Transformers β€Ί Day 226

Day 226: Self-Attention, Step by Step

Day 226 of 365 β€” Self-Attention, Step by Step

Master the multi-head subspace projection mechanics and positional encoding foundations of Transformers: understand why splitting into h heads enables diverse relationship tracking, derive Sinusoidal Positional Encodings, understand modern Rotary Position Embeddings (RoPE), and build a complete Multi-Head Self-Attention module in PyTorch.

Course
Deep Learning
Category
Transformers
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-226-self-attention-step-by-step

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-226-self-attention-step-by-step
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Yesterday, you mastered the core mathematical engine of the Transformer: Scaled Dot-Product Attention. However, a single attention head is fundamentally limited:

If a network possesses only a single attention head, it is forced to average all of these competing linguistic signals into a single blurry distribution.

By introducing Multi-Head Attention (MHA), Vaswani et al. gave the Transformer the ability to project representations into multiple parallel subspaces, allowing different heads to specialize in distinct linguistic, syntactic, and factual relationships simultaneously.

Furthermore, because self-attention is mathematically permutation-equivariant (it treats text as an unordered bag of tokens), Transformers require Positional Encodings to inject the sequential order of words.

Today, you will master the step-by-step architecture of Multi-Head Self-Attention, Sinusoidal Positional Encodings, and modern Rotary Position Embeddings (RoPE).


The idea in plain language

Imagine a committee of 8 specialized investigators examining a complex corporate legal contract:

Each investigator reads the exact same document through their own specialized analytical lens (Attention Head). At the end of the meeting, all 8 investigators combine their findings into a single unified executive dossier (Output Projection W_O).


Historical background

Multi-Head Attention architecture diagram showing linear projection into multiple subspace heads parallel attention concatenation and output projection

  1. 2017 (Vaswani et al. & Multi-Head Attention): Introduced Multi-Head Attention with h = 8 heads and d_{\text{model}} = 512, proving that multi-head projections achieved superior BLEU scores compared to single-head attention of identical total dimensionality.
  2. 2019 (Clark et al. & β€œWhat Does BERT Look At?”): Probed the internal attention maps of BERT’s 144 attention heads, discovering that individual heads naturally learned classical linguistic structures (e.g. Head 8-10 specialized in direct objects of verbs, Head 7-6 specialized in possessive pronouns).
  3. 2021 (Jianlin Su & RoPE): Introduced Rotary Position Embedding (RoPE) in RoFormer, rotating query and key vectors in the 2D complex plane to enforce relative distance encoding. RoPE became the universal standard in modern open-weight LLMs (LLaMA, Mistral, Qwen, DeepSeek).

What it is β€” and what it is not

What Multi-Head Attention IS:

What it is NOT:

Diagram: single-head attention compared with multi-head attention


Why it was created and what problems it solves

Multi-Head Attention and Positional Encodings solved two critical limitations of early attention models:

  1. Preventing Attention Smearing: In a single attention head, if a token must attend to 3 different words, their weights split (0.33, 0.33, 0.33), diluting the pooled vector. Multi-head attention allows Head 1 to assign 0.99 to word A, while Head 2 assigns 0.99 to word B.
  2. Injecting Sequential Order Without Recurrence: Positional encodings provide the network with word order awareness while retaining 100% parallel GPU execution.

How it works

Let us dissect the step-by-step mathematical mechanics of Multi-Head Self-Attention and Positional Encodings.


1. Multi-Head Self-Attention Step-by-Step

Diagram: how multi-head attention splits and recombines the representation

Given input matrix X \in \mathbb{R}^{N \times d_{\text{model}}} and number of heads h (where d_k = d_v = d_{\text{model}} / h):

  1. Linear Projections (Q, K, V): Project input X through three unified weight matrices:

    Q = X W_Q \in \mathbb{R}^{N \times d_{\text{model}}}
    K = X W_K \in \mathbb{R}^{N \times d_{\text{model}}}
    V = X W_V \in \mathbb{R}^{N \times d_{\text{model}}}
  2. Subspace Reshaping & Transposition (Head Splitting): Reshape matrices from (B, N, d_{\text{model}}) into (B, N, h, d_k) and permute axes to:

    (B, h, N, d_k)

    This formats the tensor so that all h heads across all batch samples can be processed in a single batched GPU GEMM operation.

  3. Parallel Scaled Dot-Product Attention: Compute attention for all heads simultaneously:

    \text{head}_i = \text{Softmax}\left( \frac{Q_i K_i^\top}{\sqrt{d_k}} \right) V_i \in \mathbb{R}^{N \times d_k}
  4. Head Concatenation: Transpose and reshape the h head outputs back into a unified matrix:

    \text{Concat}(\text{head}_1, \dots, \text{head}_h) \in \mathbb{R}^{N \times d_{\text{model}}}
  5. Output Linear Projection (W_O): Project the concatenated representations through W_O \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}:

    \text{MHA}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W_O

2. Sinusoidal Positional Encodings (Vaswani et al., 2017)

Diagram: why self-attention requires explicit positional information

Positional Encoding comparison illustrating Vaswani Sinusoidal absolute positions versus modern Rotary Position Embeddings rotation in the complex plane

Because self-attention contains no recurrent time loops, word positions must be added directly to the input token embeddings:

x_{\text{final}} = e_{\text{token}} + \text{PE}(pos)

Vaswani et al. defined deterministic sinusoidal functions across dimension index i \in [0, d_{\text{model}} / 2 - 1]:

\text{PE}_{(pos, 2i)} = \sin\left( \frac{pos}{10000^{2i / d_{\text{model}}}} \right)
\text{PE}_{(pos, 2i+1)} = \cos\left( \frac{pos}{10000^{2i / d_{\text{model}}}} \right)

The Linear Relative Shift Property:

For any fixed offset k, \text{PE}_{pos+k} can be represented as a linear function of \text{PE}_{pos}:

\begin{bmatrix}
\sin(\omega (pos + k)) \\
\cos(\omega (pos + k))
\end{bmatrix}
=
\begin{bmatrix}
\cos(\omega k) & \sin(\omega k) \\
-\sin(\omega k) & \cos(\omega k)
\end{bmatrix}
\begin{bmatrix}
\sin(\omega pos) \\
\cos(\omega pos)
\end{bmatrix}

This rotation matrix allows the neural network to easily learn relative positional distances.


3. Modern LLM Standard: Rotary Position Embeddings (RoPE)

Diagram: additive positional encodings compared with rotary position embeddings

In modern foundation models (Llama 3, Mistral, Gemma), additive sinusoidal encodings have been replaced by Rotary Position Embeddings (RoPE / Su et al., 2021):


4. Learned Positional Embeddings (BERT & GPT-2)

Rather than fixed sinusoidal formulas, models like original BERT and GPT-2 used Learned Positional Embeddings:


5. ALiBi: Attention with Linear Biases (Press et al., 2021)

Diagram: four approaches to positional information in transformers

To eliminate positional embeddings entirely and enable infinite context extrapolation:


6. Attention Head Redundancy & Pruning (Michel et al., 2019)

Diagram: attention head importance and prunability after training

Researchers discovered that while multi-head attention is crucial during pretraining:


7. Grouped-Query Attention (GQA) Mathematical Mechanics

Diagram: multi-head, grouped-query and multi-query attention compared by cache size

In high-throughput LLM serving, Multi-Head Attention requires caching H \times L \times d_k key/value vectors per token:


An everyday analogy

Diagram: single-microphone recording compared with multi-track recording

Think of sound recording in a professional music studio:


Examples in practice

Let us build a Complete Multi-Head Self-Attention Module in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, num_heads: int, dropout: float = 0.1):
        super().__init__()
        assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
        
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        # Linear projections for Q, K, V
        self.W_q = nn.Linear(d_model, d_model, bias=False)
        self.W_k = nn.Linear(d_model, d_model, bias=False)
        self.W_v = nn.Linear(d_model, d_model, bias=False)
        self.W_o = nn.Linear(d_model, d_model, bias=False)

        self.dropout = nn.Dropout(p=dropout)

    def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
                mask: torch.Tensor = None) -> tuple[torch.Tensor, torch.Tensor]:
        batch_size = q.size(0)

        # 1. Linear projections and reshape to (Batch, Num_Heads, Seq_Len, d_k)
        Q = self.W_q(q).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(k).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(v).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)

        # 2. Scaled Dot-Product Attention across all heads
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)

        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)

        attn_weights = F.softmax(scores, dim=-1)
        attn_weights = self.dropout(attn_weights)

        # 3. Context aggregation: (Batch, Num_Heads, Seq_Q, d_k)
        context = torch.matmul(attn_weights, V)

        # 4. Concatenate heads: transpose to (Batch, Seq_Q, Num_Heads, d_k) -> view (Batch, Seq_Q, d_model)
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)

        # 5. Final output projection
        output = self.W_o(context)

        return output, attn_weights

Implications: security, privacy, performance, scalability, and cost

Diagram: the interpretability, serving, pruning and design implications of multi-head attention

  1. Multi-Query Attention (MQA) & Grouped-Query Attention (GQA):
    • In standard Multi-Head Attention (MHA), serving long sequences requires storing Key-Value caches for all h heads, consuming huge GPU VRAM during autoregressive generation.
    • GQA (Ainslie et al., 2023): Groups Query heads (e.g. 32 Query heads sharing 8 Key/Value heads), slashing KV-cache memory by 75\% with zero loss in output quality (the standard in Llama 3).
  2. RoPE Context Window Extrapolation:
    • Techniques like YaRN and Linear RoPE Scaling allow models trained on 8k contexts to extrapolate to 128k+ tokens simply by adjusting the base frequency parameter \Theta.

Alternatives: free, open source, and commercial

ArchitectureAttention SchemeKV-Cache MemoryUsage
Standard MHAh Q heads, h K/V heads100\% BaselineOriginal Transformer, BERT
MQA (Shazeer 2019)h Q heads, 1 K/V head1/h (12.5\%)PaLM, Falcon
GQA (Llama 3)h Q heads, g K/V groupsg/h (25\%)SOTA LLMs (Llama 3, Mistral)
MLA (DeepSeek V2/V3)Low-Rank Compressed KV10\%High-efficiency foundation models

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   ATTENTION HEAD VARIANTS COMPARISON                   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Architecture       β”‚ Query Heads     β”‚ Key/Value Heads  β”‚ KV-Cache VRAMβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Single-Head        β”‚ 1               β”‚ 1                β”‚ Minimal      β”‚
β”‚ Multi-Head (MHA)   β”‚ h (e.g. 32)     β”‚ h (e.g. 32)      β”‚ High         β”‚
β”‚ Grouped-Query (GQA)β”‚ h (e.g. 32)     β”‚ g (e.g. 8)       β”‚ Medium-Low   β”‚
β”‚ Multi-Query (MQA)  β”‚ h (e.g. 32)     β”‚ 1                β”‚ Ultra-Low    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: the configuration decisions for multi-head attention

When to USE Multi-Head Attention:

When NOT to use:


The AI connection

Knowledge check

  1. Why does Multi-Head Attention split embeddings into h subspaces of dimension d_k?
  2. What happens if positional encodings are omitted from a Transformer model?
  3. How do Sinusoidal Positional Encodings enable linear relative distance shifts?
  4. How do Rotary Position Embeddings (RoPE) encode relative token distance via complex rotations?
  5. What is the difference between Multi-Head Attention (MHA) and Grouped-Query Attention (GQA)?

Hands-on exercise

In this lab, you will build and test a complete MultiHeadAttention module and Sinusoidal Positional Encoding generator in PyTorch: verify head splitting and recombination tensor shapes, implement causal and padding masks across all heads, generate sinusoidal positional encoding matrices, and verify multi-head attention weight distributions.

Expected output

[Multi-Head Self-Attention Verification Suite]
Testing MultiHeadAttention Forward Pass:
  Input Tensor: torch.Size([2, 8, 64]) (Batch=2, Seq=8, d_model=64)
  Configuration: num_heads = 4, d_k = 16
  Output Tensor: torch.Size([2, 8, 64]) [DIMENSIONS VERIFIED]
  Attention Weights Shape: torch.Size([2, 4, 8, 8]) [4 HEADS VERIFIED]
Testing Sinusoidal Positional Encodings:
  PE Matrix Shape: torch.Size([1, 16, 64])
  Geometric Wavelength Span: 6.28 to 62831.85 [VERIFIED]
Test Suite: 3 passed in 0.41s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Rotary Position Embedding (RoPE) in pure PyTorch and apply it to Query and Key tensors before computing attention.
  2. Implement Grouped-Query Attention (GQA) with 8 Query heads and 2 Key/Value heads.

Extension challenge

Implement FlashAttention-Style Online Softmax in Multi-Head Attention:

Quiz

Q1. Why does Multi-Head Attention split the total hidden dimension d_model into h parallel heads of dimension d_k = d_model / h rather than using a single giant attention head?

  1. To make the output matrix smaller
  2. Because a single attention head cannot run on GPUs
  3. Heads attend to different subspaces at once
  4. To eliminate the need for training weights
Show answer

Answer: C. Heads attend to different subspaces at once

Splitting into multiple heads allows different heads to specialize in distinct linguistic and semantic relationships across different subspaces.

Q2. What is the fundamental purpose of Positional Encodings in Transformer models?

  1. To normalize gradients to zero
  2. To encrypt the input text for extra security
  3. To reduce vocabulary size
  4. Attention is permutation-equivariant
Show answer

Answer: D. Attention is permutation-equivariant

Pure self-attention has no inherent concept of word order; positional encodings explicitly inject position information into token vectors.

Q3. How do Rotary Position Embeddings (RoPE / Su et al., 2021) encode token positions in modern LLMs like Llama 3 and Mistral?

  1. It rotates pairs by an angle proportional to position
  2. By adding a random integer to each word
  3. By sorting the words alphabetically
  4. By deleting every second token
Show answer

Answer: A. It rotates pairs by an angle proportional to position

RoPE applies complex rotations to query and key vectors such that their inner product naturally encodes relative positional offsets.

Q4. What is the computational cost of Multi-Head Attention compared to a Single-Head Attention layer of dimension d_model?

  1. Multi-head attention is 100 times slower
  2. Roughly identical β€” each head works on d_model / h
  3. Multi-head attention requires 100 times more parameters
  4. Single-head attention requires O(N^3) compute
Show answer

Answer: B. Roughly identical β€” each head works on d_model / h

Because d_k = d_model / h, computing h parallel heads of size d_k requires the exact same total FLOPs as one head of size d_model.

Q5. What does the final output projection matrix W_O in Multi-Head Attention accomplish?

  1. It computes the loss
  2. It normalizes the weights to zero
  3. It projects the concatenated heads back to d_model
  4. It tokenizes the output words
Show answer

Answer: C. It projects the concatenated heads back to d_model

W_O acts as an integration layer that blends the multi-head subspace features back into a unified representation vector.

Glossary

Multi-Head Attention (MHA)
An attention module that linearly projects queries, keys, and values into h parallel subspaces before performing attention.
Subspace
A lower-dimensional representation slice (dimension d_k = d_model / h) dedicated to capturing specific feature patterns.
Sinusoidal Positional Encoding
Deterministic geometric position vectors generated using sine and cosine functions across geometric frequency spectra.
Rotary Position Embedding (RoPE)
A relative position encoding method that rotates query and key vectors in 2D coordinate pairs by position-dependent angles.
Permutation Equivariance
A mathematical property where permuting input elements results in an identically permuted output sequence.
Output Projection (W_O)
A linear transformation layer that recombines the concatenated outputs of all attention heads.
Relative Position Encoding
A positional mechanism where attention depends on the relative distance (i - j) between tokens rather than absolute indices.
Head Dimension (d_k)
The dimensionality of an individual attention head, typically computed as d_model / num_heads (e.g. 768 / 12 = 64).

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.