Deep Learning β€Ί Transformers β€Ί Day 227

Day 227: The Transformer Architecture

Day 227 of 365 β€” The Transformer Architecture

Master the full Transformer block and Encoder-Decoder architecture: understand Pre-LN vs Post-LN Layer Normalization, Residual Connections, Position-Wise Feedforward Networks (FFN / MLP with 4x dimension expansion and GELU/SwiGLU), Cross-Attention mechanisms, and build a full stacked Transformer block in PyTorch.

Course
Deep Learning
Category
Transformers
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-227-the-transformer-architecture

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-227-the-transformer-architecture
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

In Days 225 and 226, you mastered the core components of the Transformer: Scaled Dot-Product Attention, Multi-Head subspace projections, and Positional Encodings.

However, Multi-Head Attention by itself is only a linear spatial mixer. If you only stack raw self-attention layers on top of each other:

To turn Multi-Head Attention into a scalable foundation model capable of deep reasoning, Vaswani et al. combined attention with three crucial architectural pillars:

  1. Position-Wise Feedforward Networks (FFN / MLP): Expanding hidden representations by 4\times to perform dense non-linear feature synthesis.
  2. Residual Connections (x + \text{SubLayer}(x)): Creating direct identity highways that preserve gradient flow across 100+ stacked layers.
  3. Layer Normalization (Pre-LN vs Post-LN): Stabilizing activation statistics across deep layer cascades.

Today, you will assemble all of these components into the complete, production-grade Transformer Block and Encoder-Decoder Architecture.


The idea in plain language

Diagram: attention as a spatial mixer compared with the feedforward network as a channel mixer

Diagram: the four components of a transformer block

Think of an international diplomatic summit drafting a complex global treaty:


Historical background

Complete Transformer Encoder Decoder architecture diagram showing stacked attention blocks residual connections feed forward networks and cross attention

  1. 2017 (Vaswani et al. & Post-LN Architecture): The original Transformer used Post-LN (applying LayerNorm after the residual addition). While highly effective on translation, training required an intricate learning rate warmup schedule to prevent early divergence.
  2. 2020 (Xiong et al. & The Pre-LN Revolution): Demonstrated mathematically that placing LayerNorm before the sublayer (Pre-LN) provided an uninterrupted identity gradient highway, allowing 100+ layer Transformers to train stably without warmup.
  3. 2020 (Noam Shazeer & SwiGLU): Published β€œGLU Variants Improve Transformer”, replacing standard ReLU/GELU activations in the FFN with Swish-Gated Linear Units, becoming the standard in modern LLMs (LLaMA, Mistral, PaLM).

What it is β€” and what it is not

What a Complete Transformer Block IS:

What it is NOT:


Why it was created and what problems it solves

The complete Transformer block solved three fundamental challenges in deep sequential modeling:

  1. Separation of Communication and Computation: Multi-Head Attention handles inter-token communication; the FFN handles internal token computation.
  2. Factual Memory Storage (MLP as Key-Value Memories): Research by Geva et al. (2021) showed that the 4\times expanded FFN layers act as associative memory banks where factual knowledge (e.g. β€œEiffel Tower is in Paris”) is stored and retrieved.
  3. Stable Deep Scaling: Pre-LN residual connections allow scaling model depth to hundreds of layers without training instability.

How it works

Let us dissect the three core components that complete the Transformer block.


1. Position-Wise Feedforward Networks (FFN)

Diagram: the structure and parameter share of the position-wise feedforward network

After Multi-Head Attention mixes token representations, each token vector x_i \in \mathbb{R}^{d_{\text{model}}} is passed independently through a two-layer multi-layer perceptron:

\text{FFN}(x) = \text{Activation}(x W_1 + b_1) W_2 + b_2

2. Pre-LN vs. Post-LN Layer Normalization

Diagram: post-normalisation compared with pre-normalisation in a transformer block

Pre-LN versus Post-LN Transformer block architecture comparison demonstrating direct identity residual gradient flow

Where should Layer Normalization be placed inside the block?

1. Post-LN (Original Vaswani 2017):

x_{l+1} = \text{LayerNorm}(x_l + \text{SubLayer}(x_l))

2. Pre-LN (Modern Standard):

x_{l+1} = x_l + \text{SubLayer}(\text{LayerNorm}(x_l))

3. Cross-Attention Mechanics in Encoder-Decoder Models

In full sequence-to-sequence translation models (T5, BART, Whisper):

  1. The Encoder processes the source sequence X_{\text{src}} and outputs memory representations H_{\text{enc}} \in \mathbb{R}^{N_{\text{src}} \times d_{\text{model}}}.
  2. The Decoder contains two distinct attention sublayers in each block:
    • Sublayer 1: Masked Multi-Head Self-Attention: Causal self-attention over previously generated target tokens.

    • Sublayer 2: Multi-Head Cross-Attention:

      Q = H_{\text{dec}} W_Q, \quad K = H_{\text{enc}} W_K, \quad V = H_{\text{enc}} W_V
      \text{CrossAttention}(Q, K, V) = \text{Softmax}\left( \frac{Q K^\top}{\sqrt{d_k}} \right) V
    • Decoder Queries (Q) search across Encoder Keys (K) and retrieve weighted Encoder Values (V).


4. Root Mean Square Normalization (RMSNorm / Zhang & Sennrich, 2019)

Diagram: four refinements to the standard transformer block

Standard LayerNorm computes both the mean \mu and variance \sigma^2 across the hidden dimension:

\text{LN}(x) = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \odot \gamma + \beta

5. SwiGLU Feedforward Gating Derivation

In traditional MLPs, activation is applied after linear projection: \text{FFN}(x) = \text{GELU}(x W_1) W_2.


6. DeepNorm and Sandwich-LN for 1,000-Layer Transformers

When scaling Transformers beyond 100 layers:


7. Sparse Mixture-of-Experts (MoE) Transformer Blocks

Diagram: how sparse expert routing decouples parameter count from per-token compute

To scale model parameter capacity without increasing per-token FLOPs:


8. LayerScale: Diagonal Scaling for Extreme Depth (Touvron et al., 2021)

When training deep Vision Transformers (CaiT) and deep LLMs beyond 50 layers:


9. Regularization & Dropout Placement in Transformer Blocks

Diagram: the three dropout positions in a transformer block

A standard Transformer block incorporates three distinct dropout mechanisms:

  1. Embedding Dropout: Applied immediately after adding token and positional embeddings (p=0.1).
  2. Attention Weight Dropout: Applied to the softmax attention probability matrix A = \text{Softmax}(Q K^\top / \sqrt{d_k}), randomly masking out full attention connections (p=0.1).
  3. Sublayer Residual Dropout: Applied to the output of Multi-Head Attention and FFN projections before the residual addition (p=0.1).

An everyday analogy

Diagram: a compound lens as an analogy for stacked transformer blocks

Think of a high-end camera lens assembly:


Examples in practice

Let us build a Complete Pre-LN Transformer Encoder Layer in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F

class TransformerEncoderLayer(nn.Module):
    def __init__(self, d_model: int, num_heads: int, d_ffn: int = None,
                 dropout: float = 0.1):
        super().__init__()
        if d_ffn is None:
            d_ffn = 4 * d_model

        # Sublayer 1: Multi-Head Self-Attention with Pre-LN
        self.norm1 = nn.LayerNorm(d_model)
        self.self_attn = nn.MultiheadAttention(
            embed_dim=d_model,
            num_heads=num_heads,
            dropout=dropout,
            batch_first=True
        )
        self.dropout1 = nn.Dropout(dropout)

        # Sublayer 2: Position-Wise Feedforward Network with Pre-LN
        self.norm2 = nn.LayerNorm(d_model)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ffn),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(d_ffn, d_model),
            nn.Dropout(dropout)
        )

    def forward(self, x: torch.Tensor, mask: torch.Tensor = None) -> torch.Tensor:
        # Pre-LN Block 1: Self-Attention with Residual Highway
        normed_x = self.norm1(x)
        attn_out, _ = self.self_attn(
            query=normed_x,
            key=normed_x,
            value=normed_x,
            key_padding_mask=mask
        )
        x = x + self.dropout1(attn_out)

        # Pre-LN Block 2: Position-Wise FFN with Residual Highway
        x = x + self.ffn(self.norm2(x))

        return x

Implications: security, privacy, performance, scalability, and cost

  1. Memory Complexity in Stacked Transformers:
    • In a 32-layer Transformer, backpropagation requires caching intermediate activations for all 32 layers during the forward pass.
    • Activation Checkpointing (Gradient Checkpointing): Discards intermediate activations during the forward pass and recomputes them on-the-fly during the backward pass, reducing VRAM usage by 70\% at the cost of a 25\% compute overhead.
  2. Weight Tying (Press & Wolf, 2017):
    • Sharing parameter weights between the input embedding matrix E and the final output classification head W_{\text{vocab}} saves tens of millions of parameters in large-vocabulary models.

Alternatives: free, open source, and commercial

ArchitectureParadigmProcessing ModeBest For
Encoder-Only (BERT)Bidirectional Pre-LNNon-AutoregressiveClassification, NER, Embedding Search
Decoder-Only (GPT)Causal AutoregressiveLeft-to-Right RolloutText Generation, Coding, Chat LLMs
Encoder-Decoder (T5)Bidirectional + CausalSequence-to-SequenceTranslation, Summarization, ASR
Hybrid SSM (Mamba)State Space ModelLinear Time O(N)Streaming long-context processing

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   TRANSFORMER BLOCK DESIGN COMPARISON                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Variant            β”‚ Norm Placement  β”‚ FFN Activation   β”‚ Depth Scalingβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Original (2017)    β”‚ Post-LN         β”‚ ReLU             β”‚ Fragile >12L β”‚
β”‚ BERT / GPT-2 (2018)β”‚ Post-LN         β”‚ GELU             β”‚ Moderate     β”‚
β”‚ Modern GPT (2020)  β”‚ Pre-LN          β”‚ GELU             β”‚ 100+ Layers  β”‚
β”‚ LLaMA 3 (2024)     β”‚ RMSNorm Pre-LN  β”‚ SwiGLU           β”‚ 128+ Layers  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when stacked transformer blocks are appropriate

When to USE Stacked Transformer Blocks:

When NOT to use:


The AI connection

Knowledge check

  1. Why does Pre-LN provide greater training stability than Post-LN in deep Transformers?
  2. What is the Position-Wise Feedforward Network (FFN) and why is its hidden dimension expanded 4\times?
  3. How do Residual Connections create uninhibited identity gradient highways?
  4. In Cross-Attention, what provides the Query tensor versus the Key/Value tensors?
  5. How does Activation Checkpointing reduce peak GPU VRAM consumption during training?

Hands-on exercise

In this lab, you will build and test a complete TransformerEncoderLayer and stacked TransformerEncoder in PyTorch: implement Pre-LN residual streams, construct the position-wise feedforward network with GELU activation and 4\times hidden expansion, handle padding masks across layers, and verify forward-pass output tensor stability.

Expected output

[Transformer Block Architecture Verification Suite]
Testing TransformerEncoderLayer Forward Pass:
  Input Tensor: torch.Size([2, 8, 32]) (Batch=2, Seq=8, d_model=32)
  Layer Configuration: num_heads = 4, d_ffn = 128 (4x Expansion)
  Output Tensor: torch.Size([2, 8, 32]) [SHAPES MATCH]
Testing Stacked TransformerEncoder (4 Layers):
  Deep Stack Forward Pass: torch.Size([2, 8, 32]) [STABLE]
  Residual Gradient Flow: Non-zero gradients present across all 4 layers [VERIFIED]
Testing Padding Masking:
  Padded positions correctly masked throughout all sublayers.
Test Suite: 3 passed in 0.40s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement SwiGLU Feedforward Layer in pure PyTorch and compare convergence against standard GELU FFN.
  2. Implement RMSNorm (Root Mean Square Normalization) and measure the speedup over standard LayerNorm.

Extension challenge

Implement Activation Checkpointing in a 12-Layer Transformer:

Quiz

Q1. Why does almost every modern foundation model (e.g. GPT-3/4, Llama 3, Mistral) use Pre-Layer Normalization (Pre-LN) instead of original Post-LN (Vaswani et al., 2017)?

  1. Pre-LN makes the model run on CPUs only
  2. Pre-LN keeps a clean residual path
  3. Pre-LN eliminates the need for attention matrices
  4. Post-LN requires 10 times more memory
Show answer

Answer: B. Pre-LN keeps a clean residual path

Pre-LN preserves an uninterrupted residual identity highway, enabling robust gradient backpropagation in deep architectures.

Q2. What is the structural role of the Position-Wise Feedforward Network (FFN / MLP) in a Transformer block?

  1. The FFN computes the learning rate
  2. The FFN sorts the tokens in alphabetical order
  3. Attention mixes tokens; the FFN transforms each
  4. The FFN encrypts the weights
Show answer

Answer: C. Attention mixes tokens; the FFN transforms each

Self-attention mixes information across tokens; the FFN operates position-wise to transform representations non-linearly.

Q3. In a Transformer Decoder layer, where do the Queries (Q), Keys (K), and Values (V) originate during Cross-Attention?

  1. They are randomly initialized at every generation step
  2. All three originate from the dataset labels
  3. Queries originate from the encoder; Keys and Values originate from the decoder
  4. Q from the decoder; K and V from the encoder
Show answer

Answer: D. Q from the decoder; K and V from the encoder

In cross-attention, decoder queries search against the memory keys and values produced by the encoder.

Q4. What is the SwiGLU activation function used in modern Transformer feedforward networks (LLaMA, PaLM) instead of standard ReLU?

  1. A Swish-gated linear unit
  2. A convolutional filter
  3. A replacement for the softmax function
  4. A linear activation with zero bias
Show answer

Answer: A. A Swish-gated linear unit

SwiGLU (Shazeer, 2020) provides superior gradient flow and empirical performance over standard ReLU/GELU activations.

Q5. Why are Residual Connections (He et al., 2016) essential to the functioning of deep Transformer stacks?

  1. They reduce the number of tokens in the sentence
  2. They give gradients a direct path to the embeddings
  3. They prevent the model from using GPU memory
  4. They eliminate the need for Layer Normalization
Show answer

Answer: B. They give gradients a direct path to the embeddings

Residual shortcuts prevent gradient vanishing, allowing gradients to propagate directly through deep Transformer stacks.

Glossary

Transformer Block
A fundamental repeating structural unit composed of a Multi-Head Attention layer, a Feedforward Network, Layer Normalization, and Residual Connections.
Position-Wise Feedforward Network (FFN)
A two-layer MLP applied identically and independently to each token position, expanding the hidden dimension by 4x.
Pre-LN (Pre-Layer Normalization)
A Transformer block topology where LayerNorm is applied to the input branch before Multi-Head Attention and FFN sublayers.
Post-LN (Post-Layer Normalization)
The original Vaswani topology where LayerNorm is applied directly after the residual addition.
Cross-Attention (Encoder-Decoder Attention)
An attention sublayer where decoder queries attend across encoder key-value memory representations.
Residual Connection
An additive skip connection (x + SubLayer(x)) that preserves gradient magnitude across deep layers.
SwiGLU
A Swish-gated linear unit activation function used in modern Transformer FFNs.
Layer Normalization
An operation that normalizes features across the embedding channel dimension independently for each token.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.