Deep Learning β€Ί Sequences and Text β€Ί Day 220

Day 220: Recurrent Neural Networks

Day 220 of 365 β€” Recurrent Neural Networks

Master stateful sequential computation: derive recurrent hidden state transition equations, unroll computational graphs across time, implement Backpropagation Through Time (BPTT) from scratch, analyze vanishing and exploding gradient mechanics, apply gradient norm clipping, and build a character-level sequence generator in PyTorch.

Course
Deep Learning
Category
Sequences and Text
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-220-recurrent-neural-networks

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-220-recurrent-neural-networks
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Feedforward neural networks (like MLPs and CNNs) operate under a rigid architectural constraint: they assume all input samples are independent and identically distributed (i.i.d.).

However, the vast majority of real-world information is fundamentally sequential and temporal:

If you pass a sentence into a feedforward network as a flat vector, the network cannot easily handle variable-length inputs and is blind to word order.

In 1990, cognitive scientist Jeffrey Elman introduced the Simple Recurrent Neural Network (Elman RNN). By adding a recurrent feedback loop that passes an internal Hidden State Vector ($h_t$) from one time step to the next, neural networks gained an internal memory buffer.

Today, you will master the mechanics of recurrent networks: unrolling computation graphs across time, deriving Backpropagation Through Time (BPTT), diagnosing the mathematical causes of vanishing and exploding gradients, implementing gradient norm clipping, and building a character-level sequence generator in PyTorch.


The idea in plain language

Diagram: how a recurrent network carries state across time steps

Imagine reading a thrilling murder mystery novel:


Historical background

Recurrent neural network folded loop representation unrolled into an explicit sequence of time steps with shared weight matrices

  1. 1986 (Rumelhart, Hinton, & Williams): Introduced Backpropagation Through Time (BPTT), showing that unrolling recurrent networks into deep feedforward graphs allowed standard gradient descent to train temporal states.
  2. 1990 (Jeffrey Elman & Elman RNNs): Introduced the simple recurrent network architecture ($h_t = \sigma(W_(hh) h_(t-1) + W_(xh) x_t)$), demonstrating that networks could learn grammar and syntactic word boundaries without explicit rules.
  3. 1994 (Bengio et al. & The Vanishing Gradient Proof): Yoshua Bengio et al. published the mathematical proof explaining why gradient descent struggled on long temporal sequences: repeated Jacobian matrix multiplications caused gradients to vanish exponentially.
  4. 2013 (Mikolov, Pascanu, & Gradient Clipping): Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio systematized gradient clipping to eliminate exploding gradients, enabling large-scale RNN training.

What it is β€” and what it is not

What an RNN IS:

What it is NOT:


Why it was created and what problems it solves

Recurrent networks resolved three critical limitations of classical feedforward architectures:

  1. Handling Arbitrary Variable Sequence Lengths: Feedforward networks require fixed input dimensions ($N$ inputs). An RNN processes sequences of length 3, 30, or 300 using the identical weight tensors.
  2. Temporal Equivariance: A phrase like "not good" produces a similar shift in hidden state whether it appears at word #2 or word #50.
  3. Flexible Sequence Topologies:
    • Many-to-One: Sentiment analysis (Sequence of words $\rightarrow$ Single sentiment class).
    • One-to-Many: Image captioning (Single image vector $\rightarrow$ Sequence of generated words).
    • Many-to-Many (Synchronized): Video frame classification or Part-of-Speech tagging.
    • Many-to-Many (Delayed / Seq2Seq): Machine translation (Full input sentence $\rightarrow$ Full output translation).

How it works

Let us dissect the mathematical formulation of recurrent forward propagation and Backpropagation Through Time.


1. Forward Propagation Equations

At each discrete time step $t \in [1, T]$:

  1. Receive input vector $x_t \in R^(d_(\text(in)))$ (e.g. word embedding) and previous hidden state $h_(t-1) \in R^(d_h)$.

  2. Compute the new hidden state using the non-linear hyperbolic tangent activation ($\tanh$):

    h_t = tanh(W_{xh} * x_t + W_{hh} * h_{t-1} + b_h)
  3. Compute the output prediction logits $\hat(y)t \in R^(d(\text(out)))$:

    \hat{y}_t = W_{hy} * h_t + b_y
  4. For multi-class classification / language modeling, compute the softmax probability distribution:

    p_t = softmax(\hat{y}_t)

2. Backpropagation Through Time (BPTT)

Diagram: how backpropagation through time unrolls a recurrent network

Backpropagation Through Time BPTT gradient propagation demonstrating exponential vanishing and exploding gradients across sequential steps

To train the shared parameters ($W_(xh), W_(hh), W_(hy)$), we compute total sequence loss $L = \sum_(t=1)^T L_t$.

By the multivariate chain rule, the gradient of loss at step $T$ with respect to the recurrent weight matrix $W_(hh)$ accumulates across all prior time steps:

partial L_T / partial W_{hh} = sum_{k=1}^T (partial L_T / partial h_T) * (partial h_T / partial h_k) * (partial h_k / partial W_{hh})

The crucial term is the temporal Jacobian product:

partial h_T / partial h_k = prod_{j=k+1}^T (partial h_j / partial h_{j-1}) = prod_{j=k+1}^T W_{hh}^\top * diag(1 - h_j^2)

3. The Mathematics of Vanishing and Exploding Gradients

Diagram: vanishing and exploding gradients in recurrent networks

The Jacobian product involves multiplying $T - k$ matrices together:


4. Gradient Norm Clipping

To eliminate exploding gradients, Mikolov et al. introduced L2 Gradient Norm Clipping:

  1. Compute global L2 norm of all network parameter gradients:

    ||g||_2 = sqrt( sum_i ||g_i||_2^2 )
  2. If $|g|_2 > C$ (where $C$ is a maximum threshold, typically $C = 1.0$ or $5.0$), rescale all gradients:

    g_clipped = g * (C / ||g||_2)

This preserves the exact descent direction while strictly capping the step magnitude.


5. Bidirectional Recurrent Neural Networks (BiRNN)

Diagram: unidirectional compared with bidirectional recurrent processing

In many NLP tasks (like text classification, named entity recognition, or translation encoding), the model has access to the entire sequence at once:


6. Deep Stacked Recurrent Networks & Recurrent Dropout

To increase expressive capacity, multiple recurrent layers are stacked vertically:


7. Teacher Forcing vs Exposure Bias

Diagram: teacher forcing during training compared with free-running generation

When training autoregressive sequence generators (e.g. character language models):


8. Modern Linear Recurrence: From RNNs to Mamba (SSMs)

Diagram: the trajectory of recurrent architectures from dominance to revival

In 2023–2024, researchers revived recurrent principles with Selective State Space Models (Mamba / Gu & Dao):


An everyday analogy

Diagram: message degradation along a chain as an analogy for recurrent memory

Think of a whispering game (Chinese Whispers / Telephone) played down a line of 50 people:


Examples in practice

Let us build a Vanilla RNN Cell from Scratch in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F

class CustomRNNCell(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int):
        super().__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim

        # Input-to-hidden and Hidden-to-hidden weights
        self.W_xh = nn.Linear(input_dim, hidden_dim)
        self.W_hh = nn.Linear(hidden_dim, hidden_dim, bias=False)

    def forward(self, x_t: torch.Tensor, h_prev: torch.Tensor) -> torch.Tensor:
        # x_t: (Batch, input_dim)
        # h_prev: (Batch, hidden_dim)
        h_t = torch.tanh(self.W_xh(x_t) + self.W_hh(h_prev))
        return h_t

class VanillaRNN(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int, output_dim: int):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.cell = CustomRNNCell(input_dim, hidden_dim)
        self.fc_out = nn.Linear(hidden_dim, output_dim)

    def forward(self, x: torch.Tensor, h_0: torch.Tensor = None) -> torch.Tensor:
        # x shape: (Batch, Seq_Len, input_dim)
        batch_size, seq_len, _ = x.size()

        if h_0 is None:
            h_t = torch.zeros(batch_size, self.hidden_dim, device=x.device)
        else:
            h_t = h_0

        # Unroll across time
        for t in range(seq_len):
            x_t = x[:, t, :]
            h_t = self.cell(x_t, h_t)

        # Output logits from final hidden state
        logits = self.fc_out(h_t)
        return logits

Implications: security, privacy, performance, scalability, and cost

Diagram: the parallelism, memory, length and inference implications of recurrent networks

  1. Sequential Compute Bottleneck on GPUs:
    • Unlike CNNs and Transformers which process all tokens in parallel, RNNs are strictly bound by sequential data dependencies: step $t$ must wait for step $t-1$. This leads to poor GPU compute utilization and slow training speeds on long documents.
  2. Truncated BPTT in Production:
    • When training RNNs on streaming audio or long texts, backpropagating over 10,000 steps exhausts GPU memory. Truncated BPTT splits sequences into windows of $k_1$ forward steps and $k_2$ backward steps, capping memory requirements.

Alternatives: free, open source, and commercial

ArchitectureMemory SpanParallelizabilityBest For
Vanilla RNNShort (5 - 10 steps)Sequential (Low)Microcontroller / Low-latency simple signals
LSTM / GRUMedium (100 - 500 steps)Sequential (Low)Classical NLP, speech, tabular time-series
1D CNN (WaveNet)Medium (Receptive field)Fully ParallelFast audio waveform generation
Transformer / MambaLong / InfiniteFully ParallelModern LLMs, long document reasoning

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   SEQUENCE MODELING EVOLUTION                          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Architecture       β”‚ State Mechanism β”‚ Gradient Highway β”‚ GPU Scaling  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Vanilla RNN        β”‚ Single h_t      β”‚ Vanishing (tanh) β”‚ Sequential   β”‚
β”‚ LSTM               β”‚ Hidden + Cell C β”‚ Constant Carouselβ”‚ Sequential   β”‚
β”‚ GRU                β”‚ Gated Single h  β”‚ Gated Highway    β”‚ Sequential   β”‚
β”‚ Transformer        β”‚ Self-Attention  β”‚ Residual Highway β”‚ Fully Para   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when recurrent architectures remain appropriate

When to USE Vanilla RNNs:

When NOT to use:


The AI connection

Knowledge check

  1. What is the fundamental recurrence equation defining a simple Elman RNN cell?
  2. Why are weight matrices shared across all time steps during sequence processing?
  3. What causes the vanishing gradient problem during Backpropagation Through Time (BPTT)?
  4. How does L2 gradient norm clipping prevent numerical overflow during RNN training?
  5. What is the sequential execution bottleneck that makes RNNs slower to train on GPUs than Transformers?

Hands-on exercise

In this lab, you will build and test a custom Recurrent Neural Network (RNN) in PyTorch: implement the recurrent cell forward equation from first principles, construct a sequence unrolling loop, implement an automated Gradient Norm Clipping utility, train a many-to-one sequence classifier on synthetic time-series data, and verify gradient backpropagation stability.

Expected output

[Recurrent Neural Network Verification Suite]
Testing Custom RNN Cell & Forward Unrolling:
  Input Sequence Tensor Shape: torch.Size([16, 10, 8]) (Batch=16, Steps=10, Dim=8)
  Final Hidden State Shape:    torch.Size([16, 32])
  Output Logits Shape:         torch.Size([16, 2])
Testing Gradient Norm Clipping Controller:
  Original Gradient Norm:  14.82 [EXPLODING GRADIENT SIMULATED]
  Clipped Gradient Norm:   1.00  [BOUNDED TO THRESHOLD C=1.0]
Testing Sequence Classification Training Loop:
  Epoch 1/5: Loss = 0.6931 -> Epoch 5/5: Loss = 0.2104 [TRAINING CONVERGED]
Test Suite: 3 passed in 0.35s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a Bidirectional Vanilla RNN (BiRNN) that processes the input sequence forward ($1 \rightarrow T$) and backward ($T \rightarrow 1$), concatenating both hidden states at each step.
  2. Build a Character-Level Text Generator that samples characters one by one with temperature scaling from an RNN trained on Shakespeare’s sonnets.

Extension challenge

Implement Exact Manual Backpropagation Through Time (BPTT) in Pure NumPy:

Quiz

Q1. What is the core mathematical mechanism that distinguishes a Recurrent Neural Network (RNN) from a standard Feedforward Multilayer Perceptron (MLP)?

  1. RNNs only process images
  2. RNNs use two-dimensional convolutional kernels instead
  3. RNNs have no activation functions
  4. A hidden state carried and updated at every step
Show answer

Answer: D. A hidden state carried and updated at every step

Recurrent networks maintain a hidden state vector passed from step to step, allowing information to persist across sequential inputs.

Q2. Why are the weight matrices W_xh, W_hh, and W_hy shared (identical) across all sequential time steps in an unrolled RNN?

  1. Sharing weights handles any length with fixed parameters
  2. It prevents PyTorch from running out of RAM
  3. Because backpropagation cannot calculate gradients for different matrices
  4. It converts the network into a Transformer
Show answer

Answer: A. Sharing weights handles any length with fixed parameters

Parameter sharing across time enforces temporal equivariance and allows the model to process sequences of arbitrary lengths.

Q3. What is the primary cause of the Vanishing Gradient problem during Backpropagation Through Time (BPTT) in vanilla RNNs?

  1. The learning rate is too high
  2. Repeated multiplication by W_hh shrinks gradients
  3. The vocabulary size is too small
  4. Using float32 instead of float64
Show answer

Answer: B. Repeated multiplication by W_hh shrinks gradients

Repeated Jacobian matrix products over long temporal horizons cause gradients to vanish exponentially, preventing long-range dependency learning.

Q4. How does Gradient Norm Clipping prevent training divergence when encountering exploding gradients in RNNs?

  1. It stops training immediately
  2. It sets all of the negative gradients to zero instead
  3. It rescales the gradient when its norm exceeds C
  4. It reduces the batch size to 1
Show answer

Answer: C. It rescales the gradient when its norm exceeds C

Gradient clipping rescales the gradient vector proportionally when its total L2 norm exceeds threshold C, preserving the optimization descent direction.

Q5. Which sequence modeling topology is used when an RNN takes a variable-length paragraph as input and predicts a single binary sentiment label?

  1. Many-to-Many (Synchronized)
  2. One-to-One
  3. One-to-Many
  4. Many-to-one, reading the final hidden state
Show answer

Answer: D. Many-to-one, reading the final hidden state

Many-to-one processes a full input sequence and uses the final time step hidden state for global sequence classification.

Glossary

Recurrent Neural Network (RNN)
A class of neural networks where connections between nodes form a directed cycle along a temporal sequence, enabling internal memory states.
Hidden State
A continuous vector (h_t) serving as the sequential memory buffer updated at each time step based on current input and past hidden state.
Backpropagation Through Time (BPTT)
The algorithm that unrolls an RNN across sequential time steps and applies the chain rule backward from the final loss to early steps.
Vanishing Gradient Problem
The exponential decay of error gradients as they backpropagate across multiple recurrent time steps, preventing long-range dependency learning.
Exploding Gradient Problem
The exponential growth of error gradients across time steps, causing catastrophic numerical overflow and model divergence.
Gradient Norm Clipping
A regularization technique that rescales gradient vectors whose L2 norm exceeds a maximum threshold C to prevent optimization instability.
Truncated BPTT
An engineering practice of splitting long sequences into chunks of k time steps, limiting backpropagation depth to bound compute and memory.
Temporal Equivariance
The mathematical property ensuring that a sequential pattern is processed identically regardless of its absolute timestamp in the sequence.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.