Deep Learning β€Ί Sequences and Text β€Ί Day 221

Day 221: LSTMs and GRUs

Day 221 of 365 β€” LSTMs and GRUs

Master gated sequence memory: derive Long Short-Term Memory (LSTM) gating equations and the Constant Error Carousel, understand the Gated Recurrent Unit (GRU) convex interpolation mechanism, compare computational complexity, handle variable-length sequence batching with packed padded sequences, and build a production BiLSTM in PyTorch.

Course
Deep Learning
Category
Sequences and Text
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-221-lstms-and-grus

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-221-lstms-and-grus
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Yesterday in Day 220, we uncovered the fundamental fatal pathology of simple vanilla Recurrent Neural Networks: The Vanishing Gradient Problem.

Because a standard RNN repeatedly multiplies hidden state gradients by the recurrent weight matrix $W_(hh)^\top$ at every single time step, error gradients decay exponentially toward zero over 10 to 15 steps. As a consequence:

In 1997, Sepp Hochreiter and JΓΌrgen Schmidhuber published a landmark breakthrough that revolutionized natural language processing, speech recognition, and time-series modeling: The Long Short-Term Memory (LSTM) network.

By introducing a dedicated Cell State Conveyor Belt ($C_t$) regulated by multiplicative Gating Mechanisms (Forget, Input, Output), LSTMs created the Constant Error Carousel (CEC) β€” an additive gradient highway that allows neural networks to remember dependencies across hundreds of time steps without vanishing gradients.

In 2014, Kyunghyun Cho et al. introduced the Gated Recurrent Unit (GRU), streamlining the LSTM into an efficient 2-gate architecture.

Today, you will master the mathematical derivations, internal gating mechanics, and PyTorch implementations of LSTMs and GRUs.


The idea in plain language

Imagine managing a company’s internal knowledge base over 20 years:

Information can travel down the archive untouched for 100 years unless explicitly deleted.


Historical background

Long Short-Term Memory LSTM cell internal architecture showing cell state conveyor belt forget gate input gate and output gate

  1. 1997 (Hochreiter & Schmidhuber): Introduced the original LSTM architecture to solve the vanishing gradient problem, proving that constant error carousels preserved gradients indefinitely.
  2. 2000 (Gers, Schmidhuber, & Cummins): Added the Forget Gate to LSTMs, allowing networks to actively reset internal cell states when a task or document ended.
  3. 2014 (Cho et al. & The Gated Recurrent Unit): Introduced the GRU for machine translation, proving that merging cell state and hidden state into a 2-gate structure reduced parameter counts by 25% while matching LSTM performance.
  4. 2015–2017 (The Golden Era of LSTMs): LSTMs powered Google Translate, Apple Siri, Amazon Alexa, and state-of-the-art machine translation until the rise of Transformers.

What it is β€” and what it is not

What LSTMs and GRUs ARE:

Diagram: the multiplicative recurrent path compared with the additive gated path

What they are NOT:


Why it was created and what problems it solves

Gated recurrent units solved three fundamental dilemmas in sequential machine learning:

  1. Vanishing Gradient Immunity: The derivative $\frac(\partial C_t)(\partial C_(t-1)) = f_t$. If the forget gate is near 1, error gradients pass backward through time completely unattenuated.
  2. Selective Forgetting: Explicitly clearing memory when context shifts (e.g. at sentence or paragraph boundaries).
  3. Multi-Scale Temporal Feature Extraction: Allowing slow-moving features (like document topic) to persist across hundreds of steps while fast-moving features (like part-of-speech tags) update every step.

How it works

Let us dissect the mathematical formulation of the LSTM and GRU gating systems.


1. The Full LSTM Gating Derivation

Diagram: the three gates of an LSTM cell and the cell state they control

At each time step $t$, given input $x_t \in R^(d_(\text(in)))$ and previous states $h_(t-1) \in R^(d_h), C_(t-1) \in R^(d_h)$:

1. Forget Gate:   f_t = sigma(W_f * [h_{t-1}, x_t] + b_f)
2. Input Gate:    i_t = sigma(W_i * [h_{t-1}, x_t] + b_i)
3. Candidate Cell: ~C_t = tanh(W_c * [h_{t-1}, x_t] + b_c)
4. Cell State:    C_t = f_t (x) C_{t-1} + i_t (x) ~C_t
5. Output Gate:   o_t = sigma(W_o * [h_{t-1}, x_t] + b_o)
6. Hidden State:  h_t = o_t (x) tanh(C_t)

where $\sigma(z) = \frac(1)(1 + e^(-z))$ is the sigmoid activation (outputs in range $[0, 1]$), and $\odot$ represents element-wise Hadamard multiplication.


2. The Gated Recurrent Unit (GRU) Derivation

Diagram: the LSTM compared with the gated recurrent unit

Gated Recurrent Unit GRU cell internal architecture showing reset gate update gate and candidate hidden state

The GRU eliminates the separate cell state $C_t$, maintaining only hidden state $h_t$:

1. Reset Gate:     r_t = sigma(W_r * [h_{t-1}, x_t] + b_r)
2. Update Gate:    z_t = sigma(W_z * [h_{t-1}, x_t] + b_z)
3. Candidate State: ~h_t = tanh(W_h * [r_t (x) h_{t-1}, x_t] + b_h)
4. Hidden State:   h_t = (1 - z_t) (x) h_{t-1} + z_t (x) ~h_t

3. LSTM vs GRU Parameter and Efficiency Comparison

For an input dimension $d_(\text(in))$ and hidden dimension $d_h$:


4. Handling Variable-Length Batches with PackedSequence

Diagram: why variable-length recurrent batches need packing rather than padding alone

When training on batches of sentences of varying lengths:

import torch
import torch.nn.utils.rnn as rnn_utils

# Given padded tensor of shape (Batch, Max_Seq_Len, Dim) and lengths list
lengths = [12, 8, 5, 3] # Must be sorted in descending order
packed_input = rnn_utils.pack_padded_sequence(padded_tensor, lengths, batch_first=True)

# Pass packed sequence directly to LSTM
packed_output, (h_n, c_n) = lstm(packed_input)

# Unpack back to rectangular padded tensor
output, output_lengths = rnn_utils.pad_packed_sequence(packed_output, batch_first=True)

Using PackedSequence prevents wasted compute on <pad> slots and guarantees that h_n reflects the true final non-pad token of each sentence.


5. LSTM with Peephole Connections

In standard LSTMs, the gates only observe $x_t$ and $h_(t-1)$; they have no direct access to the internal cell state $C_(t-1)$:


6. Highway Networks & ResNets: The Gated Legacy

Diagram: how the gated additive path propagated into later architectures

The discovery of the LSTM additive cell state directly inspired deep feedforward architectures:


7. Attention Pooling Over BiLSTM Hidden States

Diagram: three ways to summarise a sequence of recurrent hidden states

When classifying a document using a Bidirectional LSTM, taking only the final hidden state discards rich intermediate features:


8. Minimal Gated Units (MGU) and Quasi-RNNs (QRNN)

In high-throughput environments, researchers developed even leaner gated alternatives:


An everyday analogy

Diagram: a reservoir with controlled inflow and outflow as an analogy for gated memory

Think of a water filtration and reservoir station:

Because water flows smoothly through large pipes without being crushed through tiny pumps, the system operates continuously without pressure loss.


Examples in practice

Let us build a Modular Multi-Layer Bidirectional LSTM Classifier in PyTorch:

import torch
import torch.nn as nn
import torch.nn.utils.rnn as rnn_utils

class BiLSTMTextClassifier(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int, hidden_dim: int,
                 num_classes: int, num_layers: int = 2, dropout: float = 0.3):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(
            input_size=embed_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            bidirectional=True,
            batch_first=True,
            dropout=dropout if num_layers > 1 else 0.0
        )
        # Bidirectional concatenates forward and backward: hidden_dim * 2
        self.fc = nn.Sequential(
            nn.Dropout(p=dropout),
            nn.Linear(hidden_dim * 2, num_classes)
        )

    def forward(self, text_indices: torch.Tensor, lengths: torch.Tensor) -> torch.Tensor:
        # text_indices: (Batch, Max_Len)
        embeds = self.embedding(text_indices) # (Batch, Max_Len, embed_dim)

        # Pack variable-length padded sequence
        packed_embeds = rnn_utils.pack_padded_sequence(
            embeds, lengths.cpu(), batch_first=True, enforce_sorted=False
        )

        packed_out, (h_n, c_n) = self.lstm(packed_embeds)

        # Concatenate final forward and backward hidden states
        # h_n shape: (num_layers * 2, Batch, hidden_dim)
        forward_final = h_n[-2, :, :]
        backward_final = h_n[-1, :, :]
        bilstm_rep = torch.cat([forward_final, backward_final], dim=1) # (Batch, hidden_dim * 2)

        logits = self.fc(bilstm_rep)
        return logits

Implications: security, privacy, performance, scalability, and cost

Diagram: the memory, speed, streaming and capacity implications of gated recurrent networks

  1. CuDNN Hardware Acceleration:
    • Standard PyTorch nn.LSTM uses NVIDIA CuDNN fused C++ kernels, which are $10\times$ faster than custom loop implementations. If you write custom python loops over LSTMCell, GPU memory throughput drops drastically.
  2. Forget Gate Bias Initialization Trick:
    • In 2015, Jozefowicz et al. proved that initializing the Forget Gate bias vector to $+1.0$ (instead of $0.0$) forces $f_t \approx \sigma(1.0) \approx 0.73$ at the start of training, ensuring past information is preserved by default during early epochs.

Alternatives: free, open source, and commercial

ArchitectureGating MechanismRelative SpeedBest For
LSTM3 Gates (f, i, o)Baseline (1.0x)Complex long-term memory tasks
GRU2 Gates (r, z)1.3x FasterFast training on smaller text corpora
QRNN (Quasi-RNN)Convolutions + Gating4.0x FasterGPU-accelerated recurrent NLP
TransformerMulti-Head Self-AttentionHighly ScalableModern LLM generation

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   GATED RECURRENT REASONING MATRIX                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Property           β”‚ Vanilla RNN     β”‚ LSTM             β”‚ GRU          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ State Vectors      β”‚ Hidden h only   β”‚ Hidden h + Cell Cβ”‚ Hidden h onlyβ”‚
β”‚ Number of Gates    β”‚ 0               β”‚ 3 (f, i, o)      β”‚ 2 (r, z)     β”‚
β”‚ Parameter Scale    β”‚ 1x              β”‚ 4x               β”‚ 3x           β”‚
β”‚ Effective Horizon  β”‚ ~10 steps       β”‚ ~300 steps       β”‚ ~250 steps   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when LSTMs and GRUs remain the right choice

When to USE LSTMs / GRUs:

When NOT to use:


The AI connection

Knowledge check

  1. What is the role of the Constant Error Carousel (CEC) in LSTM memory retention?
  2. What are the three gates of an LSTM and what mathematical role does each play?
  3. How does the GRU simplify the LSTM architecture while preserving long-term memory?
  4. Why should you initialize the Forget Gate bias to $+1.0$ at the start of training?
  5. Why does PyTorch pack_padded_sequence improve training efficiency on variable-length text?

Hands-on exercise

In this lab, you will build and test an LSTM and GRU neural toolkit in PyTorch: implement custom LSTM and GRU cell update formulations from scratch, build a variable-length PackedSequence text processing pipeline, train a Bidirectional LSTM on a sequential classification task, and verify gradient propagation across long temporal horizons.

Expected output

[LSTM and GRU Verification Suite]
Testing Custom LSTM Cell vs GRU Cell Forward Step:
  LSTM Cell Output: Hidden Shape=(4, 32), Cell Shape=(4, 32)
  GRU Cell Output:  Hidden Shape=(4, 32) (25% Fewer Parameters Confirmed)
Testing Variable-Length PackedSequence Pipeline:
  Batch Sequences (Lengths: [15, 10, 6, 3]):
  Packed Tensor Verification: Shape=(34, 16) [PADDING SLOTS BYPASSED]
Testing Bidirectional LSTM Classification Harness:
  Epoch 1/5: Loss = 0.6842 -> Epoch 5/5: Loss = 0.1523 [CONVERGED]
  Test Accuracy: 96.5%
Test Suite: 3 passed in 0.38s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Peephole Connections (Gers & Schmidhuber, 2000) in an LSTM cell, allowing gate layers to inspect the current cell state $C_(t-1)$ directly.
  2. Build an Audio Keyword Spotting (KWS) Model using a 2-layer GRU that classifies 1-second audio MFCC spectrogram sequences into target voice commands.

Extension challenge

Implement Custom CUDA-Equivalent Fused LSTM Cell in PyTorch C++ Extension:

Quiz

Q1. How does the LSTM Cell State (C_t) eliminate the vanishing gradient problem that plagues vanilla RNNs?

  1. It multiplies all gradients by 1000
  2. The cell state adds rather than multiplies, so gradients survive
  3. It deletes negative numbers
  4. It stops backpropagation after 5 steps
Show answer

Answer: B. The cell state adds rather than multiplies, so gradients survive

The additive identity gradient path of the cell state allows error signals to flow back across time without exponential decay.

Q2. What is the role of the Forget Gate (f_t) in an LSTM cell?

  1. It deletes corrupted input text files
  2. It resets the model weights to random noise
  3. A sigmoid deciding how much of C_{t-1} to keep
  4. It clears GPU cache memory
Show answer

Answer: C. A sigmoid deciding how much of C_{t-1} to keep

The forget gate uses a sigmoid activation to selectively attenuate or preserve past cell state information.

Q3. What is the primary architectural simplification that distinguishes a Gated Recurrent Unit (GRU) from an LSTM?

  1. The GRU requires no matrix multiplications whatsoever
  2. The GRU removes all activation functions
  3. The GRU only processes numbers, not text
  4. The GRU merges the states and uses two gates, not three
Show answer

Answer: D. The GRU merges the states and uses two gates, not three

GRUs eliminate the separate cell state and use a single update gate to interpolate between past hidden state and candidate state, saving 25% of weights.

Q4. Why is torch.nn.utils.rnn.pack_padded_sequence used when feeding variable-length padded sentence batches into PyTorch RNNs/LSTMs?

  1. It skips padding, so pads never corrupt the final state
  2. It encrypts the model tensors on disk
  3. It converts float32 tensors to int8
  4. It sorts the vocabulary alphabetically
Show answer

Answer: A. It skips padding, so pads never corrupt the final state

PackedSequence objects store only valid sequence elements and their lengths, skipping redundant computations on padding slots.

Q5. In a Bidirectional LSTM (BiLSTM) with hidden_dim = 128, what is the dimension of the concatenated output hidden state vector at each time step?

  1. 128
  2. 256
  3. 512
  4. 64
Show answer

Answer: B. 256

A bidirectional layer concatenates forward and backward hidden state vectors, doubling the feature dimension (128 * 2 = 256).

Glossary

LSTM
Long Short-Term Memory: a recurrent neural architecture featuring an additive cell state conveyor belt regulated by three multiplicative gates.
Cell State
The persistent internal memory highway (C_t) of an LSTM that transports information across long sequence horizons with minimal attenuation.
Forget Gate
A sigmoid gate (f_t) in an LSTM that controls what fraction of past cell state memory is discarded versus retained.
Input Gate
A sigmoid gate (i_t) in an LSTM that modulates how much new candidate information is written into the cell state.
Output Gate
A sigmoid gate (o_t) in an LSTM that filters the non-linear activated cell state into the visible hidden state h_t.
GRU
Gated Recurrent Unit: a simplified 2-gate recurrent architecture (Reset and Update gates) that interpolates directly between hidden states.
PackedSequence
A PyTorch memory layout that packs variable-length padded sequences to skip redundant computation on padding tokens.
Constant Error Carousel (CEC)
The theoretical principle in LSTMs ensuring unit derivative flow through linear recurrent connections during backpropagation.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.