Deep Learning › Sequences and Text › Day 221

Hands-on lab — Day 221: LSTMs and GRUs

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/lstms_and_grus_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/lstms_and_grus_lib.py
examples/test_lstms_and_grus_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/lstms_and_grus_lib.py
starter/test_lstms_and_grus_lib.py
tests/run_tests.sh
tests/test_lstms_and_grus_lib.py
troubleshooting.md

Lab README

Lab: Day 221 -- LSTMs and GRUs

Lesson

Day number: 221 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Gated Recurrent Units (LSTMs and GRUs).

Purpose

Build and test custom LSTM and GRU neural cells in PyTorch. Implement the mathematical gating equations for Forget, Input, Output, and Update gates from first principles, and verify hidden state and cell state propagation.

Learning objectives

  • Implement the 4-gate LSTM cell forward equations and Constant Error Carousel.
  • Implement the 2-gate GRU cell forward update equations.
  • Verify dimensional compatibility and activation ranges in recurrent states.
  • Compare parameter complexity between LSTM and GRU cell architectures.

Prerequisites

  • Day 220 (Recurrent Neural Networks).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/lstms_and_grus_lib.py: Student scaffold file.
  • examples/lstms_and_grus_lib.py: Complete reference implementation.
  • tests/test_lstms_and_grus_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/lstms_and_grus_lib.py

What the commands do

  • Executes forward passes through custom LSTM and GRU cells.
  • Verifies output tensor dimensions and forget gate bias initialization.
  • Runs unit test assertions.

Expected output

LSTM/GRU Demo: LSTM h Shape = torch.Size([4, 16]), GRU h Shape = torch.Size([4, 16])

Validation steps

  1. Verify CustomLSTMCell.forward returns valid h_t and c_t tensors.
  2. Confirm CustomGRUCell.forward returns valid h_t tensor matching hidden dimensions.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Dimensions mismatch in chunk: Ensure linear projection maps to 4 * hidden_dim for LSTM and 2 * hidden_dim for GRU reset/update gates.

Security notes

All neural gating routines execute locally in memory.

Extension exercises

  1. Implement a complete multi-layer Bidirectional LSTM classifier with PackedSequence support.
  2. Add peephole connections connecting cell state to gate activations.
  • Lesson title: LSTMs and GRUs
  • Day number: 221 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-221-lstms-and-grus
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-221-lstms-and-grus when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 221

- `LSTM h Shape`: Tensor shape of the output hidden state from LSTM cell.
- `GRU h Shape`: Tensor shape of the output hidden state from GRU cell.

examples-run.txt

LSTM/GRU Demo: LSTM h Shape = torch.Size([4, 16]), GRU h Shape = torch.Size([4, 16])

measured-values.txt

LSTM h Shape: torch.Size([4, 16])
GRU h Shape: torch.Size([4, 16])

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_lstms_and_grus_lib.py::test_lstm_cell_dimensions PASSED       [ 50%]
tests/test_lstms_and_grus_lib.py::test_gru_cell_dimensions PASSED        [100%]

============================== 2 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/lstms_and_grus_lib.py (2381 bytes)
import torch
import torch.nn as nn
from typing import Tuple

class CustomLSTMCell(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int):
        super().__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        # Fused gate linear projection: 4 * hidden_dim
        self.gates = nn.Linear(input_dim + hidden_dim, 4 * hidden_dim)
        # Initialize forget gate bias to 1.0
        with torch.no_grad():
            self.gates.bias[0:hidden_dim].fill_(1.0)

    def forward(self, x_t: torch.Tensor, states: Tuple[torch.Tensor, torch.Tensor]) -> Tuple[torch.Tensor, torch.Tensor]:
        h_prev, c_prev = states
        combined = torch.cat([x_t, h_prev], dim=1)
        gate_outputs = self.gates(combined)

        f_t, i_t, c_cand, o_t = torch.chunk(gate_outputs, 4, dim=1)
        
        f_t = torch.sigmoid(f_t)
        i_t = torch.sigmoid(i_t)
        c_cand = torch.tanh(c_cand)
        o_t = torch.sigmoid(o_t)

        c_t = f_t * c_prev + i_t * c_cand
        h_t = o_t * torch.tanh(c_t)
        return h_t, c_t

class CustomGRUCell(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int):
        super().__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.gate_rz = nn.Linear(input_dim + hidden_dim, 2 * hidden_dim)
        self.gate_cand = nn.Linear(input_dim + hidden_dim, hidden_dim)

    def forward(self, x_t: torch.Tensor, h_prev: torch.Tensor) -> torch.Tensor:
        combined = torch.cat([x_t, h_prev], dim=1)
        r_z = self.gate_rz(combined)
        r_t, z_t = torch.chunk(r_z, 2, dim=1)
        
        r_t = torch.sigmoid(r_t)
        z_t = torch.sigmoid(z_t)

        combined_cand = torch.cat([x_t, r_t * h_prev], dim=1)
        h_cand = torch.tanh(self.gate_cand(combined_cand))

        h_t = (1 - z_t) * h_prev + z_t * h_cand
        return h_t

def run_lstm_gru_demo():
    torch.manual_seed(42)
    lstm_cell = CustomLSTMCell(input_dim=8, hidden_dim=16)
    gru_cell = CustomGRUCell(input_dim=8, hidden_dim=16)

    x = torch.randn(4, 8)
    h0 = torch.zeros(4, 16)
    c0 = torch.zeros(4, 16)

    h_lstm, c_lstm = lstm_cell(x, (h0, c0))
    h_gru = gru_cell(x, h0)

    print(f"LSTM/GRU Demo: LSTM h Shape = {h_lstm.shape}, GRU h Shape = {h_gru.shape}")
    return h_lstm, c_lstm, h_gru

if __name__ == "__main__":
    run_lstm_gru_demo()
examples/test_lstms_and_grus_lib.py (610 bytes)
import pytest
import torch
from examples.lstms_and_grus_lib import CustomLSTMCell, CustomGRUCell

def test_lstm_cell_dimensions():
    torch.manual_seed(42)
    cell = CustomLSTMCell(input_dim=10, hidden_dim=20)
    x = torch.randn(2, 10)
    h0 = torch.zeros(2, 20)
    c0 = torch.zeros(2, 20)

    h1, c1 = cell(x, (h0, c0))
    assert h1.shape == (2, 20)
    assert c1.shape == (2, 20)

def test_gru_cell_dimensions():
    torch.manual_seed(42)
    cell = CustomGRUCell(input_dim=10, hidden_dim=20)
    x = torch.randn(3, 10)
    h0 = torch.zeros(3, 20)

    h1 = cell(x, h0)
    assert h1.shape == (3, 20)
metadata.yml (427 bytes)
lesson_id: D221
day: 221
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/lstms_and_grus_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/lstms_and_grus_lib.py (847 bytes)
import torch
import torch.nn as nn
from typing import Tuple

class CustomLSTMCell(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int):
        super().__init__()
        # TODO: Initialize linear projections for forget, input, candidate, output gates
        pass

    def forward(self, x_t: torch.Tensor, states: Tuple[torch.Tensor, torch.Tensor]) -> Tuple[torch.Tensor, torch.Tensor]:
        # TODO: Compute LSTM gating equations and return (h_t, C_t)
        pass

class CustomGRUCell(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int):
        super().__init__()
        # TODO: Initialize linear projections for reset, update, and candidate states
        pass

    def forward(self, x_t: torch.Tensor, h_prev: torch.Tensor) -> torch.Tensor:
        # TODO: Compute GRU equations and return h_t
        pass
starter/test_lstms_and_grus_lib.py (610 bytes)
import pytest
import torch
from examples.lstms_and_grus_lib import CustomLSTMCell, CustomGRUCell

def test_lstm_cell_dimensions():
    torch.manual_seed(42)
    cell = CustomLSTMCell(input_dim=10, hidden_dim=20)
    x = torch.randn(2, 10)
    h0 = torch.zeros(2, 20)
    c0 = torch.zeros(2, 20)

    h1, c1 = cell(x, (h0, c0))
    assert h1.shape == (2, 20)
    assert c1.shape == (2, 20)

def test_gru_cell_dimensions():
    torch.manual_seed(42)
    cell = CustomGRUCell(input_dim=10, hidden_dim=20)
    x = torch.randn(3, 10)
    h0 = torch.zeros(3, 20)

    h1 = cell(x, h0)
    assert h1.shape == (3, 20)
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 221 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_lstms_and_grus_lib.py (610 bytes)
import pytest
import torch
from examples.lstms_and_grus_lib import CustomLSTMCell, CustomGRUCell

def test_lstm_cell_dimensions():
    torch.manual_seed(42)
    cell = CustomLSTMCell(input_dim=10, hidden_dim=20)
    x = torch.randn(2, 10)
    h0 = torch.zeros(2, 20)
    c0 = torch.zeros(2, 20)

    h1, c1 = cell(x, (h0, c0))
    assert h1.shape == (2, 20)
    assert c1.shape == (2, 20)

def test_gru_cell_dimensions():
    torch.manual_seed(42)
    cell = CustomGRUCell(input_dim=10, hidden_dim=20)
    x = torch.randn(3, 10)
    h0 = torch.zeros(3, 20)

    h1 = cell(x, h0)
    assert h1.shape == (3, 20)

Troubleshooting

Troubleshooting: Day 221 - LSTMs and GRUs

Common Issues

  1. Forget gate saturating at zero:
    • Cause: Negative initial bias in forget gate.
    • Fix: Initialize forget gate bias to +1.0.

Security notes

Security & Privacy: Day 221 - LSTMs and GRUs

Security Guidance

  • All recurrent computations execute in local process memory.