Deep Learning › Sequences and Text › Day 221
Hands-on lab — Day 221: LSTMs and GRUs
- ← Back to the Day 221 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-221-lstms-and-grus/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/lstms_and_grus_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/lstms_and_grus_lib.py examples/test_lstms_and_grus_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/lstms_and_grus_lib.py starter/test_lstms_and_grus_lib.py tests/run_tests.sh tests/test_lstms_and_grus_lib.py troubleshooting.md
Lab README
Lab: Day 221 -- LSTMs and GRUs
Lesson
Day number: 221 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Gated Recurrent Units (LSTMs and GRUs).
Purpose
Build and test custom LSTM and GRU neural cells in PyTorch. Implement the mathematical gating equations for Forget, Input, Output, and Update gates from first principles, and verify hidden state and cell state propagation.
Learning objectives
- Implement the 4-gate LSTM cell forward equations and Constant Error Carousel.
- Implement the 2-gate GRU cell forward update equations.
- Verify dimensional compatibility and activation ranges in recurrent states.
- Compare parameter complexity between LSTM and GRU cell architectures.
Prerequisites
- Day 220 (Recurrent Neural Networks).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/lstms_and_grus_lib.py: Student scaffold file.examples/lstms_and_grus_lib.py: Complete reference implementation.tests/test_lstms_and_grus_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/lstms_and_grus_lib.py
What the commands do
- Executes forward passes through custom LSTM and GRU cells.
- Verifies output tensor dimensions and forget gate bias initialization.
- Runs unit test assertions.
Expected output
LSTM/GRU Demo: LSTM h Shape = torch.Size([4, 16]), GRU h Shape = torch.Size([4, 16])
Validation steps
- Verify
CustomLSTMCell.forwardreturns validh_tandc_ttensors. - Confirm
CustomGRUCell.forwardreturns validh_ttensor matching hidden dimensions. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Dimensions mismatch in chunk: Ensure linear projection maps to
4 * hidden_dimfor LSTM and2 * hidden_dimfor GRU reset/update gates.
Security notes
All neural gating routines execute locally in memory.
Extension exercises
- Implement a complete multi-layer Bidirectional LSTM classifier with PackedSequence support.
- Add peephole connections connecting cell state to gate activations.
Navigation
- Lesson title: LSTMs and GRUs
- Day number: 221 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-221-lstms-and-grus
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-221-lstms-and-gruswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 221
- `LSTM h Shape`: Tensor shape of the output hidden state from LSTM cell.
- `GRU h Shape`: Tensor shape of the output hidden state from GRU cell.
examples-run.txt
LSTM/GRU Demo: LSTM h Shape = torch.Size([4, 16]), GRU h Shape = torch.Size([4, 16])
measured-values.txt
LSTM h Shape: torch.Size([4, 16])
GRU h Shape: torch.Size([4, 16])
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_lstms_and_grus_lib.py::test_lstm_cell_dimensions PASSED [ 50%]
tests/test_lstms_and_grus_lib.py::test_gru_cell_dimensions PASSED [100%]
============================== 2 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/lstms_and_grus_lib.py (2381 bytes)
import torch
import torch.nn as nn
from typing import Tuple
class CustomLSTMCell(nn.Module):
def __init__(self, input_dim: int, hidden_dim: int):
super().__init__()
self.input_dim = input_dim
self.hidden_dim = hidden_dim
# Fused gate linear projection: 4 * hidden_dim
self.gates = nn.Linear(input_dim + hidden_dim, 4 * hidden_dim)
# Initialize forget gate bias to 1.0
with torch.no_grad():
self.gates.bias[0:hidden_dim].fill_(1.0)
def forward(self, x_t: torch.Tensor, states: Tuple[torch.Tensor, torch.Tensor]) -> Tuple[torch.Tensor, torch.Tensor]:
h_prev, c_prev = states
combined = torch.cat([x_t, h_prev], dim=1)
gate_outputs = self.gates(combined)
f_t, i_t, c_cand, o_t = torch.chunk(gate_outputs, 4, dim=1)
f_t = torch.sigmoid(f_t)
i_t = torch.sigmoid(i_t)
c_cand = torch.tanh(c_cand)
o_t = torch.sigmoid(o_t)
c_t = f_t * c_prev + i_t * c_cand
h_t = o_t * torch.tanh(c_t)
return h_t, c_t
class CustomGRUCell(nn.Module):
def __init__(self, input_dim: int, hidden_dim: int):
super().__init__()
self.input_dim = input_dim
self.hidden_dim = hidden_dim
self.gate_rz = nn.Linear(input_dim + hidden_dim, 2 * hidden_dim)
self.gate_cand = nn.Linear(input_dim + hidden_dim, hidden_dim)
def forward(self, x_t: torch.Tensor, h_prev: torch.Tensor) -> torch.Tensor:
combined = torch.cat([x_t, h_prev], dim=1)
r_z = self.gate_rz(combined)
r_t, z_t = torch.chunk(r_z, 2, dim=1)
r_t = torch.sigmoid(r_t)
z_t = torch.sigmoid(z_t)
combined_cand = torch.cat([x_t, r_t * h_prev], dim=1)
h_cand = torch.tanh(self.gate_cand(combined_cand))
h_t = (1 - z_t) * h_prev + z_t * h_cand
return h_t
def run_lstm_gru_demo():
torch.manual_seed(42)
lstm_cell = CustomLSTMCell(input_dim=8, hidden_dim=16)
gru_cell = CustomGRUCell(input_dim=8, hidden_dim=16)
x = torch.randn(4, 8)
h0 = torch.zeros(4, 16)
c0 = torch.zeros(4, 16)
h_lstm, c_lstm = lstm_cell(x, (h0, c0))
h_gru = gru_cell(x, h0)
print(f"LSTM/GRU Demo: LSTM h Shape = {h_lstm.shape}, GRU h Shape = {h_gru.shape}")
return h_lstm, c_lstm, h_gru
if __name__ == "__main__":
run_lstm_gru_demo()
examples/test_lstms_and_grus_lib.py (610 bytes)
import pytest
import torch
from examples.lstms_and_grus_lib import CustomLSTMCell, CustomGRUCell
def test_lstm_cell_dimensions():
torch.manual_seed(42)
cell = CustomLSTMCell(input_dim=10, hidden_dim=20)
x = torch.randn(2, 10)
h0 = torch.zeros(2, 20)
c0 = torch.zeros(2, 20)
h1, c1 = cell(x, (h0, c0))
assert h1.shape == (2, 20)
assert c1.shape == (2, 20)
def test_gru_cell_dimensions():
torch.manual_seed(42)
cell = CustomGRUCell(input_dim=10, hidden_dim=20)
x = torch.randn(3, 10)
h0 = torch.zeros(3, 20)
h1 = cell(x, h0)
assert h1.shape == (3, 20)
metadata.yml (427 bytes)
lesson_id: D221
day: 221
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/lstms_and_grus_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/lstms_and_grus_lib.py (847 bytes)
import torch
import torch.nn as nn
from typing import Tuple
class CustomLSTMCell(nn.Module):
def __init__(self, input_dim: int, hidden_dim: int):
super().__init__()
# TODO: Initialize linear projections for forget, input, candidate, output gates
pass
def forward(self, x_t: torch.Tensor, states: Tuple[torch.Tensor, torch.Tensor]) -> Tuple[torch.Tensor, torch.Tensor]:
# TODO: Compute LSTM gating equations and return (h_t, C_t)
pass
class CustomGRUCell(nn.Module):
def __init__(self, input_dim: int, hidden_dim: int):
super().__init__()
# TODO: Initialize linear projections for reset, update, and candidate states
pass
def forward(self, x_t: torch.Tensor, h_prev: torch.Tensor) -> torch.Tensor:
# TODO: Compute GRU equations and return h_t
pass
starter/test_lstms_and_grus_lib.py (610 bytes)
import pytest
import torch
from examples.lstms_and_grus_lib import CustomLSTMCell, CustomGRUCell
def test_lstm_cell_dimensions():
torch.manual_seed(42)
cell = CustomLSTMCell(input_dim=10, hidden_dim=20)
x = torch.randn(2, 10)
h0 = torch.zeros(2, 20)
c0 = torch.zeros(2, 20)
h1, c1 = cell(x, (h0, c0))
assert h1.shape == (2, 20)
assert c1.shape == (2, 20)
def test_gru_cell_dimensions():
torch.manual_seed(42)
cell = CustomGRUCell(input_dim=10, hidden_dim=20)
x = torch.randn(3, 10)
h0 = torch.zeros(3, 20)
h1 = cell(x, h0)
assert h1.shape == (3, 20)
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 221 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_lstms_and_grus_lib.py (610 bytes)
import pytest
import torch
from examples.lstms_and_grus_lib import CustomLSTMCell, CustomGRUCell
def test_lstm_cell_dimensions():
torch.manual_seed(42)
cell = CustomLSTMCell(input_dim=10, hidden_dim=20)
x = torch.randn(2, 10)
h0 = torch.zeros(2, 20)
c0 = torch.zeros(2, 20)
h1, c1 = cell(x, (h0, c0))
assert h1.shape == (2, 20)
assert c1.shape == (2, 20)
def test_gru_cell_dimensions():
torch.manual_seed(42)
cell = CustomGRUCell(input_dim=10, hidden_dim=20)
x = torch.randn(3, 10)
h0 = torch.zeros(3, 20)
h1 = cell(x, h0)
assert h1.shape == (3, 20)
Troubleshooting
Troubleshooting: Day 221 - LSTMs and GRUs
Common Issues
- Forget gate saturating at zero:
- Cause: Negative initial bias in forget gate.
- Fix: Initialize forget gate bias to
+1.0.
Security notes
Security & Privacy: Day 221 - LSTMs and GRUs
Security Guidance
- All recurrent computations execute in local process memory.