Deep Learning › Sequences and Text › Day 220
Hands-on lab — Day 220: Recurrent Neural Networks
- ← Back to the Day 220 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-220-recurrent-neural-networks/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/recurrent_neural_networks_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/recurrent_neural_networks_lib.py examples/test_recurrent_neural_networks_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/recurrent_neural_networks_lib.py starter/test_recurrent_neural_networks_lib.py tests/run_tests.sh tests/test_recurrent_neural_networks_lib.py troubleshooting.md
Lab README
Lab: Day 220 -- Recurrent Neural Networks
Lesson
Day number: 220 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Recurrent Neural Networks and Sequential State Modeling.
Purpose
Build and test a custom Recurrent Neural Network (RNN) from first principles in PyTorch. Implement the recurrent hidden state transition equations, construct temporal sequence unrolling loops, and implement gradient norm clipping to stabilize recurrent optimization.
Learning objectives
- Implement simple recurrent cell state updates (W_xh, W_hh, W_hy).
- Unroll recurrent networks over arbitrary variable-length sequences.
- Compute global L2 gradient norms and apply gradient clipping.
- Verify gradient propagation stability through sequential backpropagation.
Prerequisites
- Day 219 (Word Embeddings).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/recurrent_neural_networks_lib.py: Student scaffold file.examples/recurrent_neural_networks_lib.py: Complete reference implementation.tests/test_recurrent_neural_networks_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/recurrent_neural_networks_lib.py
What the commands do
- Unrolls a custom RNN over multi-step sequential input tensors.
- Performs backpropagation and measures global gradient norm.
- Clips gradient norm to threshold C=1.0 and executes unit test assertions.
Expected output
RNN Demo: Logits Shape = torch.Size([2, 2]), Grad Norm = 2.1481
Validation steps
- Verify
SimpleRNNModel.forwardoutputs a tensor of shape(batch_size, num_classes). - Confirm
clip_gradient_normrescales large gradients to exactlymax_norm. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Exploding Loss to NaN: Ensure gradient clipping is called before
optimizer.step().
Security notes
All neural forward and backward passes execute in local memory.
Extension exercises
- Implement a bidirectional recurrent unrolling loop (BiRNN).
- Implement character-level language generation with temperature sampling.
Navigation
- Lesson title: Recurrent Neural Networks
- Day number: 220 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-220-recurrent-neural-networks
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-220-recurrent-neural-networkswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 220
- `Logits Shape`: Shape of the final classification projection tensor.
- `Grad Norm`: Global L2 norm of the model gradients before clipping.
examples-run.txt
RNN Demo: Logits Shape = torch.Size([2, 2]), Grad Norm = 2.1481
measured-values.txt
Logits Shape: torch.Size([2, 2])
Grad Norm: 2.1481
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_recurrent_neural_networks_lib.py::test_rnn_forward_pass_dimensions PASSED [ 50%]
tests/test_recurrent_neural_networks_lib.py::test_gradient_norm_clipping PASSED [100%]
============================== 2 passed in 0.35s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/recurrent_neural_networks_lib.py (1700 bytes)
import torch
import torch.nn as nn
import math
from typing import Tuple
class SimpleRNNModel(nn.Module):
def __init__(self, input_dim: int, hidden_dim: int, num_classes: int):
super().__init__()
self.hidden_dim = hidden_dim
self.W_xh = nn.Linear(input_dim, hidden_dim)
self.W_hh = nn.Linear(hidden_dim, hidden_dim, bias=False)
self.W_hy = nn.Linear(hidden_dim, num_classes)
def forward(self, x: torch.Tensor) -> torch.Tensor:
# x: (Batch, Seq_Len, input_dim)
batch_size, seq_len, _ = x.size()
h_t = torch.zeros(batch_size, self.hidden_dim, device=x.device)
for t in range(seq_len):
x_t = x[:, t, :]
h_t = torch.tanh(self.W_xh(x_t) + self.W_hh(h_t))
logits = self.W_hy(h_t)
return logits
def clip_gradient_norm(parameters, max_norm: float = 1.0) -> float:
params = [p for p in parameters if p.grad is not None]
if not params:
return 0.0
total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), 2) for p in params]), 2).item()
clip_coef = max_norm / (total_norm + 1e-6)
if clip_coef < 1.0:
for p in params:
p.grad.detach().mul_(clip_coef)
return total_norm
def run_rnn_demo():
torch.manual_seed(42)
model = SimpleRNNModel(input_dim=4, hidden_dim=8, num_classes=2)
x = torch.randn(2, 5, 4) # Batch=2, Seq=5, Dim=4
logits = model(x)
loss = logits.sum()
loss.backward()
norm = clip_gradient_norm(model.parameters(), max_norm=1.0)
print(f"RNN Demo: Logits Shape = {logits.shape}, Grad Norm = {norm:.4f}")
return logits.shape, norm
if __name__ == "__main__":
run_rnn_demo()
examples/test_recurrent_neural_networks_lib.py (924 bytes)
import pytest
import torch
from examples.recurrent_neural_networks_lib import SimpleRNNModel, clip_gradient_norm
def test_rnn_forward_pass_dimensions():
torch.manual_seed(42)
model = SimpleRNNModel(input_dim=8, hidden_dim=16, num_classes=3)
x = torch.randn(4, 10, 8)
logits = model(x)
assert logits.shape == (4, 3)
def test_gradient_norm_clipping():
torch.manual_seed(42)
model = SimpleRNNModel(input_dim=4, hidden_dim=4, num_classes=2)
x = torch.randn(2, 3, 4)
loss = model(x).sum() * 100.0 # Force huge gradient
loss.backward()
orig_norm = clip_gradient_norm(model.parameters(), max_norm=1.0)
assert orig_norm > 1.0
# Verify post-clip norm is <= 1.0
params = [p for p in model.parameters() if p.grad is not None]
post_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), 2) for p in params]), 2).item()
assert pytest.approx(post_norm, 1e-3) == 1.0
metadata.yml (438 bytes)
lesson_id: D220
day: 220
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/recurrent_neural_networks_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/recurrent_neural_networks_lib.py (560 bytes)
import torch
import torch.nn as nn
from typing import Tuple
class SimpleRNNModel(nn.Module):
def __init__(self, input_dim: int, hidden_dim: int, num_classes: int):
super().__init__()
# TODO: Initialize W_xh, W_hh, and W_hy linear projections
pass
def forward(self, x: torch.Tensor) -> torch.Tensor:
# TODO: Unroll sequence across time and return logits from final hidden state
pass
def clip_gradient_norm(parameters, max_norm: float = 1.0) -> float:
# TODO: Compute and clip global gradient norm
pass
starter/test_recurrent_neural_networks_lib.py (924 bytes)
import pytest
import torch
from examples.recurrent_neural_networks_lib import SimpleRNNModel, clip_gradient_norm
def test_rnn_forward_pass_dimensions():
torch.manual_seed(42)
model = SimpleRNNModel(input_dim=8, hidden_dim=16, num_classes=3)
x = torch.randn(4, 10, 8)
logits = model(x)
assert logits.shape == (4, 3)
def test_gradient_norm_clipping():
torch.manual_seed(42)
model = SimpleRNNModel(input_dim=4, hidden_dim=4, num_classes=2)
x = torch.randn(2, 3, 4)
loss = model(x).sum() * 100.0 # Force huge gradient
loss.backward()
orig_norm = clip_gradient_norm(model.parameters(), max_norm=1.0)
assert orig_norm > 1.0
# Verify post-clip norm is <= 1.0
params = [p for p in model.parameters() if p.grad is not None]
post_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), 2) for p in params]), 2).item()
assert pytest.approx(post_norm, 1e-3) == 1.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 220 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_recurrent_neural_networks_lib.py (924 bytes)
import pytest
import torch
from examples.recurrent_neural_networks_lib import SimpleRNNModel, clip_gradient_norm
def test_rnn_forward_pass_dimensions():
torch.manual_seed(42)
model = SimpleRNNModel(input_dim=8, hidden_dim=16, num_classes=3)
x = torch.randn(4, 10, 8)
logits = model(x)
assert logits.shape == (4, 3)
def test_gradient_norm_clipping():
torch.manual_seed(42)
model = SimpleRNNModel(input_dim=4, hidden_dim=4, num_classes=2)
x = torch.randn(2, 3, 4)
loss = model(x).sum() * 100.0 # Force huge gradient
loss.backward()
orig_norm = clip_gradient_norm(model.parameters(), max_norm=1.0)
assert orig_norm > 1.0
# Verify post-clip norm is <= 1.0
params = [p for p in model.parameters() if p.grad is not None]
post_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), 2) for p in params]), 2).item()
assert pytest.approx(post_norm, 1e-3) == 1.0
Troubleshooting
Troubleshooting: Day 220 - Recurrent Neural Networks
Common Issues
- Hidden state tensor dimension mismatch:
- Cause: Passing unbatched vectors to
W_xhorW_hh. - Fix: Ensure
h_thas shape(batch_size, hidden_dim).
- Cause: Passing unbatched vectors to
Security notes
Security & Privacy: Day 220 - Recurrent Neural Networks
Security Guidance
- All recurrent state computations execute locally without external dependencies.