Deep Learning › Sequences and Text › Day 220

Hands-on lab — Day 220: Recurrent Neural Networks

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/recurrent_neural_networks_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/recurrent_neural_networks_lib.py
examples/test_recurrent_neural_networks_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/recurrent_neural_networks_lib.py
starter/test_recurrent_neural_networks_lib.py
tests/run_tests.sh
tests/test_recurrent_neural_networks_lib.py
troubleshooting.md

Lab README

Lab: Day 220 -- Recurrent Neural Networks

Lesson

Day number: 220 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Recurrent Neural Networks and Sequential State Modeling.

Purpose

Build and test a custom Recurrent Neural Network (RNN) from first principles in PyTorch. Implement the recurrent hidden state transition equations, construct temporal sequence unrolling loops, and implement gradient norm clipping to stabilize recurrent optimization.

Learning objectives

  • Implement simple recurrent cell state updates (W_xh, W_hh, W_hy).
  • Unroll recurrent networks over arbitrary variable-length sequences.
  • Compute global L2 gradient norms and apply gradient clipping.
  • Verify gradient propagation stability through sequential backpropagation.

Prerequisites

  • Day 219 (Word Embeddings).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/recurrent_neural_networks_lib.py: Student scaffold file.
  • examples/recurrent_neural_networks_lib.py: Complete reference implementation.
  • tests/test_recurrent_neural_networks_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/recurrent_neural_networks_lib.py

What the commands do

  • Unrolls a custom RNN over multi-step sequential input tensors.
  • Performs backpropagation and measures global gradient norm.
  • Clips gradient norm to threshold C=1.0 and executes unit test assertions.

Expected output

RNN Demo: Logits Shape = torch.Size([2, 2]), Grad Norm = 2.1481

Validation steps

  1. Verify SimpleRNNModel.forward outputs a tensor of shape (batch_size, num_classes).
  2. Confirm clip_gradient_norm rescales large gradients to exactly max_norm.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Exploding Loss to NaN: Ensure gradient clipping is called before optimizer.step().

Security notes

All neural forward and backward passes execute in local memory.

Extension exercises

  1. Implement a bidirectional recurrent unrolling loop (BiRNN).
  2. Implement character-level language generation with temperature sampling.
  • Lesson title: Recurrent Neural Networks
  • Day number: 220 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-220-recurrent-neural-networks
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-220-recurrent-neural-networks when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 220

- `Logits Shape`: Shape of the final classification projection tensor.
- `Grad Norm`: Global L2 norm of the model gradients before clipping.

examples-run.txt

RNN Demo: Logits Shape = torch.Size([2, 2]), Grad Norm = 2.1481

measured-values.txt

Logits Shape: torch.Size([2, 2])
Grad Norm: 2.1481

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_recurrent_neural_networks_lib.py::test_rnn_forward_pass_dimensions PASSED [ 50%]
tests/test_recurrent_neural_networks_lib.py::test_gradient_norm_clipping PASSED       [100%]

============================== 2 passed in 0.35s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/recurrent_neural_networks_lib.py (1700 bytes)
import torch
import torch.nn as nn
import math
from typing import Tuple

class SimpleRNNModel(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int, num_classes: int):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.W_xh = nn.Linear(input_dim, hidden_dim)
        self.W_hh = nn.Linear(hidden_dim, hidden_dim, bias=False)
        self.W_hy = nn.Linear(hidden_dim, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: (Batch, Seq_Len, input_dim)
        batch_size, seq_len, _ = x.size()
        h_t = torch.zeros(batch_size, self.hidden_dim, device=x.device)

        for t in range(seq_len):
            x_t = x[:, t, :]
            h_t = torch.tanh(self.W_xh(x_t) + self.W_hh(h_t))

        logits = self.W_hy(h_t)
        return logits

def clip_gradient_norm(parameters, max_norm: float = 1.0) -> float:
    params = [p for p in parameters if p.grad is not None]
    if not params:
        return 0.0
    total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), 2) for p in params]), 2).item()
    clip_coef = max_norm / (total_norm + 1e-6)
    if clip_coef < 1.0:
        for p in params:
            p.grad.detach().mul_(clip_coef)
    return total_norm

def run_rnn_demo():
    torch.manual_seed(42)
    model = SimpleRNNModel(input_dim=4, hidden_dim=8, num_classes=2)
    x = torch.randn(2, 5, 4) # Batch=2, Seq=5, Dim=4
    logits = model(x)
    
    loss = logits.sum()
    loss.backward()
    norm = clip_gradient_norm(model.parameters(), max_norm=1.0)

    print(f"RNN Demo: Logits Shape = {logits.shape}, Grad Norm = {norm:.4f}")
    return logits.shape, norm

if __name__ == "__main__":
    run_rnn_demo()
examples/test_recurrent_neural_networks_lib.py (924 bytes)
import pytest
import torch
from examples.recurrent_neural_networks_lib import SimpleRNNModel, clip_gradient_norm

def test_rnn_forward_pass_dimensions():
    torch.manual_seed(42)
    model = SimpleRNNModel(input_dim=8, hidden_dim=16, num_classes=3)
    x = torch.randn(4, 10, 8)
    logits = model(x)
    assert logits.shape == (4, 3)

def test_gradient_norm_clipping():
    torch.manual_seed(42)
    model = SimpleRNNModel(input_dim=4, hidden_dim=4, num_classes=2)
    x = torch.randn(2, 3, 4)
    loss = model(x).sum() * 100.0 # Force huge gradient
    loss.backward()

    orig_norm = clip_gradient_norm(model.parameters(), max_norm=1.0)
    assert orig_norm > 1.0

    # Verify post-clip norm is <= 1.0
    params = [p for p in model.parameters() if p.grad is not None]
    post_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), 2) for p in params]), 2).item()
    assert pytest.approx(post_norm, 1e-3) == 1.0
metadata.yml (438 bytes)
lesson_id: D220
day: 220
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/recurrent_neural_networks_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/recurrent_neural_networks_lib.py (560 bytes)
import torch
import torch.nn as nn
from typing import Tuple

class SimpleRNNModel(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int, num_classes: int):
        super().__init__()
        # TODO: Initialize W_xh, W_hh, and W_hy linear projections
        pass

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # TODO: Unroll sequence across time and return logits from final hidden state
        pass

def clip_gradient_norm(parameters, max_norm: float = 1.0) -> float:
    # TODO: Compute and clip global gradient norm
    pass
starter/test_recurrent_neural_networks_lib.py (924 bytes)
import pytest
import torch
from examples.recurrent_neural_networks_lib import SimpleRNNModel, clip_gradient_norm

def test_rnn_forward_pass_dimensions():
    torch.manual_seed(42)
    model = SimpleRNNModel(input_dim=8, hidden_dim=16, num_classes=3)
    x = torch.randn(4, 10, 8)
    logits = model(x)
    assert logits.shape == (4, 3)

def test_gradient_norm_clipping():
    torch.manual_seed(42)
    model = SimpleRNNModel(input_dim=4, hidden_dim=4, num_classes=2)
    x = torch.randn(2, 3, 4)
    loss = model(x).sum() * 100.0 # Force huge gradient
    loss.backward()

    orig_norm = clip_gradient_norm(model.parameters(), max_norm=1.0)
    assert orig_norm > 1.0

    # Verify post-clip norm is <= 1.0
    params = [p for p in model.parameters() if p.grad is not None]
    post_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), 2) for p in params]), 2).item()
    assert pytest.approx(post_norm, 1e-3) == 1.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 220 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_recurrent_neural_networks_lib.py (924 bytes)
import pytest
import torch
from examples.recurrent_neural_networks_lib import SimpleRNNModel, clip_gradient_norm

def test_rnn_forward_pass_dimensions():
    torch.manual_seed(42)
    model = SimpleRNNModel(input_dim=8, hidden_dim=16, num_classes=3)
    x = torch.randn(4, 10, 8)
    logits = model(x)
    assert logits.shape == (4, 3)

def test_gradient_norm_clipping():
    torch.manual_seed(42)
    model = SimpleRNNModel(input_dim=4, hidden_dim=4, num_classes=2)
    x = torch.randn(2, 3, 4)
    loss = model(x).sum() * 100.0 # Force huge gradient
    loss.backward()

    orig_norm = clip_gradient_norm(model.parameters(), max_norm=1.0)
    assert orig_norm > 1.0

    # Verify post-clip norm is <= 1.0
    params = [p for p in model.parameters() if p.grad is not None]
    post_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), 2) for p in params]), 2).item()
    assert pytest.approx(post_norm, 1e-3) == 1.0

Troubleshooting

Troubleshooting: Day 220 - Recurrent Neural Networks

Common Issues

  1. Hidden state tensor dimension mismatch:
    • Cause: Passing unbatched vectors to W_xh or W_hh.
    • Fix: Ensure h_t has shape (batch_size, hidden_dim).

Security notes

Security & Privacy: Day 220 - Recurrent Neural Networks

Security Guidance

  • All recurrent state computations execute locally without external dependencies.