Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Training Deep Networks › Day 209

Hands-on lab — Day 209: Debugging Training Runs

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/debugging_training_runs_lib.py

Test

./tests/run_tests.sh

File tree

examples/debugging_training_runs_lib.py
examples/test_debugging_training_runs_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/debugging_training_runs_lib.py
starter/test_debugging_training_runs_lib.py
tests/run_tests.sh
tests/test_debugging_training_runs_lib.py
troubleshooting.md

Lab README

Lab: Day 209 -- Debugging Training Runs

Lesson

Day number: 209 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Debugging Training Runs in PyTorch.

Purpose

Build and test a comprehensive neural network debugging suite in PyTorch. Implement the single-batch overfitting sanity test, derive and implement custom gradient norm clipping, and diagnose gradient vanishing/explosion pathologies.

Learning objectives

  • Implement the single-batch overfitting sanity check to verify computational graph correctness.
  • Implement gradient norm calculation and clipping from scratch.
  • Verify bitwise gradient alignment with torch.nn.utils.clip_grad_norm_.
  • Apply diagnostic protocols for hunting NaNs and uncalibrated initial losses.

Prerequisites

  • Day 208 (Dropout, Batch Norm, and Regularization).
  • Python 3.11+ with PyTorch.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is free and open-source under the modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/debugging_training_runs_lib.py: Student scaffold file.
  • examples/debugging_training_runs_lib.py: Complete reference implementation.
  • tests/test_debugging_training_runs_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/debugging_training_runs_lib.py

What the commands do

  • Executes a single-batch overfit diagnostic test on a multi-layer model.
  • Evaluates gradient norm scaling and clipping.
  • Verifies mathematical gradient integrity.

Expected output

Debugging Demo: Single-Batch Overfit Passed = True, Raw Norm = 14.82, Clipped Norm = 1.00

Validation steps

  1. Verify that single-batch overfitting drives cross-entropy loss below 0.01.
  2. Confirm that custom_clip_grad_norm scales gradients to exactly max_norm.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Overfit Test Fails to Reach 100%: Ensure optimizer.zero_grad() is called before loss.backward().

Security notes

All debugging routines execute in local memory on CPU hardware.

Extension exercises

  1. Implement forward/backward hooks that log activation sparsity across layers.
  2. Build an automated learning rate finder tracking loss derivatives.
  • Lesson title: Debugging Training Runs
  • Day number: 209 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-209-debugging-training-runs
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-209-debugging-training-runs when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 209

- `Single-Batch Overfit Passed`: Boolean indicating single batch overfit success.
- `Raw Norm`: Unclipped global gradient norm.
- `Clipped Norm`: Gradient norm after applying custom clipping.

examples-run.txt

Debugging Demo: Single-Batch Overfit Passed = True, Raw Norm = 14.82, Clipped Norm = 1.00

measured-values.txt

Single-Batch Overfit Passed: True
Raw Norm: 14.82
Clipped Norm: 1.00

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_debugging_training_runs_lib.py::test_compute_global_gradient_norm PASSED [ 33%]
tests/test_debugging_training_runs_lib.py::test_custom_clip_grad_norm_matches_torch PASSED [ 66%]
tests/test_debugging_training_runs_lib.py::test_single_batch_overfit_test_success PASSED [100%]

============================== 3 passed in 0.22s ===============================

Source files

examples/debugging_training_runs_lib.py (2462 bytes) 
import torch
import torch.nn as nn
from typing import Tuple, List, Dict, Any

def compute_global_gradient_norm(model: nn.Module) -> float:
    total_norm_sq = 0.0
    for p in model.parameters():
        if p.grad is not None:
            total_norm_sq += float((p.grad.detach() ** 2).sum().item())
    return float(total_norm_sq ** 0.5)

def custom_clip_grad_norm(model: nn.Module, max_norm: float = 1.0) -> float:
    total_norm = compute_global_gradient_norm(model)
    if total_norm > max_norm and total_norm > 0:
        clip_coef = max_norm / (total_norm + 1e-6)
        with torch.no_grad():
            for p in model.parameters():
                if p.grad is not None:
                    p.grad.mul_(clip_coef)
    return total_norm

def single_batch_overfit_test(model: nn.Module, in_features: int = 32,
                              num_classes: int = 10, batch_size: int = 16,
                              max_steps: int = 50) -> bool:
    torch.manual_seed(42)
    x = torch.randn(batch_size, in_features)
    y = torch.randint(0, num_classes, (batch_size,))

    optimizer = torch.optim.AdamW(model.parameters(), lr=0.02)
    criterion = nn.CrossEntropyLoss()

    model.train()
    for step in range(max_steps):
        optimizer.zero_grad()
        logits = model(x)
        loss = criterion(logits, y)
        loss.backward()
        custom_clip_grad_norm(model, max_norm=1.0)
        optimizer.step()

        preds = torch.argmax(logits, dim=1)
        acc = float((preds == y).float().mean().item())

        if loss.item() < 0.01 and acc == 1.0:
            return True

    return False

def run_debugging_demo():
    torch.manual_seed(42)
    model = nn.Sequential(
        nn.Linear(32, 64),
        nn.ReLU(),
        nn.Linear(64, 10)
    )
    passed = single_batch_overfit_test(model, in_features=32, num_classes=10, batch_size=16)

    # Test gradient clipping
    x = torch.randn(10, 32)
    y = torch.randint(0, 10, (10,))
    model.zero_grad()
    loss = nn.CrossEntropyLoss()(model(x), y) * 100.0 # Artificially inflate loss
    loss.backward()

    raw_norm = compute_global_gradient_norm(model)
    custom_clip_grad_norm(model, max_norm=1.0)
    clipped_norm = compute_global_gradient_norm(model)

    print(f"Debugging Demo: Single-Batch Overfit Passed = {passed}, Raw Norm = {raw_norm:.2f}, Clipped Norm = {clipped_norm:.2f}")
    return passed, raw_norm, clipped_norm

if __name__ == "__main__":
    run_debugging_demo()
examples/test_debugging_training_runs_lib.py (1616 bytes) 
import pytest
import torch
import torch.nn as nn
from examples.debugging_training_runs_lib import (
    compute_global_gradient_norm, custom_clip_grad_norm, single_batch_overfit_test
)

def test_compute_global_gradient_norm():
    model = nn.Sequential(nn.Linear(2, 2, bias=False))
    model[0].weight.grad = torch.tensor([[3.0, 4.0], [0.0, 0.0]])
    # Norm = sqrt(3^2 + 4^2) = 5.0
    norm = compute_global_gradient_norm(model)
    assert pytest.approx(norm, abs=1e-5) == 5.0

def test_custom_clip_grad_norm_matches_torch():
    torch.manual_seed(42)
    model1 = nn.Sequential(nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 2))
    model2 = nn.Sequential(nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 2))

    # Set identical weights
    model2.load_state_dict(model1.state_dict())

    x = torch.randn(16, 10)
    y = torch.randint(0, 2, (16,))

    loss1 = nn.CrossEntropyLoss()(model1(x), y) * 50.0
    loss2 = nn.CrossEntropyLoss()(model2(x), y) * 50.0

    loss1.backward()
    loss2.backward()

    norm1 = custom_clip_grad_norm(model1, max_norm=1.0)
    norm2 = float(torch.nn.utils.clip_grad_norm_(model2.parameters(), max_norm=1.0).item())

    assert pytest.approx(norm1, abs=1e-4) == norm2
    # Verify parameter gradients match
    for p1, p2 in zip(model1.parameters(), model2.parameters()):
        assert torch.allclose(p1.grad, p2.grad, atol=1e-5)

def test_single_batch_overfit_test_success():
    model = nn.Sequential(
        nn.Linear(16, 32),
        nn.ReLU(),
        nn.Linear(32, 4)
    )
    assert single_batch_overfit_test(model, in_features=16, num_classes=4, batch_size=8, max_steps=40)
metadata.yml (436 bytes) 
lesson_id: D209
day: 209
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/debugging_training_runs_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (27 bytes) 
torch>=2.2.0
pytest>=8.0.0
starter/debugging_training_runs_lib.py (629 bytes) 
import torch
import torch.nn as nn
from typing import Tuple, List, Dict, Any

def compute_global_gradient_norm(model: nn.Module) -> float:
    # TODO: Compute total Euclidean norm across all parameter gradients
    pass

def custom_clip_grad_norm(model: nn.Module, max_norm: float = 1.0) -> float:
    # TODO: Implement gradient norm clipping from scratch
    pass

def single_batch_overfit_test(model: nn.Module, in_features: int = 32,
                              num_classes: int = 10, batch_size: int = 16,
                              max_steps: int = 50) -> bool:
    # TODO: Implement single-batch overfit test
    pass
starter/test_debugging_training_runs_lib.py (1616 bytes) 
import pytest
import torch
import torch.nn as nn
from examples.debugging_training_runs_lib import (
    compute_global_gradient_norm, custom_clip_grad_norm, single_batch_overfit_test
)

def test_compute_global_gradient_norm():
    model = nn.Sequential(nn.Linear(2, 2, bias=False))
    model[0].weight.grad = torch.tensor([[3.0, 4.0], [0.0, 0.0]])
    # Norm = sqrt(3^2 + 4^2) = 5.0
    norm = compute_global_gradient_norm(model)
    assert pytest.approx(norm, abs=1e-5) == 5.0

def test_custom_clip_grad_norm_matches_torch():
    torch.manual_seed(42)
    model1 = nn.Sequential(nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 2))
    model2 = nn.Sequential(nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 2))

    # Set identical weights
    model2.load_state_dict(model1.state_dict())

    x = torch.randn(16, 10)
    y = torch.randint(0, 2, (16,))

    loss1 = nn.CrossEntropyLoss()(model1(x), y) * 50.0
    loss2 = nn.CrossEntropyLoss()(model2(x), y) * 50.0

    loss1.backward()
    loss2.backward()

    norm1 = custom_clip_grad_norm(model1, max_norm=1.0)
    norm2 = float(torch.nn.utils.clip_grad_norm_(model2.parameters(), max_norm=1.0).item())

    assert pytest.approx(norm1, abs=1e-4) == norm2
    # Verify parameter gradients match
    for p1, p2 in zip(model1.parameters(), model2.parameters()):
        assert torch.allclose(p1.grad, p2.grad, atol=1e-5)

def test_single_batch_overfit_test_success():
    model = nn.Sequential(
        nn.Linear(16, 32),
        nn.ReLU(),
        nn.Linear(32, 4)
    )
    assert single_batch_overfit_test(model, in_features=16, num_classes=4, batch_size=8, max_steps=40)
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 209 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_debugging_training_runs_lib.py (1616 bytes) 
import pytest
import torch
import torch.nn as nn
from examples.debugging_training_runs_lib import (
    compute_global_gradient_norm, custom_clip_grad_norm, single_batch_overfit_test
)

def test_compute_global_gradient_norm():
    model = nn.Sequential(nn.Linear(2, 2, bias=False))
    model[0].weight.grad = torch.tensor([[3.0, 4.0], [0.0, 0.0]])
    # Norm = sqrt(3^2 + 4^2) = 5.0
    norm = compute_global_gradient_norm(model)
    assert pytest.approx(norm, abs=1e-5) == 5.0

def test_custom_clip_grad_norm_matches_torch():
    torch.manual_seed(42)
    model1 = nn.Sequential(nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 2))
    model2 = nn.Sequential(nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 2))

    # Set identical weights
    model2.load_state_dict(model1.state_dict())

    x = torch.randn(16, 10)
    y = torch.randint(0, 2, (16,))

    loss1 = nn.CrossEntropyLoss()(model1(x), y) * 50.0
    loss2 = nn.CrossEntropyLoss()(model2(x), y) * 50.0

    loss1.backward()
    loss2.backward()

    norm1 = custom_clip_grad_norm(model1, max_norm=1.0)
    norm2 = float(torch.nn.utils.clip_grad_norm_(model2.parameters(), max_norm=1.0).item())

    assert pytest.approx(norm1, abs=1e-4) == norm2
    # Verify parameter gradients match
    for p1, p2 in zip(model1.parameters(), model2.parameters()):
        assert torch.allclose(p1.grad, p2.grad, atol=1e-5)

def test_single_batch_overfit_test_success():
    model = nn.Sequential(
        nn.Linear(16, 32),
        nn.ReLU(),
        nn.Linear(32, 4)
    )
    assert single_batch_overfit_test(model, in_features=16, num_classes=4, batch_size=8, max_steps=40)

Troubleshooting

Troubleshooting: Day 209 - Debugging Training Runs

Common Issues

  1. Loss explodes to NaN:
    • Cause: High learning rate or taking logarithm of 0.
    • Fix: Use torch.autograd.set_detect_anomaly(True) and apply clip_grad_norm_.

Security notes

Security & Privacy: Day 209 - Debugging Training Runs

Security Guidance

  • All diagnostic scripts run locally in memory without telemetric logging.