Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Training Deep Networks › Day 207

Hands-on lab — Day 207: Learning Rate Schedules

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/learning_rate_schedules_lib.py

Test

./tests/run_tests.sh

File tree

examples/learning_rate_schedules_lib.py
examples/test_learning_rate_schedules_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/learning_rate_schedules_lib.py
starter/test_learning_rate_schedules_lib.py
tests/run_tests.sh
tests/test_learning_rate_schedules_lib.py
troubleshooting.md

Lab README

Lab: Day 207 -- Learning Rate Schedules

Lesson

Day number: 207 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Learning Rate Schedules in PyTorch.

Purpose

Build and test dynamic learning rate scheduling policies in PyTorch. Implement a custom Warmup + Cosine Annealing scheduler using torch.optim.lr_scheduler.LambdaLR, verify linear warmup scaling and smooth cosine decay, and ensure checkpoint state persistence.

Learning objectives

  • Implement custom learning rate schedules with torch.optim.lr_scheduler.LambdaLR.
  • Derive linear warmup formulas for early gradient stabilization.
  • Apply half-period cosine decay for late-stage convergence.
  • Save and restore scheduler state dictionaries across training runs.

Prerequisites

  • Day 206 (Optimizers: SGD to Adam).
  • Python 3.11+ with PyTorch.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is free and open-source under the modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/learning_rate_schedules_lib.py: Student scaffold file.
  • examples/learning_rate_schedules_lib.py: Complete reference implementation.
  • tests/test_learning_rate_schedules_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/learning_rate_schedules_lib.py

What the commands do

  • Executes a 100-step training simulation with Warmup + Cosine Annealing.
  • Logs learning rate values across warmup, peak, and decay phases.
  • Verifies smooth mathematical convergence.

Expected output

Scheduler Demo: Start LR = 0.000000, Peak LR = 0.010000, Final LR = 0.000100

Validation steps

  1. Verify that learning rate starts at 0.0 and rises linearly to peak at step 20.
  2. Confirm that learning rate decays smoothly to min_lr at step 100.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Learning Rate Remains Zero: Ensure division in warmup uses float() to avoid integer truncation in Python.

Security notes

All scheduling logic runs locally in memory on CPU hardware.

Extension exercises

  1. Implement CosineAnnealingWarmRestarts with cyclical period multipliers (T_mult = 2).
  2. Add metric-based stepping with ReduceLROnPlateau.
  • Lesson title: Learning Rate Schedules
  • Day number: 207 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-207-learning-rate-schedules
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-207-learning-rate-schedules when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 207

- `Start LR`: Learning rate at step 0.
- `Peak LR`: Maximum learning rate achieved at the end of warmup.
- `Final LR`: Annealed learning rate at the final step.

examples-run.txt

Scheduler Demo: Start LR = 0.000000, Peak LR = 0.010000, Final LR = 0.000100

measured-values.txt

Start LR: 0.000000
Peak LR: 0.010000
Final LR: 0.000100

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_learning_rate_schedules_lib.py::test_warmup_cosine_scheduler_warmup_phase PASSED [ 33%]
tests/test_learning_rate_schedules_lib.py::test_warmup_cosine_scheduler_final_decay PASSED [ 66%]
tests/test_learning_rate_schedules_lib.py::test_scheduler_state_dict_persistence PASSED [100%]

============================== 3 passed in 0.22s ===============================

Source files

examples/learning_rate_schedules_lib.py (1562 bytes) 
import torch
import math
from torch.optim.lr_scheduler import LambdaLR
from typing import List, Tuple, Dict, Any

def create_warmup_cosine_scheduler(optimizer: torch.optim.Optimizer,
                                    warmup_steps: int,
                                    total_steps: int,
                                    min_lr_ratio: float = 0.01) -> LambdaLR:
    def lr_lambda(current_step: int) -> float:
        if current_step < warmup_steps:
            return float(current_step) / float(max(1, warmup_steps))
        progress = float(current_step - warmup_steps) / float(max(1, total_steps - warmup_steps))
        cosine_decay = 0.5 * (1.0 + math.cos(math.pi * progress))
        return min_lr_ratio + (1.0 - min_lr_ratio) * cosine_decay

    return LambdaLR(optimizer, lr_lambda)

def run_scheduler_demo():
    w = torch.tensor([1.0], requires_grad=True)
    base_lr = 0.01
    optimizer = torch.optim.SGD([w], lr=base_lr)
    total_steps = 100
    warmup_steps = 20

    scheduler = create_warmup_cosine_scheduler(
        optimizer, warmup_steps=warmup_steps, total_steps=total_steps, min_lr_ratio=0.01
    )

    lrs = []
    for step in range(total_steps + 1):
        current_lr = scheduler.get_last_lr()[0]
        lrs.append(current_lr)
        optimizer.zero_grad()
        loss = w ** 2
        loss.backward()
        optimizer.step()
        scheduler.step()

    print(f"Scheduler Demo: Start LR = {lrs[0]:.6f}, Peak LR = {lrs[20]:.6f}, Final LR = {lrs[-1]:.6f}")
    return lrs

if __name__ == "__main__":
    run_scheduler_demo()
examples/test_learning_rate_schedules_lib.py (1774 bytes) 
import pytest
import torch
from examples.learning_rate_schedules_lib import create_warmup_cosine_scheduler

def test_warmup_cosine_scheduler_warmup_phase():
    w = torch.tensor([1.0], requires_grad=True)
    opt = torch.optim.SGD([w], lr=0.1)
    sched = create_warmup_cosine_scheduler(opt, warmup_steps=10, total_steps=100, min_lr_ratio=0.01)

    assert sched.get_last_lr()[0] == 0.0
    for _ in range(5):
        sched.step()
    # At step 5 (halfway through warmup), lr should be 0.05
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.05

    for _ in range(5):
        sched.step()
    # At step 10 (end of warmup), lr should be peak 0.1
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.1

def test_warmup_cosine_scheduler_final_decay():
    w = torch.tensor([1.0], requires_grad=True)
    opt = torch.optim.SGD([w], lr=0.1)
    sched = create_warmup_cosine_scheduler(opt, warmup_steps=10, total_steps=100, min_lr_ratio=0.01)

    for _ in range(100):
        sched.step()

    # At final step 100, lr should decay to min_lr_ratio * base_lr = 0.001
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.001

def test_scheduler_state_dict_persistence():
    w1 = torch.tensor([1.0], requires_grad=True)
    opt1 = torch.optim.SGD([w1], lr=0.1)
    sched1 = create_warmup_cosine_scheduler(opt1, warmup_steps=20, total_steps=100)

    for _ in range(50):
        sched1.step()
    lr_at_50 = sched1.get_last_lr()[0]

    state = sched1.state_dict()

    w2 = torch.tensor([1.0], requires_grad=True)
    opt2 = torch.optim.SGD([w2], lr=0.1)
    sched2 = create_warmup_cosine_scheduler(opt2, warmup_steps=20, total_steps=100)
    sched2.load_state_dict(state)

    assert pytest.approx(sched2.get_last_lr()[0], abs=1e-5) == lr_at_50
metadata.yml (436 bytes) 
lesson_id: D207
day: 207
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/learning_rate_schedules_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (27 bytes) 
torch>=2.2.0
pytest>=8.0.0
starter/learning_rate_schedules_lib.py (441 bytes) 
import torch
import math
from torch.optim.lr_scheduler import LambdaLR
from typing import List, Tuple, Dict, Any

def create_warmup_cosine_scheduler(optimizer: torch.optim.Optimizer,
                                    warmup_steps: int,
                                    total_steps: int,
                                    min_lr_ratio: float = 0.01) -> LambdaLR:
    # TODO: Implement Warmup + Cosine Decay LambdaLR scheduler
    pass
starter/test_learning_rate_schedules_lib.py (1774 bytes) 
import pytest
import torch
from examples.learning_rate_schedules_lib import create_warmup_cosine_scheduler

def test_warmup_cosine_scheduler_warmup_phase():
    w = torch.tensor([1.0], requires_grad=True)
    opt = torch.optim.SGD([w], lr=0.1)
    sched = create_warmup_cosine_scheduler(opt, warmup_steps=10, total_steps=100, min_lr_ratio=0.01)

    assert sched.get_last_lr()[0] == 0.0
    for _ in range(5):
        sched.step()
    # At step 5 (halfway through warmup), lr should be 0.05
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.05

    for _ in range(5):
        sched.step()
    # At step 10 (end of warmup), lr should be peak 0.1
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.1

def test_warmup_cosine_scheduler_final_decay():
    w = torch.tensor([1.0], requires_grad=True)
    opt = torch.optim.SGD([w], lr=0.1)
    sched = create_warmup_cosine_scheduler(opt, warmup_steps=10, total_steps=100, min_lr_ratio=0.01)

    for _ in range(100):
        sched.step()

    # At final step 100, lr should decay to min_lr_ratio * base_lr = 0.001
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.001

def test_scheduler_state_dict_persistence():
    w1 = torch.tensor([1.0], requires_grad=True)
    opt1 = torch.optim.SGD([w1], lr=0.1)
    sched1 = create_warmup_cosine_scheduler(opt1, warmup_steps=20, total_steps=100)

    for _ in range(50):
        sched1.step()
    lr_at_50 = sched1.get_last_lr()[0]

    state = sched1.state_dict()

    w2 = torch.tensor([1.0], requires_grad=True)
    opt2 = torch.optim.SGD([w2], lr=0.1)
    sched2 = create_warmup_cosine_scheduler(opt2, warmup_steps=20, total_steps=100)
    sched2.load_state_dict(state)

    assert pytest.approx(sched2.get_last_lr()[0], abs=1e-5) == lr_at_50
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 207 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_learning_rate_schedules_lib.py (1774 bytes) 
import pytest
import torch
from examples.learning_rate_schedules_lib import create_warmup_cosine_scheduler

def test_warmup_cosine_scheduler_warmup_phase():
    w = torch.tensor([1.0], requires_grad=True)
    opt = torch.optim.SGD([w], lr=0.1)
    sched = create_warmup_cosine_scheduler(opt, warmup_steps=10, total_steps=100, min_lr_ratio=0.01)

    assert sched.get_last_lr()[0] == 0.0
    for _ in range(5):
        sched.step()
    # At step 5 (halfway through warmup), lr should be 0.05
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.05

    for _ in range(5):
        sched.step()
    # At step 10 (end of warmup), lr should be peak 0.1
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.1

def test_warmup_cosine_scheduler_final_decay():
    w = torch.tensor([1.0], requires_grad=True)
    opt = torch.optim.SGD([w], lr=0.1)
    sched = create_warmup_cosine_scheduler(opt, warmup_steps=10, total_steps=100, min_lr_ratio=0.01)

    for _ in range(100):
        sched.step()

    # At final step 100, lr should decay to min_lr_ratio * base_lr = 0.001
    assert pytest.approx(sched.get_last_lr()[0], abs=1e-5) == 0.001

def test_scheduler_state_dict_persistence():
    w1 = torch.tensor([1.0], requires_grad=True)
    opt1 = torch.optim.SGD([w1], lr=0.1)
    sched1 = create_warmup_cosine_scheduler(opt1, warmup_steps=20, total_steps=100)

    for _ in range(50):
        sched1.step()
    lr_at_50 = sched1.get_last_lr()[0]

    state = sched1.state_dict()

    w2 = torch.tensor([1.0], requires_grad=True)
    opt2 = torch.optim.SGD([w2], lr=0.1)
    sched2 = create_warmup_cosine_scheduler(opt2, warmup_steps=20, total_steps=100)
    sched2.load_state_dict(state)

    assert pytest.approx(sched2.get_last_lr()[0], abs=1e-5) == lr_at_50

Troubleshooting

Troubleshooting: Day 207 - Learning Rate Schedules

Common Issues

  1. UserWarning: Detected call of lr_scheduler.step() before optimizer.step():
    • Cause: Calling scheduler.step() before optimizer.step().
    • Fix: Ensure scheduler.step() is called after optimizer.step().

Security notes

Security & Privacy: Day 207 - Learning Rate Schedules

Security Guidance

  • All schedule calculations execute in local RAM on CPU.