Deep Learning › Transformers › Day 231

Hands-on lab — Day 231: Fine-Tuning a Small Transformer

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/fine_tuning_a_small_transformer_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/fine_tuning_a_small_transformer_lib.py
examples/test_fine_tuning_a_small_transformer_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/fine_tuning_a_small_transformer_lib.py
starter/test_fine_tuning_a_small_transformer_lib.py
tests/run_tests.sh
tests/test_fine_tuning_a_small_transformer_lib.py
troubleshooting.md

Lab README

Lab: Day 231 -- Fine-Tuning a Small Transformer

Lesson

Day number: 231 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Fine-Tuning Small Transformers, LLRD, and Regularization.

Purpose

Build and test a modular fine-tuning parameter configuration and regularization suite in PyTorch. Implement the Layer-wise Learning Rate Decay (LLRD) parameter group builder, construct decoupled weight decay mappings, and implement early stopping patience monitors.

Learning objectives

  • Implement Layer-wise Learning Rate Decay (LLRD) parameter grouping.
  • Enforce decoupled zero weight decay on bias and normalization layers.
  • Construct the EarlyStopping regularization callback.
  • Verify optimization parameter stability and learning rate assignments.

Prerequisites

  • Day 230 (Hugging Face Transformers in Practice).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/fine_tuning_a_small_transformer_lib.py: Student scaffold file.
  • examples/fine_tuning_a_small_transformer_lib.py: Complete reference implementation.
  • tests/test_fine_tuning_a_small_transformer_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/fine_tuning_a_small_transformer_lib.py

What the commands do

  • Generates LLRD optimizer parameter groups across network layers.
  • Tests early stopping convergence tracking.
  • Runs unit test assertions.

Expected output

FineTune Demo: Created 6 Parameter Groups

Validation steps

  1. Verify create_llrd_parameter_groups produces parameter groups for each active layer.
  2. Confirm biases and LayerNorm parameters receive weight_decay == 0.0.
  3. Confirm EarlyStopping triggers when validation loss fails to improve after patience steps.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Missing parameters in optimizer: Ensure all model parameters with requires_grad=True are captured in the parameter groups.

Security notes

All optimization routines execute locally in process memory.

Extension exercises

  1. Implement a Knowledge Distillation loss module.
  2. Export a PyTorch transformer to ONNX Runtime format.

Expected output

FIELDS.md

# Expected Output Fields: Day 231

- `Created Parameter Groups`: Number of differential parameter groups created for the optimizer.

examples-run.txt

FineTune Demo: Created 6 Parameter Groups

measured-values.txt

Created Parameter Groups: 6

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_fine_tuning_a_small_transformer_lib.py::test_llrd_parameter_groups PASSED [ 50%]
tests/test_fine_tuning_a_small_transformer_lib.py::test_early_stopping PASSED        [100%]

============================== 2 passed in 0.42s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/fine_tuning_a_small_transformer_lib.py (2061 bytes)
import torch
import torch.nn as nn
from typing import List, Dict, Any

def create_llrd_parameter_groups(
    model: nn.Module,
    base_lr: float = 5e-5,
    decay_rate: float = 0.85,
    weight_decay: float = 0.01
) -> List[Dict[str, Any]]:
    no_decay = ["bias", "LayerNorm.weight", "layer_norm.weight"]
    optimizer_grouped_parameters = []

    for name, param in model.named_parameters():
        if not param.requires_grad:
            continue
        
        is_no_decay = any(nd in name for nd in no_decay)
        wd = 0.0 if is_no_decay else weight_decay

        # Calculate layer-specific learning rate
        if "classifier" in name or "head" in name:
            lr = base_lr * 2.0
        elif "layer.1" in name or "layers.1" in name:
            lr = base_lr
        elif "layer.0" in name or "layers.0" in name:
            lr = base_lr * decay_rate
        else:
            lr = base_lr * (decay_rate ** 2)

        optimizer_grouped_parameters.append({
            "params": [param],
            "lr": lr,
            "weight_decay": wd,
            "name": name
        })

    return optimizer_grouped_parameters

class EarlyStopping:
    def __init__(self, patience: int = 2, min_delta: float = 0.0):
        self.patience = patience
        self.min_delta = min_delta
        self.best_loss = float('inf')
        self.counter = 0
        self.early_stop = False

    def __call__(self, val_loss: float) -> bool:
        if val_loss < self.best_loss - self.min_delta:
            self.best_loss = val_loss
            self.counter = 0
        else:
            self.counter += 1
            if self.counter >= self.patience:
                self.early_stop = True
        return self.early_stop

def run_finetune_demo():
    model = nn.Sequential(
        nn.Linear(16, 16),
        nn.LayerNorm(16),
        nn.Linear(16, 2)
    )
    groups = create_llrd_parameter_groups(model, base_lr=5e-5)
    print(f"FineTune Demo: Created {len(groups)} Parameter Groups")
    return groups

if __name__ == "__main__":
    run_finetune_demo()
examples/test_fine_tuning_a_small_transformer_lib.py (693 bytes)
import pytest
import torch
import torch.nn as nn
from examples.fine_tuning_a_small_transformer_lib import create_llrd_parameter_groups, EarlyStopping

def test_llrd_parameter_groups():
    model = nn.Sequential(
        nn.Linear(8, 8),
        nn.LayerNorm(8),
        nn.Linear(8, 2)
    )
    groups = create_llrd_parameter_groups(model, base_lr=1e-4)
    assert len(groups) > 0
    # Check that biases have zero weight decay
    for g in groups:
        if "bias" in g["name"]:
            assert g["weight_decay"] == 0.0

def test_early_stopping():
    stopper = EarlyStopping(patience=2)
    assert not stopper(1.0)
    assert not stopper(1.1)
    assert stopper(1.2) # Patience reached
metadata.yml (444 bytes)
lesson_id: D231
day: 231
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/fine_tuning_a_small_transformer_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/fine_tuning_a_small_transformer_lib.py (689 bytes)
import torch
import torch.nn as nn
from typing import List, Dict, Any

def create_llrd_parameter_groups(
    model: nn.Module,
    base_lr: float = 5e-5,
    decay_rate: float = 0.85,
    weight_decay: float = 0.01
) -> List[Dict[str, Any]]:
    # TODO: Create parameter groups with layer-wise decaying learning rates
    pass

class EarlyStopping:
    def __init__(self, patience: int = 2, min_delta: float = 0.0):
        self.patience = patience
        self.min_delta = min_delta
        self.best_loss = float('inf')
        self.counter = 0
        self.early_stop = False

    def __call__(self, val_loss: float) -> bool:
        # TODO: Implement early stopping logic
        pass
starter/test_fine_tuning_a_small_transformer_lib.py (693 bytes)
import pytest
import torch
import torch.nn as nn
from examples.fine_tuning_a_small_transformer_lib import create_llrd_parameter_groups, EarlyStopping

def test_llrd_parameter_groups():
    model = nn.Sequential(
        nn.Linear(8, 8),
        nn.LayerNorm(8),
        nn.Linear(8, 2)
    )
    groups = create_llrd_parameter_groups(model, base_lr=1e-4)
    assert len(groups) > 0
    # Check that biases have zero weight decay
    for g in groups:
        if "bias" in g["name"]:
            assert g["weight_decay"] == 0.0

def test_early_stopping():
    stopper = EarlyStopping(patience=2)
    assert not stopper(1.0)
    assert not stopper(1.1)
    assert stopper(1.2) # Patience reached
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 231 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_fine_tuning_a_small_transformer_lib.py (693 bytes)
import pytest
import torch
import torch.nn as nn
from examples.fine_tuning_a_small_transformer_lib import create_llrd_parameter_groups, EarlyStopping

def test_llrd_parameter_groups():
    model = nn.Sequential(
        nn.Linear(8, 8),
        nn.LayerNorm(8),
        nn.Linear(8, 2)
    )
    groups = create_llrd_parameter_groups(model, base_lr=1e-4)
    assert len(groups) > 0
    # Check that biases have zero weight decay
    for g in groups:
        if "bias" in g["name"]:
            assert g["weight_decay"] == 0.0

def test_early_stopping():
    stopper = EarlyStopping(patience=2)
    assert not stopper(1.0)
    assert not stopper(1.1)
    assert stopper(1.2) # Patience reached

Troubleshooting

Troubleshooting: Day 231 - Fine-Tuning a Small Transformer

Common Issues

  1. Weight decay applied to LayerNorm:
    • Cause: Parameter name filter missed LayerNorm.weight.
    • Fix: Include LayerNorm.weight in no_decay list.

Security notes

Security & Privacy: Day 231 - Fine-Tuning a Small Transformer

Security Guidance

  • All fine-tuning calculations execute in local memory.