Deep Learning › Transformers › Day 231
Hands-on lab — Day 231: Fine-Tuning a Small Transformer
- ← Back to the Day 231 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-231-fine-tuning-a-small-transformer/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/fine_tuning_a_small_transformer_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/fine_tuning_a_small_transformer_lib.py examples/test_fine_tuning_a_small_transformer_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/fine_tuning_a_small_transformer_lib.py starter/test_fine_tuning_a_small_transformer_lib.py tests/run_tests.sh tests/test_fine_tuning_a_small_transformer_lib.py troubleshooting.md
Lab README
Lab: Day 231 -- Fine-Tuning a Small Transformer
Lesson
Day number: 231 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Fine-Tuning Small Transformers, LLRD, and Regularization.
Purpose
Build and test a modular fine-tuning parameter configuration and regularization suite in PyTorch. Implement the Layer-wise Learning Rate Decay (LLRD) parameter group builder, construct decoupled weight decay mappings, and implement early stopping patience monitors.
Learning objectives
- Implement Layer-wise Learning Rate Decay (LLRD) parameter grouping.
- Enforce decoupled zero weight decay on bias and normalization layers.
- Construct the EarlyStopping regularization callback.
- Verify optimization parameter stability and learning rate assignments.
Prerequisites
- Day 230 (Hugging Face Transformers in Practice).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/fine_tuning_a_small_transformer_lib.py: Student scaffold file.examples/fine_tuning_a_small_transformer_lib.py: Complete reference implementation.tests/test_fine_tuning_a_small_transformer_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/fine_tuning_a_small_transformer_lib.py
What the commands do
- Generates LLRD optimizer parameter groups across network layers.
- Tests early stopping convergence tracking.
- Runs unit test assertions.
Expected output
FineTune Demo: Created 6 Parameter Groups
Validation steps
- Verify
create_llrd_parameter_groupsproduces parameter groups for each active layer. - Confirm biases and LayerNorm parameters receive
weight_decay == 0.0. - Confirm
EarlyStoppingtriggers when validation loss fails to improve after patience steps. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Missing parameters in optimizer: Ensure all model parameters with
requires_grad=Trueare captured in the parameter groups.
Security notes
All optimization routines execute locally in process memory.
Extension exercises
- Implement a Knowledge Distillation loss module.
- Export a PyTorch transformer to ONNX Runtime format.
Navigation
- Lesson title: Fine-Tuning a Small Transformer
- Day number: 231 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-231-fine-tuning-a-small-transformer
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-231-fine-tuning-a-small-transformerwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 231
- `Created Parameter Groups`: Number of differential parameter groups created for the optimizer.
examples-run.txt
FineTune Demo: Created 6 Parameter Groups
measured-values.txt
Created Parameter Groups: 6
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_fine_tuning_a_small_transformer_lib.py::test_llrd_parameter_groups PASSED [ 50%]
tests/test_fine_tuning_a_small_transformer_lib.py::test_early_stopping PASSED [100%]
============================== 2 passed in 0.42s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/fine_tuning_a_small_transformer_lib.py (2061 bytes)
import torch
import torch.nn as nn
from typing import List, Dict, Any
def create_llrd_parameter_groups(
model: nn.Module,
base_lr: float = 5e-5,
decay_rate: float = 0.85,
weight_decay: float = 0.01
) -> List[Dict[str, Any]]:
no_decay = ["bias", "LayerNorm.weight", "layer_norm.weight"]
optimizer_grouped_parameters = []
for name, param in model.named_parameters():
if not param.requires_grad:
continue
is_no_decay = any(nd in name for nd in no_decay)
wd = 0.0 if is_no_decay else weight_decay
# Calculate layer-specific learning rate
if "classifier" in name or "head" in name:
lr = base_lr * 2.0
elif "layer.1" in name or "layers.1" in name:
lr = base_lr
elif "layer.0" in name or "layers.0" in name:
lr = base_lr * decay_rate
else:
lr = base_lr * (decay_rate ** 2)
optimizer_grouped_parameters.append({
"params": [param],
"lr": lr,
"weight_decay": wd,
"name": name
})
return optimizer_grouped_parameters
class EarlyStopping:
def __init__(self, patience: int = 2, min_delta: float = 0.0):
self.patience = patience
self.min_delta = min_delta
self.best_loss = float('inf')
self.counter = 0
self.early_stop = False
def __call__(self, val_loss: float) -> bool:
if val_loss < self.best_loss - self.min_delta:
self.best_loss = val_loss
self.counter = 0
else:
self.counter += 1
if self.counter >= self.patience:
self.early_stop = True
return self.early_stop
def run_finetune_demo():
model = nn.Sequential(
nn.Linear(16, 16),
nn.LayerNorm(16),
nn.Linear(16, 2)
)
groups = create_llrd_parameter_groups(model, base_lr=5e-5)
print(f"FineTune Demo: Created {len(groups)} Parameter Groups")
return groups
if __name__ == "__main__":
run_finetune_demo()
examples/test_fine_tuning_a_small_transformer_lib.py (693 bytes)
import pytest
import torch
import torch.nn as nn
from examples.fine_tuning_a_small_transformer_lib import create_llrd_parameter_groups, EarlyStopping
def test_llrd_parameter_groups():
model = nn.Sequential(
nn.Linear(8, 8),
nn.LayerNorm(8),
nn.Linear(8, 2)
)
groups = create_llrd_parameter_groups(model, base_lr=1e-4)
assert len(groups) > 0
# Check that biases have zero weight decay
for g in groups:
if "bias" in g["name"]:
assert g["weight_decay"] == 0.0
def test_early_stopping():
stopper = EarlyStopping(patience=2)
assert not stopper(1.0)
assert not stopper(1.1)
assert stopper(1.2) # Patience reached
metadata.yml (444 bytes)
lesson_id: D231
day: 231
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/fine_tuning_a_small_transformer_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/fine_tuning_a_small_transformer_lib.py (689 bytes)
import torch
import torch.nn as nn
from typing import List, Dict, Any
def create_llrd_parameter_groups(
model: nn.Module,
base_lr: float = 5e-5,
decay_rate: float = 0.85,
weight_decay: float = 0.01
) -> List[Dict[str, Any]]:
# TODO: Create parameter groups with layer-wise decaying learning rates
pass
class EarlyStopping:
def __init__(self, patience: int = 2, min_delta: float = 0.0):
self.patience = patience
self.min_delta = min_delta
self.best_loss = float('inf')
self.counter = 0
self.early_stop = False
def __call__(self, val_loss: float) -> bool:
# TODO: Implement early stopping logic
pass
starter/test_fine_tuning_a_small_transformer_lib.py (693 bytes)
import pytest
import torch
import torch.nn as nn
from examples.fine_tuning_a_small_transformer_lib import create_llrd_parameter_groups, EarlyStopping
def test_llrd_parameter_groups():
model = nn.Sequential(
nn.Linear(8, 8),
nn.LayerNorm(8),
nn.Linear(8, 2)
)
groups = create_llrd_parameter_groups(model, base_lr=1e-4)
assert len(groups) > 0
# Check that biases have zero weight decay
for g in groups:
if "bias" in g["name"]:
assert g["weight_decay"] == 0.0
def test_early_stopping():
stopper = EarlyStopping(patience=2)
assert not stopper(1.0)
assert not stopper(1.1)
assert stopper(1.2) # Patience reached
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 231 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_fine_tuning_a_small_transformer_lib.py (693 bytes)
import pytest
import torch
import torch.nn as nn
from examples.fine_tuning_a_small_transformer_lib import create_llrd_parameter_groups, EarlyStopping
def test_llrd_parameter_groups():
model = nn.Sequential(
nn.Linear(8, 8),
nn.LayerNorm(8),
nn.Linear(8, 2)
)
groups = create_llrd_parameter_groups(model, base_lr=1e-4)
assert len(groups) > 0
# Check that biases have zero weight decay
for g in groups:
if "bias" in g["name"]:
assert g["weight_decay"] == 0.0
def test_early_stopping():
stopper = EarlyStopping(patience=2)
assert not stopper(1.0)
assert not stopper(1.1)
assert stopper(1.2) # Patience reached
Troubleshooting
Troubleshooting: Day 231 - Fine-Tuning a Small Transformer
Common Issues
- Weight decay applied to LayerNorm:
- Cause: Parameter name filter missed
LayerNorm.weight. - Fix: Include
LayerNorm.weightinno_decaylist.
- Cause: Parameter name filter missed
Security notes
Security & Privacy: Day 231 - Fine-Tuning a Small Transformer
Security Guidance
- All fine-tuning calculations execute in local memory.