Deep Learning › Convolutional Networks and Vision › Day 213

Hands-on lab — Day 213: Transfer Learning

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/transfer_learning_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/test_transfer_learning_lib.py
examples/transfer_learning_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_transfer_learning_lib.py
starter/transfer_learning_lib.py
tests/run_tests.sh
tests/test_transfer_learning_lib.py
troubleshooting.md

Lab README

Lab: Day 213 -- Transfer Learning

Lesson

Day number: 213 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Transfer Learning and Fine-Tuning in PyTorch.

Purpose

Build and test a modular Transfer Learning architecture in PyTorch. Implement parameter freezing routines with requires_grad, replace classification heads for custom target datasets, configure differential layer-wise learning rates, and verify that frozen backbones receive zero gradient updates during backward propagation.

Learning objectives

  • Freeze and unfreeze parameter subsets in PyTorch vision models.
  • Replace fully connected heads to support arbitrary target classes.
  • Verify gradient isolation during backpropagation.
  • Configure multi-group differential learning rate optimizers.

Prerequisites

  • Day 212 (CNN Architectures).
  • Python 3.11+ with PyTorch.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is free and open-source under the modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/transfer_learning_lib.py: Student scaffold file.
  • examples/transfer_learning_lib.py: Complete reference implementation.
  • tests/test_transfer_learning_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/transfer_learning_lib.py

What the commands do

  • Instantiates a TransferVisionModel with a frozen backbone.
  • Evaluates forward pass and counts frozen vs trainable parameters.
  • Validates gradient isolation and differential optimizer grouping.

Expected output

Transfer Demo: Out = torch.Size([2, 5]), Frozen = 5120, Trainable = 613

Validation steps

  1. Verify that requires_grad=False parameters receive None gradients during backprop.
  2. Confirm that newly added classification heads receive valid gradients.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Backbone Receives Gradients: Ensure requires_grad = False is set before invoking the optimizer.

Security notes

All neural models execute in local system memory without external telemetry.

Extension exercises

  1. Implement embedding caching: extract backbone feature vectors to disk and train a linear classifier in 1 second.
  2. Implement gradual unfreezing across successive training epochs.
  • Lesson title: Transfer Learning
  • Day number: 213 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-213-transfer-learning
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-213-transfer-learning when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 213

- `Out`: Output logits shape.
- `Frozen`: Total frozen parameters in backbone.
- `Trainable`: Total trainable parameters in new head.

examples-run.txt

Transfer Demo: Out = torch.Size([2, 5]), Frozen = 5120, Trainable = 613

measured-values.txt

Out: torch.Size([2, 5])
Frozen: 5120
Trainable: 613

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_transfer_learning_lib.py::test_frozen_backbone_parameters PASSED [ 33%]
tests/test_transfer_learning_lib.py::test_gradient_isolation PASSED         [ 66%]
tests/test_transfer_learning_lib.py::test_differential_optimizer_groups PASSED [100%]

============================== 3 passed in 0.29s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/test_transfer_learning_lib.py (1315 bytes)
import pytest
import torch
from examples.transfer_learning_lib import build_transfer_learning_model, configure_differential_optimizer

def test_frozen_backbone_parameters():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=True)
    # Check that backbone is frozen
    for p in model.backbone.parameters():
        assert p.requires_grad is False
    # Check that head is trainable
    for p in model.fc.parameters():
        assert p.requires_grad is True

def test_gradient_isolation():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=True)
    x = torch.randn(4, 3, 32, 32)
    y = torch.randint(0, 5, (4,))

    logits = model(x)
    loss = torch.nn.CrossEntropyLoss()(logits, y)
    loss.backward()

    # Backbone weights must have None grad
    for p in model.backbone.parameters():
        assert p.grad is None
    # Head weights must have non-zero grad
    for p in model.fc.parameters():
        assert p.grad is not None

def test_differential_optimizer_groups():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=False)
    opt = configure_differential_optimizer(model, head_lr=1e-3, backbone_lr=1e-5)
    assert len(opt.param_groups) == 2
    assert opt.param_groups[0]["lr"] == 1e-5
    assert opt.param_groups[1]["lr"] == 1e-3
examples/transfer_learning_lib.py (2503 bytes)
import torch
import torch.nn as nn
from typing import Tuple, Dict, Any, List

class SimpleBackbone(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(16)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=2, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(32)
        self.gap = nn.AdaptiveAvgPool2d((1, 1))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.relu(self.bn1(self.conv1(x)))
        x = self.relu(self.bn2(self.conv2(x)))
        x = self.gap(x)
        return torch.flatten(x, 1)

class TransferVisionModel(nn.Module):
    def __init__(self, num_classes: int = 5, freeze_backbone: bool = True):
        super().__init__()
        self.backbone = SimpleBackbone()
        if freeze_backbone:
            for param in self.backbone.parameters():
                param.requires_grad = False

        self.fc = nn.Sequential(
            nn.Linear(32, 16),
            nn.ReLU(inplace=True),
            nn.Dropout(p=0.2),
            nn.Linear(16, num_classes)
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        feat = self.backbone(x)
        return self.fc(feat)

def build_transfer_learning_model(num_classes: int = 5, freeze_backbone: bool = True) -> nn.Module:
    return TransferVisionModel(num_classes=num_classes, freeze_backbone=freeze_backbone)

def configure_differential_optimizer(model: TransferVisionModel, head_lr: float = 1e-3,
                                     backbone_lr: float = 1e-5) -> torch.optim.Optimizer:
    optimizer = torch.optim.AdamW([
        {"params": [p for p in model.backbone.parameters() if p.requires_grad], "lr": backbone_lr},
        {"params": model.fc.parameters(), "lr": head_lr}
    ], weight_decay=1e-2)
    return optimizer

def run_transfer_demo():
    torch.manual_seed(42)
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=True)
    x = torch.randn(2, 3, 32, 32)
    out = model(x)

    frozen_params = sum(p.numel() for p in model.backbone.parameters() if not p.requires_grad)
    trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

    print(f"Transfer Demo: Out = {out.shape}, Frozen = {frozen_params}, Trainable = {trainable_params}")
    return out.shape, frozen_params, trainable_params

if __name__ == "__main__":
    run_transfer_demo()
metadata.yml (430 bytes)
lesson_id: D213
day: 213
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/transfer_learning_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (27 bytes)
torch>=2.2.0
pytest>=8.0.0
starter/test_transfer_learning_lib.py (1315 bytes)
import pytest
import torch
from examples.transfer_learning_lib import build_transfer_learning_model, configure_differential_optimizer

def test_frozen_backbone_parameters():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=True)
    # Check that backbone is frozen
    for p in model.backbone.parameters():
        assert p.requires_grad is False
    # Check that head is trainable
    for p in model.fc.parameters():
        assert p.requires_grad is True

def test_gradient_isolation():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=True)
    x = torch.randn(4, 3, 32, 32)
    y = torch.randint(0, 5, (4,))

    logits = model(x)
    loss = torch.nn.CrossEntropyLoss()(logits, y)
    loss.backward()

    # Backbone weights must have None grad
    for p in model.backbone.parameters():
        assert p.grad is None
    # Head weights must have non-zero grad
    for p in model.fc.parameters():
        assert p.grad is not None

def test_differential_optimizer_groups():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=False)
    opt = configure_differential_optimizer(model, head_lr=1e-3, backbone_lr=1e-5)
    assert len(opt.param_groups) == 2
    assert opt.param_groups[0]["lr"] == 1e-5
    assert opt.param_groups[1]["lr"] == 1e-3
starter/transfer_learning_lib.py (529 bytes)
import torch
import torch.nn as nn
from typing import Tuple, Dict, Any, List

def build_transfer_learning_model(num_classes: int = 5, freeze_backbone: bool = True) -> nn.Module:
    # TODO: Build modular model with frozen backbone and custom classifier head
    pass

def configure_differential_optimizer(model: nn.Module, head_lr: float = 1e-3,
                                     backbone_lr: float = 1e-5) -> torch.optim.Optimizer:
    # TODO: Configure PyTorch optimizer with differential layer-wise learning rates
    pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 213 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_transfer_learning_lib.py (1315 bytes)
import pytest
import torch
from examples.transfer_learning_lib import build_transfer_learning_model, configure_differential_optimizer

def test_frozen_backbone_parameters():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=True)
    # Check that backbone is frozen
    for p in model.backbone.parameters():
        assert p.requires_grad is False
    # Check that head is trainable
    for p in model.fc.parameters():
        assert p.requires_grad is True

def test_gradient_isolation():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=True)
    x = torch.randn(4, 3, 32, 32)
    y = torch.randint(0, 5, (4,))

    logits = model(x)
    loss = torch.nn.CrossEntropyLoss()(logits, y)
    loss.backward()

    # Backbone weights must have None grad
    for p in model.backbone.parameters():
        assert p.grad is None
    # Head weights must have non-zero grad
    for p in model.fc.parameters():
        assert p.grad is not None

def test_differential_optimizer_groups():
    model = build_transfer_learning_model(num_classes=5, freeze_backbone=False)
    opt = configure_differential_optimizer(model, head_lr=1e-3, backbone_lr=1e-5)
    assert len(opt.param_groups) == 2
    assert opt.param_groups[0]["lr"] == 1e-5
    assert opt.param_groups[1]["lr"] == 1e-3

Troubleshooting

Troubleshooting: Day 213 - Transfer Learning

Common Issues

  1. Backbone weights changing when frozen:
    • Cause: Passing frozen parameters to the optimizer without checking p.requires_grad.
    • Fix: Filter optimizer parameters: [p for p in model.parameters() if p.requires_grad].

Security notes

Security & Privacy: Day 213 - Transfer Learning

Security Guidance

  • Always verify SHA-256 integrity when downloading pretrained vision weights.