Deep Learning β€Ί Convolutional Networks and Vision β€Ί Day 213

Day 213: Transfer Learning

Day 213 of 365 β€” Transfer Learning

Master Transfer Learning in Computer Vision: leverage ImageNet pretrained backbones (ResNet, ConvNeXt, ViT), implement linear probing with frozen parameters, apply differential layer-wise learning rates, prevent catastrophic forgetting, and adapt pretrained models to custom target datasets.

Course
Deep Learning
Category
Convolutional Networks and Vision
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-213-transfer-learning

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-213-transfer-learning
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Training a modern deep vision model (such as a 50-layer ResNet or Vision Transformer) from scratch requires:

  1. Massive Datasets: Over $1.2 million curated images$ (e.g. ImageNet-1k).
  2. Heavy Compute Budgets: Hundreds of GPU hours costing thousands of dollars.
  3. Severe Overfitting Risks: If you have only 500 images of rare plant diseases or solar panel defects, a network with 25 million random parameters will simply memorize the training examples and fail catastrophically on new data.

In industrial practice, nobody trains computer vision models from scratch.

Instead, the foundation of modern AI engineering is Transfer Learning: we take a network that has already learned universal visual primitives (edges, textures, spatial geometries, lighting cues) from millions of natural images, freeze its core backbone, and adapt it to our specific downstream task with just a few hundred labeled samples in minutes.

Today, you will master the theory, parameter freezing mechanics, and PyTorch implementation of Linear Probing, Differential Layer-Wise Fine-Tuning, and Catastrophic Forgetting Prevention.


The idea in plain language

Diagram: linear probing compared with full fine-tuning

Imagine hiring an experienced classical landscape painter who has spent 20 years mastering lighting, perspective, color blending, and brush technique:


Historical background

  1. 2014 (Yosinski et al. - Feature Transferability): Jason Yosinski and collaborators quantified that early layers of deep CNNs learn Gabor-like filters and color blobs that are 100% universal across all vision tasks, while deep layers become specialized to target dataset semantics.
  2. 2014 (DeCAF & Razavian et al.): Demonstrated that extracting features from pre-trained AlexNet and training a simple Support Vector Machine (SVM) on top beat almost all custom-engineered vision pipelines across dozens of independent benchmarks.
  3. 2018 (ULMFiT & Jeremy Howard): Introduced Discriminative Fine-Tuning and Gradual Unfreezing to prevent catastrophic forgetting, techniques that rapidly crossed over from NLP into Computer Vision.

What it is β€” and what it is not

Transfer learning paradigms comparing feature extraction linear probing with frozen backbone versus full fine-tuning with differential learning rates

What Transfer Learning IS:

What it is NOT:


Why it was created and what problems it solves

Transfer Learning resolves three critical bottlenecks:

  1. The Small Data Barrier: Enables high-accuracy visual classifiers on datasets with as few as 50 to 500 images per class.
  2. Training Compute & Carbon Footprint: Reduces training time from days on a GPU cluster to 5 minutes on a single laptop CPU/GPU.
  3. Faster Convergence: Pretrained models start with meaningful representations, reaching 90%+ accuracy in a single epoch.

How it works

Diagram: the three constraints transfer learning removes

Let us dissect the mathematical foundations of Layer-Wise Transferability, Linear Probing vs Fine-Tuning, and Catastrophic Forgetting.


1. The Layer-Wise Feature Transferability Spectrum

Diagram: how feature transferability changes with depth in a pretrained network

Feature transferability spectrum showing how early layers learn universal low-level visual primitives while late layers learn task-specific semantic representations

As visual data flows through a deep convolutional network:

  1. Layers 1–2 (Universal Primitives): Gabor-like directional edge filters, color gradient detectors, micro-corners. These representations are universally transferable across every visual domain in the universe.
  2. Layers 3–4 (Mid-Level Motifs): Textures, grids, geometric curves, object parts (eyes, wheels, mesh patterns). Highly transferable across natural images; requires minor fine-tuning for exotic domains.
  3. Layer 5 & Classifier Head (Task Semantics): 1000-class ImageNet specific decision boundaries (e.g. distinguishing a Golden Retriever from a Labrador). Non-transferable; must be replaced by a custom classification head.

2. Strategy 1: Feature Extraction (Linear Probing)

In Feature Extraction:

  1. Load a pretrained model (e.g. torchvision.models.resnet50(weights="DEFAULT")).

  2. Freeze all backbone parameters by setting param.requires_grad = False.

  3. Replace the final Linear layer (model.fc) with a new Linear layer matching your target class count num_classes:

    num_features = model.fc.in_features
    model.fc = nn.Linear(num_features, num_classes) # requires_grad is True by default
  4. Train only the new linear head.

Advantages:


3. Strategy 2: Differential Fine-Tuning & Gradual Unfreezing

In Fine-Tuning: After the new linear head has been trained for a few epochs and its loss has stabilized:

  1. Unfreeze the deeper convolutional stages of the backbone (requires_grad = True).
  2. Configure Differential (Layer-Wise) Learning Rates:
    • Early Layers (Stem): $LR = 10^-6$ (Extremely gentle updates).
    • Mid Layers: $LR = 10^-5$ (Moderate adaptation).
    • New Classifier Head: $LR = 10^-3$ (Standard learning rate).
optimizer = torch.optim.AdamW([
    {"params": model.layer1.parameters(), "lr": 1e-6},
    {"params": model.layer2.parameters(), "lr": 5e-6},
    {"params": model.layer3.parameters(), "lr": 1e-5},
    {"params": model.layer4.parameters(), "lr": 5e-5},
    {"params": model.fc.parameters(),     "lr": 1e-3}
], weight_decay=1e-2)

4. Preventing Catastrophic Forgetting

Diagram: the two-phase transfer learning protocol

Catastrophic Forgetting occurs when backpropagating large gradients into a pretrained network overwrites its delicately balanced feature detectors with noise.

The Golden 2-Stage Transfer Protocol:


5. The Timm Ecosystem & Hugging Face Hub

While torchvision.models provides standard classic architectures, Ross Wightman’s timm (PyTorch Image Models) library is the industry standard for production vision backbones:


6. Self-Supervised Pretraining: DINOv2 & MAE

Diagram: three pretraining paradigms for vision backbones

Historically, transfer learning models were pretrained on supervised ImageNet-1k labels. Today, the cutting edge of visual representation learning is Self-Supervised Pretraining:


7. Contrastive Vision-Language Models: CLIP & OpenCLIP

In 2021, Alec Radford et al. introduced CLIP (Contrastive Language-Image Pretraining):


8. Parameter-Efficient Fine-Tuning (PEFT & LoRA for Vision)

Diagram: full fine-tuning compared with parameter-efficient adaptation

When fine-tuning multi-hundred-million-parameter Vision Transformers (ViT-H, Swin-Large, EVA-CLIP), saving full checkpoint copies for dozens of different downstream tasks becomes prohibitive for disk storage and memory bandwidth:


An everyday analogy

Diagram: retraining an experienced professional compared with training from nothing

Think of adopting an internationally trained master chef into a specialized Italian pizzeria:


Examples in practice

Let us build a complete Transfer Learning harness in PyTorch adapting a pretrained vision backbone to a 5-class target dataset:

import torch
import torch.nn as nn
import torchvision.models as models

# 1. Load pretrained ResNet-18 backbone with modern weights API
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)

# 2. Freeze all feature extraction backbone parameters
for param in model.parameters():
    param.requires_grad = False

# 3. Replace the final 1000-class classifier head with a 5-class head
in_features = model.fc.in_features  # 512
model.fc = nn.Sequential(
    nn.Linear(in_features, 128),
    nn.ReLU(inplace=True),
    nn.Dropout(p=0.3),
    nn.Linear(128, 5) # 5 target classes
)

# Verify trainable parameters
trainable = [name for name, p in model.named_parameters() if p.requires_grad]
frozen = [name for name, p in model.named_parameters() if not p.requires_grad]

print(f"Total Parameter Tensors: {len(list(model.parameters()))}")
print(f"Frozen Backbone Tensors: {len(frozen)}")
print(f"Trainable Head Tensors:  {len(trainable)}")

Implications: security, privacy, performance, scalability, and cost

Diagram: the licensing, bias, distribution and efficiency implications of transfer learning

  1. Pretrained Weight Poisoning & Backdoor Attacks:
    • Pretrained weights downloaded from untrusted public hubs can contain hidden trojans/backdoors (e.g. activating a malicious classification when a specific pixel watermark is present). Always verify model SHA-256 hashes and download only from official torchvision/HuggingFace repositories.
  2. Inference Latency & Quantization:
    • Pretrained backbones can be quantized from FP32 down to INT8 post-training using PyTorch native quantization, achieving 4x memory reduction and 3x faster inference on CPU/mobile devices without retraining.

Alternatives: free, open source, and commercial

Pretrained Vision BackboneArchitecture TypeParametersImageNet Top-1Best For
ResNet-18 / 50Residual CNN11M / 25M69.8% / 76.1%Fast edge & baseline deployment
MobileNet-v3-LargeDepthwise Separable CNN5.4M75.2%Mobile / iOS / Android apps
ConvNeXt-TinyModernized CNN28M82.1%High-accuracy convolutional vision
ViT-B/16 (Vision Transformer)Self-Attention Transformer86M81.8%Large target datasets with high compute
CLIP (OpenAI / OpenCLIP)Contrastive Language-Image150MZero-ShotZero-shot & open-vocabulary search

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   TRANSFER LEARNING STRATEGY MATRIX                    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Dataset Size       β”‚ Visual Similarity     β”‚ Optimal Strategy          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Tiny (< 500 imgs) β”‚ High (Natural images) β”‚ Freeze Backbone + Train FCβ”‚
β”‚ Tiny (< 500 imgs) β”‚ Low (Medical/Thermal) β”‚ Early Layer Freeze + FC   β”‚
β”‚ Medium (1k-10k)    β”‚ High (Natural images) β”‚ 2-Stage Fine-Tuning       β”‚
β”‚ Large (> 50k imgs) β”‚ Any                   β”‚ Full Fine-Tuning (Diff LR)β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when transfer learning is the right approach

When to USE Transfer Learning:

When NOT to use:


The AI connection

Knowledge check

  1. What causes Catastrophic Forgetting when fine-tuning a pretrained model?
  2. Why are early convolutional layers universally transferable across distinct image domains?
  3. How do you construct an optimizer with differential learning rates in PyTorch?
  4. What is the difference in compute requirements between Linear Probing and Full Fine-Tuning?
  5. Why must image input tensors be normalized with the pretraining dataset’s mean and standard deviation?

Hands-on exercise

In this lab, you will build and test a production-grade Transfer Learning pipeline in PyTorch: instantiate a pretrained vision backbone, implement parameter freezing and verification routines, attach a custom multi-layer classification head, configure a two-stage differential learning rate optimizer, and benchmark training speedups against from-scratch models.

Expected output

[Transfer Learning Verification Suite]
Initializing Pretrained Backbone (ResNet Architecture):
  Freezing Backbone Parameters: 60 tensors frozen (requires_grad = False)
  Attaching Custom Classifier Head (in_features=512 -> 5 classes):
  Trainable Parameters: 65,925 (0.58% of total network)
Testing 2-Stage Optimizer Configuration:
  Phase 1 (Linear Probing): Head LR = 1.00e-03, Backbone LR = 0.00e+00 [HEAD WARMUP READY]
  Phase 2 (Differential Fine-Tuning): Head LR = 1.00e-03, Layer3 LR = 1.00e-05, Layer4 LR = 5.00e-05
Forward & Backward Verification on Synthetic Batch (4x3x32x32):
  Loss Backward Computed: [GRADIENTS CONFUSED ONLY TO UNFROZEN PARAMETERS]
Test Suite: 4 passed in 0.29s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Embedding Extraction & Caching: Run your dataset once through a frozen backbone, save the 512-dimensional feature vectors to disk, and train a Logistic Regression / Linear layer in 2 seconds.
  2. Implement Gradual Unfreezing: Unfreeze layer 4 first, train for 2 epochs, then unfreeze layer 3, train for 2 epochs, demonstrating steady loss reduction.

Extension challenge

Implement CLIP Zero-Shot Image Classification:

Quiz

Q1. What is Catastrophic Forgetting during deep neural network fine-tuning, and how does a two-stage training schedule prevent it?

  1. When batch size is set to a prime number
  2. When a model loses its connection to the GPU memory bus
  3. When the validation dataset is accidentally deleted
  4. A random head's gradients wreck the pretrained backbone
Show answer

Answer: D. A random head's gradients wreck the pretrained backbone

A newly initialized linear head produces huge random losses. If the backbone is trainable immediately, large gradients scramble the carefully pretrained feature extractors. Freezing the backbone first stabilizes the head before gentle fine-tuning.

Q2. How do you freeze all convolutional backbone parameters in a PyTorch vision model so that they do not receive gradient updates?

  1. requires_grad = False
  2. model.eval()
  3. torch.no_grad()
  4. del model.backbone
Show answer

Answer: A. requires_grad = False

Setting param.requires_grad = False excludes those tensors from autograd graph tracking and gradient computation during backward().

Q3. If you have a very small target dataset (e.g. 200 medical X-ray images) that is visually distinct from ImageNet, which transfer learning strategy is most mathematically and empirically sound?

  1. Train a 152-layer ResNet from scratch with random initialization
  2. Freeze the backbone and train a linear head on its features
  3. Fine-tune all 152 layers with a huge learning rate (0.1)
  4. Delete all convolution layers and use an MLP
Show answer

Answer: B. Freeze the backbone and train a linear head on its features

With only 200 samples, fine-tuning millions of parameters guarantees severe overfitting. Freezing early/mid universal features and training a linear head is the optimal strategy.

Q4. How are differential (layer-wise) learning rates configured in a PyTorch optimizer?

  1. Set learning rate in model.forward()
  2. Create two separate Python scripts
  3. Pass per-group learning rates to the optimizer
  4. Multiply the loss by 0.01
Show answer

Answer: C. Pass per-group learning rates to the optimizer

PyTorch optimizers natively accept lists of dicts containing specific param subsets and custom learning rates, enabling fine-grained layer-wise tuning.

Q5. Why is it critical to normalize input images with the exact mean and standard deviation of the pretraining dataset (e.g. ImageNet mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])?

  1. It converts RGB images to CMYK
  2. PyTorch throws a compile error if standard deviation is not 0.229
  3. It makes the image file sizes smaller on disk
  4. The filters were trained under that exact input distribution
Show answer

Answer: D. The filters were trained under that exact input distribution

Convolutional kernels learn specific numerical scales for edge responses. Using the exact pretraining normalization ensures the input aligns with the distribution the weights expect.

Glossary

Transfer Learning
A machine learning paradigm where knowledge gained from solving one problem is applied to a different but related problem.
Linear Probing
Training only a linear classifier head on top of frozen pretrained feature representations.
Fine-Tuning
Continuing gradient descent training across the entire pretrained network (or subset of layers) on target dataset data.
Catastrophic Forgetting
The phenomenon where a neural network abruptly forgets previously learned knowledge upon learning new information.
Differential Learning Rates
Assigning smaller learning rates to early pretrained layers and larger learning rates to newly initialized decision heads.
Pretrained Weights
Neural network weight matrices saved after extensive training on a massive dataset (such as ImageNet-1k or LAION-5B).
Backbone
The core convolutional or transformer feature extractor that generates dense visual representations before the classifier head.
Distribution Shift
A change in the statistical distribution of input data between the source pretraining dataset and the target downstream task.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.