Deep Learning βΊ Convolutional Networks and Vision βΊ Day 213
Day 213: Transfer Learning
Master Transfer Learning in Computer Vision: leverage ImageNet pretrained backbones (ResNet, ConvNeXt, ViT), implement linear probing with frozen parameters, apply differential layer-wise learning rates, prevent catastrophic forgetting, and adapt pretrained models to custom target datasets.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-213-transfer-learning
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/deep-learning/day-213-transfer-learning - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Explain why transfer learning is the dominant paradigm in computer vision and deep learning.
- Distinguish between Feature Extraction (Linear Probing) and Full Fine-Tuning with differential learning rates.
- Freeze and unfreeze parameter subsets in PyTorch using requires_grad.
- Replace classification heads to adapt pretrained models to custom class counts.
- Implement a two-stage transfer learning training regimen to prevent catastrophic forgetting.
Prerequisites
Why this matters
Training a modern deep vision model (such as a 50-layer ResNet or Vision Transformer) from scratch requires:
- Massive Datasets: Over $1.2 million curated images$ (e.g. ImageNet-1k).
- Heavy Compute Budgets: Hundreds of GPU hours costing thousands of dollars.
- Severe Overfitting Risks: If you have only 500 images of rare plant diseases or solar panel defects, a network with 25 million random parameters will simply memorize the training examples and fail catastrophically on new data.
In industrial practice, nobody trains computer vision models from scratch.
Instead, the foundation of modern AI engineering is Transfer Learning: we take a network that has already learned universal visual primitives (edges, textures, spatial geometries, lighting cues) from millions of natural images, freeze its core backbone, and adapt it to our specific downstream task with just a few hundred labeled samples in minutes.
Today, you will master the theory, parameter freezing mechanics, and PyTorch implementation of Linear Probing, Differential Layer-Wise Fine-Tuning, and Catastrophic Forgetting Prevention.
The idea in plain language
Imagine hiring an experienced classical landscape painter who has spent 20 years mastering lighting, perspective, color blending, and brush technique:
- Training from Scratch: You hire a toddler who has never seen a paintbrush and try to teach them how to identify hairline cracks in airplane turbines. They need years of basic visual education.
- Transfer Learning (Feature Extraction / Linear Probing): You hire the master painter. They already know what edges, surfaces, and shadows look like (Pretrained Backbone). You simply spend 15 minutes teaching them: βWhen you see a dark jagged line on this titanium surface, that is called a Fractureβ (New Classification Head).
- Fine-Tuning: If the task requires subtle domain-specific adjustments (e.g. infrared thermal imaging), the master painter gently refines their brush technique without forgetting how light and perspective work.
Historical background
- 2014 (Yosinski et al. - Feature Transferability): Jason Yosinski and collaborators quantified that early layers of deep CNNs learn Gabor-like filters and color blobs that are 100% universal across all vision tasks, while deep layers become specialized to target dataset semantics.
- 2014 (DeCAF & Razavian et al.): Demonstrated that extracting features from pre-trained AlexNet and training a simple Support Vector Machine (SVM) on top beat almost all custom-engineered vision pipelines across dozens of independent benchmarks.
- 2018 (ULMFiT & Jeremy Howard): Introduced Discriminative Fine-Tuning and Gradual Unfreezing to prevent catastrophic forgetting, techniques that rapidly crossed over from NLP into Computer Vision.
What it is β and what it is not
What Transfer Learning IS:
- Repurposing Representational Knowledge: Reusing spatial feature representations learned on source tasks (ImageNet, LAION) for target tasks (medical pathology, manufacturing QA, autonomous driving).
- A Two-Stage Training Regimen: Freezing the backbone to stabilize a newly initialized classifier head before unfreezing layers with small differential learning rates.
What it is NOT:
- Not a Magic Bullet for Severe Distribution Shift: If a model is pretrained on natural photographs, transferring to raw radar point clouds or genomic sequence spectrograms may require retraining mid-level feature blocks.
- Not Full-Speed Retraining: Fine-tuning requires learning rates that are 10x to 1000x smaller than pretraining learning rates to avoid destroying pretrained weights.
Why it was created and what problems it solves
Transfer Learning resolves three critical bottlenecks:
- The Small Data Barrier: Enables high-accuracy visual classifiers on datasets with as few as 50 to 500 images per class.
- Training Compute & Carbon Footprint: Reduces training time from days on a GPU cluster to 5 minutes on a single laptop CPU/GPU.
- Faster Convergence: Pretrained models start with meaningful representations, reaching 90%+ accuracy in a single epoch.
How it works
Let us dissect the mathematical foundations of Layer-Wise Transferability, Linear Probing vs Fine-Tuning, and Catastrophic Forgetting.
1. The Layer-Wise Feature Transferability Spectrum
As visual data flows through a deep convolutional network:
- Layers 1β2 (Universal Primitives): Gabor-like directional edge filters, color gradient detectors, micro-corners. These representations are universally transferable across every visual domain in the universe.
- Layers 3β4 (Mid-Level Motifs): Textures, grids, geometric curves, object parts (eyes, wheels, mesh patterns). Highly transferable across natural images; requires minor fine-tuning for exotic domains.
- Layer 5 & Classifier Head (Task Semantics): 1000-class ImageNet specific decision boundaries (e.g. distinguishing a Golden Retriever from a Labrador). Non-transferable; must be replaced by a custom classification head.
2. Strategy 1: Feature Extraction (Linear Probing)
In Feature Extraction:
-
Load a pretrained model (e.g.
torchvision.models.resnet50(weights="DEFAULT")). -
Freeze all backbone parameters by setting
param.requires_grad = False. -
Replace the final Linear layer (
model.fc) with a new Linear layer matching your target class countnum_classes:num_features = model.fc.in_features model.fc = nn.Linear(num_features, num_classes) # requires_grad is True by default -
Train only the new linear head.
Advantages:
- Extremely fast (can precompute and cache backbone embeddings).
- Zero risk of overfitting the backbone on small datasets.
3. Strategy 2: Differential Fine-Tuning & Gradual Unfreezing
In Fine-Tuning: After the new linear head has been trained for a few epochs and its loss has stabilized:
- Unfreeze the deeper convolutional stages of the backbone (
requires_grad = True). - Configure Differential (Layer-Wise) Learning Rates:
- Early Layers (Stem): $LR = 10^-6$ (Extremely gentle updates).
- Mid Layers: $LR = 10^-5$ (Moderate adaptation).
- New Classifier Head: $LR = 10^-3$ (Standard learning rate).
optimizer = torch.optim.AdamW([
{"params": model.layer1.parameters(), "lr": 1e-6},
{"params": model.layer2.parameters(), "lr": 5e-6},
{"params": model.layer3.parameters(), "lr": 1e-5},
{"params": model.layer4.parameters(), "lr": 5e-5},
{"params": model.fc.parameters(), "lr": 1e-3}
], weight_decay=1e-2)
4. Preventing Catastrophic Forgetting
Catastrophic Forgetting occurs when backpropagating large gradients into a pretrained network overwrites its delicately balanced feature detectors with noise.
The Golden 2-Stage Transfer Protocol:
- Phase 1 (Warmup Head): Freeze backbone (
requires_grad=False). Train onlymodel.fcfor 3β5 epochs until validation loss stabilizes. - Phase 2 (Fine-Tune Backbone): Unfreeze deep residual blocks. Train with Cosine Annealing learning rate schedule starting at $10^-5$.
5. The Timm Ecosystem & Hugging Face Hub
While torchvision.models provides standard classic architectures, Ross Wightmanβs timm (PyTorch Image Models) library is the industry standard for production vision backbones:
-
Over 1,000 Pretrained Models: Contains every modern architecture (ConvNeXt, Swin Transformer, EVA-02, MobileOne, EfficientNet-v2).
-
Unified Clean API:
import timm # Load pretrained ConvNeXt-Tiny with custom 5-class head in a single line model = timm.create_model('convnext_tiny.fb_in22k_ft_in1k', pretrained=True, num_classes=5) -
Feature Extraction Mode: Passing
features_only=Truereturns intermediate multi-scale feature maps for dense downstream tasks like object detection and segmentation.
6. Self-Supervised Pretraining: DINOv2 & MAE
Historically, transfer learning models were pretrained on supervised ImageNet-1k labels. Today, the cutting edge of visual representation learning is Self-Supervised Pretraining:
- DINO & DINOv2 (Meta AI): Self-distillation without labels. Trains Vision Transformers to produce matching representations across multiple crops of the same image. Features exhibit remarkable emergent properties: zero-shot semantic part segmentation and dense depth estimation without any human annotations.
- Masked Autoencoders (MAE / Kaiming He et al.): Randomly masks 75% of image patches and trains a Vision Transformer autoencoder to reconstruct missing pixels. Learns deep structural understanding of physical visual reality.
7. Contrastive Vision-Language Models: CLIP & OpenCLIP
In 2021, Alec Radford et al. introduced CLIP (Contrastive Language-Image Pretraining):
- Instead of predicting 1,000 arbitrary fixed categorical IDs, CLIP trains an image encoder and a text encoder simultaneously on 400 million image-caption pairs using a symmetric InfoNCE contrastive loss.
- Zero-Shot Classification: You can classify any image without fine-tuning by comparing its visual embedding $z_img$ against the text embeddings of candidate prompts $z_text$ (βa photo of a defectβ, βa photo of a normal componentβ) via cosine similarity:
P(class = k | x) = exp(sim(z_img, z_text_k) / T) / sum_j exp(sim(z_img, z_text_j) / T)
8. Parameter-Efficient Fine-Tuning (PEFT & LoRA for Vision)
When fine-tuning multi-hundred-million-parameter Vision Transformers (ViT-H, Swin-Large, EVA-CLIP), saving full checkpoint copies for dozens of different downstream tasks becomes prohibitive for disk storage and memory bandwidth:
-
Low-Rank Adaptation (LoRA): Freezes the pretrained weight matrix $W_0 \in R^d \times k$ and injects trainable rank-decomposition matrices $\Delta W = B \cdot A$ (where $B \in R^d \times r$ and $A \in R^r \times k$ with rank $r \ll d$).
-
Visual Prompt Tuning (VPT): Prepends a small set of continuous learnable prompt tokens to the sequence of image patch tokens at the input layer, keeping the entire transformer backbone 100% frozen.
-
Benefits: Updates less than 1% of parameters, eliminates catastrophic forgetting, and enables serving hundreds of customized domain models from a single shared backbone in production memory.
-
BatchNorm Tracking During Transfer Learning: When freezing a backbone, decide whether BatchNorm running statistics ($\mu_run, \sigma^2_run$) should remain frozen (
model.eval()) or update with target batches. Freezing BatchNorm stats prevents small unrepresentative target batches from corrupting calibrated population statistics.
An everyday analogy
Think of adopting an internationally trained master chef into a specialized Italian pizzeria:
- Training from Scratch: Teaching someone who has never boiled water how to make authentic Neapolitan pizza.
- Linear Probing: The master chef already knows knife skills, dough fermentation, and oven temperatures. You give them a one-page menu of your 5 pizza recipes (New Linear Head).
- Fine-Tuning: The chef subtly adjusts their kneading technique to account for the local flour brand and altitude (Differential Learning Rate).
Examples in practice
Let us build a complete Transfer Learning harness in PyTorch adapting a pretrained vision backbone to a 5-class target dataset:
import torch
import torch.nn as nn
import torchvision.models as models
# 1. Load pretrained ResNet-18 backbone with modern weights API
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)
# 2. Freeze all feature extraction backbone parameters
for param in model.parameters():
param.requires_grad = False
# 3. Replace the final 1000-class classifier head with a 5-class head
in_features = model.fc.in_features # 512
model.fc = nn.Sequential(
nn.Linear(in_features, 128),
nn.ReLU(inplace=True),
nn.Dropout(p=0.3),
nn.Linear(128, 5) # 5 target classes
)
# Verify trainable parameters
trainable = [name for name, p in model.named_parameters() if p.requires_grad]
frozen = [name for name, p in model.named_parameters() if not p.requires_grad]
print(f"Total Parameter Tensors: {len(list(model.parameters()))}")
print(f"Frozen Backbone Tensors: {len(frozen)}")
print(f"Trainable Head Tensors: {len(trainable)}")
Implications: security, privacy, performance, scalability, and cost
- Pretrained Weight Poisoning & Backdoor Attacks:
- Pretrained weights downloaded from untrusted public hubs can contain hidden trojans/backdoors (e.g. activating a malicious classification when a specific pixel watermark is present). Always verify model SHA-256 hashes and download only from official torchvision/HuggingFace repositories.
- Inference Latency & Quantization:
- Pretrained backbones can be quantized from FP32 down to INT8 post-training using PyTorch native quantization, achieving 4x memory reduction and 3x faster inference on CPU/mobile devices without retraining.
Alternatives: free, open source, and commercial
| Pretrained Vision Backbone | Architecture Type | Parameters | ImageNet Top-1 | Best For |
|---|---|---|---|---|
| ResNet-18 / 50 | Residual CNN | 11M / 25M | 69.8% / 76.1% | Fast edge & baseline deployment |
| MobileNet-v3-Large | Depthwise Separable CNN | 5.4M | 75.2% | Mobile / iOS / Android apps |
| ConvNeXt-Tiny | Modernized CNN | 28M | 82.1% | High-accuracy convolutional vision |
| ViT-B/16 (Vision Transformer) | Self-Attention Transformer | 86M | 81.8% | Large target datasets with high compute |
| CLIP (OpenAI / OpenCLIP) | Contrastive Language-Image | 150M | Zero-Shot | Zero-shot & open-vocabulary search |
Comparison with related concepts
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β TRANSFER LEARNING STRATEGY MATRIX β
ββββββββββββββββββββββΌββββββββββββββββββββββββΌββββββββββββββββββββββββββββ€
β Dataset Size β Visual Similarity β Optimal Strategy β
ββββββββββββββββββββββΌββββββββββββββββββββββββΌββββββββββββββββββββββββββββ€
β Tiny (< 500 imgs) β High (Natural images) β Freeze Backbone + Train FCβ
β Tiny (< 500 imgs) β Low (Medical/Thermal) β Early Layer Freeze + FC β
β Medium (1k-10k) β High (Natural images) β 2-Stage Fine-Tuning β
β Large (> 50k imgs) β Any β Full Fine-Tuning (Diff LR)β
ββββββββββββββββββββββ΄ββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββ
When to use it β and when not to
When to USE Transfer Learning:
- Every production computer vision project with fewer than 1 million training images.
- When development timelines require high accuracy within hours instead of weeks.
When NOT to use:
- When the target domain fundamentally violates 2D optical spatial physics (e.g. raw 1D audio time-series where 1D models or spectrogram-specific architectures are required).
The AI connection
- Almost nobody trains a vision model from scratch. Transfer learning is the default path, and training from random initialisation is the exception that needs justifying.
- Features transfer in a spectrum. Early layers learn edges that are useful for any image; later layers learn class-specific structure that is not.
- Catastrophic forgetting is the risk being managed β a large gradient into a pretrained backbone overwrites delicately balanced detectors with noise.
- Fine-tuning learning rates are orders of magnitude smaller than pretraining ones, which is a different regime rather than a smaller version of the same thing.
- Self-supervised pretraining removed the label bottleneck, which is why the strongest backbones now come from data nobody annotated.
Knowledge check
- What causes Catastrophic Forgetting when fine-tuning a pretrained model?
- Why are early convolutional layers universally transferable across distinct image domains?
- How do you construct an optimizer with differential learning rates in PyTorch?
- What is the difference in compute requirements between Linear Probing and Full Fine-Tuning?
- Why must image input tensors be normalized with the pretraining datasetβs mean and standard deviation?
Hands-on exercise
In this lab, you will build and test a production-grade Transfer Learning pipeline in PyTorch: instantiate a pretrained vision backbone, implement parameter freezing and verification routines, attach a custom multi-layer classification head, configure a two-stage differential learning rate optimizer, and benchmark training speedups against from-scratch models.
Expected output
[Transfer Learning Verification Suite]
Initializing Pretrained Backbone (ResNet Architecture):
Freezing Backbone Parameters: 60 tensors frozen (requires_grad = False)
Attaching Custom Classifier Head (in_features=512 -> 5 classes):
Trainable Parameters: 65,925 (0.58% of total network)
Testing 2-Stage Optimizer Configuration:
Phase 1 (Linear Probing): Head LR = 1.00e-03, Backbone LR = 0.00e+00 [HEAD WARMUP READY]
Phase 2 (Differential Fine-Tuning): Head LR = 1.00e-03, Layer3 LR = 1.00e-05, Layer4 LR = 5.00e-05
Forward & Backward Verification on Synthetic Batch (4x3x32x32):
Loss Backward Computed: [GRADIENTS CONFUSED ONLY TO UNFROZEN PARAMETERS]
Test Suite: 4 passed in 0.29s
Validate your work
Run the automated test runner:
./tests/run_tests.sh
Troubleshooting
- If backbone parameters receive gradients during Phase 1, verify that
param.requires_grad = Falsewas executed on all backbone layers before creating the optimizer. - Ensure the optimizer is re-instantiated or parameter groups are updated when transitioning from Phase 1 to Phase 2.
Common mistakes
- Using Default Random Init for the Whole Network: Forgetting to pass
weights="DEFAULT"or loading weights properly, resulting in training an uninitialized model.
Practice assignment
- Implement Embedding Extraction & Caching: Run your dataset once through a frozen backbone, save the 512-dimensional feature vectors to disk, and train a Logistic Regression / Linear layer in 2 seconds.
- Implement Gradual Unfreezing: Unfreeze layer 4 first, train for 2 epochs, then unfreeze layer 3, train for 2 epochs, demonstrating steady loss reduction.
Extension challenge
Implement CLIP Zero-Shot Image Classification:
- Load an open-source OpenAI / OpenCLIP model (
ViT-B/32). - Encode text prompt embeddings for target classes (βa photo of a catβ, βa photo of a dogβ).
- Compute cosine similarity between image embeddings and text embeddings to classify images with zero training examples.
Quiz
Q1. What is Catastrophic Forgetting during deep neural network fine-tuning, and how does a two-stage training schedule prevent it?
- When batch size is set to a prime number
- When a model loses its connection to the GPU memory bus
- When the validation dataset is accidentally deleted
- A random head's gradients wreck the pretrained backbone
Show answer
Answer: D. A random head's gradients wreck the pretrained backbone
A newly initialized linear head produces huge random losses. If the backbone is trainable immediately, large gradients scramble the carefully pretrained feature extractors. Freezing the backbone first stabilizes the head before gentle fine-tuning.
Q2. How do you freeze all convolutional backbone parameters in a PyTorch vision model so that they do not receive gradient updates?
- requires_grad = False
- model.eval()
- torch.no_grad()
- del model.backbone
Show answer
Answer: A. requires_grad = False
Setting param.requires_grad = False excludes those tensors from autograd graph tracking and gradient computation during backward().
Q3. If you have a very small target dataset (e.g. 200 medical X-ray images) that is visually distinct from ImageNet, which transfer learning strategy is most mathematically and empirically sound?
- Train a 152-layer ResNet from scratch with random initialization
- Freeze the backbone and train a linear head on its features
- Fine-tune all 152 layers with a huge learning rate (0.1)
- Delete all convolution layers and use an MLP
Show answer
Answer: B. Freeze the backbone and train a linear head on its features
With only 200 samples, fine-tuning millions of parameters guarantees severe overfitting. Freezing early/mid universal features and training a linear head is the optimal strategy.
Q4. How are differential (layer-wise) learning rates configured in a PyTorch optimizer?
- Set learning rate in model.forward()
- Create two separate Python scripts
- Pass per-group learning rates to the optimizer
- Multiply the loss by 0.01
Show answer
Answer: C. Pass per-group learning rates to the optimizer
PyTorch optimizers natively accept lists of dicts containing specific param subsets and custom learning rates, enabling fine-grained layer-wise tuning.
Q5. Why is it critical to normalize input images with the exact mean and standard deviation of the pretraining dataset (e.g. ImageNet mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])?
- It converts RGB images to CMYK
- PyTorch throws a compile error if standard deviation is not 0.229
- It makes the image file sizes smaller on disk
- The filters were trained under that exact input distribution
Show answer
Answer: D. The filters were trained under that exact input distribution
Convolutional kernels learn specific numerical scales for edge responses. Using the exact pretraining normalization ensures the input aligns with the distribution the weights expect.
Glossary
- Transfer Learning
- A machine learning paradigm where knowledge gained from solving one problem is applied to a different but related problem.
- Linear Probing
- Training only a linear classifier head on top of frozen pretrained feature representations.
- Fine-Tuning
- Continuing gradient descent training across the entire pretrained network (or subset of layers) on target dataset data.
- Catastrophic Forgetting
- The phenomenon where a neural network abruptly forgets previously learned knowledge upon learning new information.
- Differential Learning Rates
- Assigning smaller learning rates to early pretrained layers and larger learning rates to newly initialized decision heads.
- Pretrained Weights
- Neural network weight matrices saved after extensive training on a massive dataset (such as ImageNet-1k or LAION-5B).
- Backbone
- The core convolutional or transformer feature extractor that generates dense visual representations before the classifier head.
- Distribution Shift
- A change in the statistical distribution of input data between the source pretraining dataset and the target downstream task.
Sources and further reading
- How transferable are features in deep neural networks? β NeurIPS (accessed 2026-08-29)
- Do Better ImageNet Models Transfer Better? β CVPR (accessed 2026-08-29)
- Universal Language Model Fine-tuning for Text Classification β ACL (accessed 2026-08-29)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.