Deep Learning β€Ί Convolutional Networks and Vision β€Ί Day 212

Day 212: CNN Architectures

Day 212 of 365 β€” CNN Architectures

Master landmark Convolutional Neural Network architectures: trace the evolution from LeNet-5 and AlexNet to VGG-16 and ResNet, derive receptive fields, implement residual skip connections from scratch in PyTorch, and understand feature hierarchy emergence.

Course
Deep Learning
Category
Convolutional Networks and Vision
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-212-cnn-architectures

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-212-cnn-architectures
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

In Day 211, we mastered the local mechanics of a single 2D Convolutional layer. However, building an accurate, robust visual perception system requires arranging dozens β€” or hundreds β€” of convolutional, normalization, pooling, and residual blocks into a coherent Deep Architectural Topology.

Between 2012 and 2015, the field of computer vision underwent an unprecedented evolutionary leap:

Understanding how and why these architectures were designed is essential for every AI engineer. The architectural principles pioneered here β€” homogeneous $3 \times 3$ stacking, receptive field expansion, and residual skip connections β€” form the bedrock not only of modern computer vision (ConvNeXt, YOLO, U-Net) but also of Large Language Model Transformers (which rely fundamentally on residual streams).

Today, you will master the architectural evolution, receptive field mathematics, and PyTorch implementation of VGG and Deep Residual Networks (ResNet).


The idea in plain language

Diagram: the degradation problem in deep plain networks and the residual solution

Imagine constructing a high-rise skyscraper:


Historical background

Timeline evolution of modern CNN architectures from LeNet-5 and AlexNet to VGG-16 and ResNet showing layer depth and structural innovations

  1. 1998 (LeNet-5): Yann LeCun designed a 5-layer network using $5 \times 5$ convolutions and average pooling to classify handwritten digits on bank checks.
  2. 2012 (AlexNet): Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton scaled CNNs to 8 layers with ReLUs, Dropout, and GPU parallelization, destroying classical computer vision benchmarks.
  3. 2014 (VGG-16 & GoogLeNet/Inception): Visual Geometry Group (Oxford) proved that deep networks with purely $3 \times 3$ filters outclassed shallow networks with large filters. Simultaneously, Google introduced Inception multi-scale inception modules.
  4. 2015 (ResNet): Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun introduced Residual Learning, winning CVPR 2016 Best Paper and enabling training of networks with over 1,000 layers.

What it is β€” and what it is not

What Modern CNN Architectures ARE:

What they are NOT:

Diagram: a dense classification head compared with global average pooling


Why it was created and what problems it solves

Deep CNN architectures solve three major architectural challenges:

  1. The Receptive Field vs Parameter Tradeoff: Stacking small $3 \times 3$ filters achieves massive receptive fields while using 28% to 50% fewer parameters than large filters.
  2. The Degradation Problem: Plain 56-layer networks exhibited higher training and test error than 20-layer networks. ResNet eliminated degradation entirely.
  3. The Dense Layer Bottleneck: Replacing 100M-parameter fully connected heads with Global Average Pooling reduced model sizes by 80% while dramatically reducing overfitting.

How it works

Diagram: how the receptive field grows with depth in a convolutional network

Diagram: four convolutional architecture families and their defining ideas

Let us dissect the mathematical foundations of Receptive Fields, VGG $3 \times 3$ Stacking, and ResNet Residual Blocks.


1. The Power of Stacking $3 \times 3$ Convolutions (VGG Principle)

Diagram: one large convolution kernel compared with three stacked small ones

Why did VGG discard the $7 \times 7$ and $11 \times 11$ filters of AlexNet?

Consider stacking two consecutive $3 \times 3$ convolutions (with stride 1):

Mathematical Comparison for $C$ Channels:


2. The ResNet Residual Block & Gradient Highway

Diagram: how a residual connection preserves gradient magnitude across depth

ResNet residual block architecture showing convolutional path and identity shortcut connection creating direct gradient flow

In a traditional plain feedforward network, a block learns an underlying mapping:

H(x)

In a Residual Network (ResNet), we hypothesize that it is significantly easier to optimize the residual mapping:

F(x) = H(x) - x

The output of the block is computed as:

y = F(x, {W_i}) + x

where F(x) is the residual function (e.g. two $3 \times 3$ convolutions with BatchNorm and ReLU) and x is the Identity Shortcut Connection.

Why Residual Connections Solve Vanishing Gradients (Backpropagation Math):

Using the chain rule, the gradient of the loss L with respect to the input x of a residual block is:

dL / dx = (dL / dy) * (d / dx)(F(x) + x)
dL / dx = (dL / dy) * (dF / dx + 1.0)
dL / dx = (dL / dy) * (dF / dx) + (dL / dy)

Look closely at the second term: + (dL / dy).


3. BasicBlock vs BottleneckBlock

Diagram: the basic residual block compared with the bottleneck block

  1. BasicBlock (ResNet-18, ResNet-34):
    • Two $3 \times 3$ convolutions with matching channel depth.
  2. BottleneckBlock (ResNet-50, ResNet-101, ResNet-152):
    • Designed for deep networks to reduce computational complexity:
      1. $1 \times 1$ Conv (Compresses 256 channels down to 64).
      2. $3 \times 3$ Conv (Performs spatial filtering on 64 channels).
      3. $1 \times 1$ Conv (Expands 64 channels back up to 256).
    • This β€œsqueeze-and-expand” design enables 3x deeper networks for the same computational budget.

4. Inception & GoogLeNet: Multi-Scale Convolutions

While VGG deepened networks using purely $3 \times 3$ filters, Christian Szegedy and Google Research took an alternative architectural approach in 2014 with GoogLeNet (Inception-v1):


5. DenseNet: Feature Reuse and Concatenation

In 2017, Gao Huang et al. proposed DenseNet (Densely Connected Convolutional Networks):


6. EfficientNet: Compound Model Scaling

In 2019, Mingxing Tan and Quoc Le addressed how to scale vision networks systematically:


An everyday analogy

Diagram: adding floors without structural support compared with a designed frame

Think of a multi-tiered corporate hierarchy:


Examples in practice

Let us build a complete, modular ResNet-18 / BasicBlock implementation in PyTorch:

import torch
import torch.nn as nn
from typing import Type

class BasicResidualBlock(nn.Module):
    def __init__(self, in_channels: int, out_channels: int, stride: int = 1):
        super().__init__()
        # Conv Path
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3,
                               stride=stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)

        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3,
                               stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)

        # Shortcut Projection (if dimensions change)
        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(out_channels)
            )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        residual = self.shortcut(x)
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += residual
        return self.relu(out)

class MiniResNet(nn.Module):
    def __init__(self, num_classes: int = 10):
        super().__init__()
        # Initial stem
        self.stem = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1, bias=False),
            nn.BatchNorm2d(32),
            nn.ReLU(inplace=True)
        )
        # Residual stages
        self.layer1 = BasicResidualBlock(32, 32, stride=1)
        self.layer2 = BasicResidualBlock(32, 64, stride=2) # Downsample to 16x16
        self.layer3 = BasicResidualBlock(64, 128, stride=2) # Downsample to 8x8

        # Global Average Pooling + Classifier Head
        self.gap = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(128, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.stem(x)
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.gap(x)
        x = torch.flatten(x, 1)
        return self.fc(x)

# Verify forward pass
model = MiniResNet(num_classes=10)
x = torch.randn(4, 3, 32, 32)
logits = model(x)
print(f"Input: {x.shape} -> MiniResNet Logits: {logits.shape}")

Implications: security, privacy, performance, scalability, and cost

Diagram: the depth, memory, transfer and default implications of convolutional architectures

  1. Memory Bandwidth in Residual Blocks:
    • Identity addition out += residual requires storing the input activation tensor x in GPU memory throughout the forward pass until the end of the block, increasing VRAM usage during training.
  2. Modern ConvNeXt Architecture (2022):
    • Zhuang Liu et al. modernized standard ResNet by incorporating Transformer design choices (depthwise $7 \times 7$ convolutions, inverted bottleneck, LayerNorm, GELU), matching Vision Transformer performance with standard convolution efficiency.

Alternatives: free, open source, and commercial

ArchitectureDefining InnovationLayersImageNet Top-1 Acc
VGG-16 (2014)Stacks of $3 \times 3$ Convolutions1671.5%
ResNet-50 (2015)Residual Skip Connections ($y = F(x) + x$)5076.1%
EfficientNet-B0 (2019)Compound Model Scaling (Depth, Width, Resolution)~5077.1%
ConvNeXt-T (2022)Modernized CNN with Transformer Design~5082.1%
Vision Transformer (ViT-B)Pure Multi-Head Self-Attention on Patches1281.8%

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   CNN ARCHITECTURAL EVOLUTION MATRIX                   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Model             β”‚ Key Innovation     β”‚ Classifier Head               β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ AlexNet (2012)    β”‚ ReLUs + Dropout    β”‚ 3x Fully Connected (4096-4096)β”‚
β”‚ VGG-16 (2014)     β”‚ Pure 3x3 Stacking  β”‚ 3x Fully Connected (100M wt)  β”‚
β”‚ ResNet-50 (2015)  β”‚ Skip Connections   β”‚ Global Average Pooling + 1 FC β”‚
β”‚ ConvNeXt (2022)   β”‚ 7x7 Depthwise Conv β”‚ Global Average Pooling + 1 FC β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: how to choose a convolutional architecture

When to USE ResNet / ConvNeXt:

When to USE Vision Transformers (ViT):


The AI connection

Knowledge check

  1. Why does a stack of three $3 \times 3$ convolutions have an effective $7 \times 7$ receptive field?
  2. What is the mathematical formulation of the backward gradient through a ResNet skip connection?
  3. How does Global Average Pooling (GAP) prevent overfitting compared to fully connected heads?
  4. What is the structural difference between a ResNet BasicBlock and a BottleneckBlock?
  5. What is the Degradation Problem, and why is it distinct from standard overfitting?

Hands-on exercise

In this lab, you will build and test a modular ResNet vision framework in PyTorch: implement BasicResidualBlock from scratch with 1x1 projection shortcuts, construct a MiniResNet model with stem, residual stages, and global average pooling, verify uninhibited gradient backpropagation across deep residual chains, and benchmark parameter efficiency.

Expected output

[ResNet Architecture Verification Suite]
Constructing Custom BasicResidualBlock (in=32, out=64, stride=2):
  Forward Pass (Input: 4x32x16x16 -> Output: 4x64x8x8): [DIMENSION MATCH]
  Shortcut Projection 1x1 Conv Verified: [PROJECTION ALIGNED]
Constructing Complete MiniResNet [Stem -> Stage1 -> Stage2 -> Stage3 -> GAP -> FC10]:
  Total Learnable Parameters: 142,570
  Evaluating Gradient Flow across 6 Residual Blocks:
  Minimum Layer Gradient Norm: 0.0842 [NO VANISHING GRADIENTS]
  Maximum Layer Gradient Norm: 0.2410 [STABLE GRADIENT FLOW]
Test Suite: 4 passed in 0.28s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement the ResNet Bottleneck Block (1x1 reduce -> 3x3 conv -> 1x1 expand) and verify its tensor dimension transformations.
  2. Implement Squeeze-and-Excitation (SE) Attention inside a residual block to dynamically recalibrate channel feature responses.

Extension challenge

Implement Stochastic Depth (DropPath) inside your residual blocks:

Quiz

Q1. Why did VGG-16 replace large convolutional filters (like 11x11 and 5x5 in AlexNet) with stacks of small 3x3 convolutional filters?

  1. Two 3x3s match a 5x5 with fewer parameters and two ReLUs
  2. 3x3 convolutions do not require GPUs
  3. 5x5 convolutions are banned by PyTorch
  4. 3x3 convolutions eliminate the need for training data
Show answer

Answer: A. Two 3x3s match a 5x5 with fewer parameters and two ReLUs

A stack of two 3x3 convolutions has a 5x5 receptive field with 2 * (3^2 * C^2) = 18 C^2 parameters versus 25 C^2 for a single 5x5 layer (28% parameter reduction), plus an extra non-linear activation.

Q2. What fundamental optimization barrier did Kaiming He et al. identify that caused deep 56-layer networks to perform worse on both training and test sets than 20-layer networks, and how did ResNet solve it?

  1. Lack of RAM on GPUs; solved by deleting layers
  2. Degradation, solved by residual skip connections
  3. Overfitting; solved by adding 90% dropout
  4. Floating point rounding errors; solved by 64-bit precision
Show answer

Answer: B. Degradation, solved by residual skip connections

The degradation problem was not overfitting (since training loss was worse too). Residual skip connections allow gradients to flow directly through the identity mapping dL/dx = dL/dy * (dF/dx + 1).

Q3. In a ResNet BasicBlock, what must be done when the residual function F(x) halves the spatial resolution (stride=2) or increases the number of feature channels?

  1. Discard the skip connection entirely for that block
  2. Pad x with random Gaussian noise
  3. Use a 1x1 convolution with stride 2 on the shortcut
  4. Downsample the output using MaxPool3D
Show answer

Answer: C. Use a 1x1 convolution with stride 2 on the shortcut

Tensor addition F(x) + x requires identical shapes. When channels increase or spatial size decreases, a 1x1 projection conv with matching stride aligns x to F(x).

Q4. What is Global Average Pooling (GAP), and why did modern CNNs (ResNet, Inception) replace giant fully connected classifier heads with GAP?

  1. GAP replaces the loss function
  2. GAP doubles the image resolution
  3. GAP multiplies all pixel values by 255
  4. GAP averages each map to one number, dropping dense params
Show answer

Answer: D. GAP averages each map to one number, dropping dense params

In VGG-16, the first dense layer FC1 accounted for over 100 million parameters. Modern networks use Global Average Pooling to reduce (C, H, W) to a C-dimensional vector, requiring only a single small linear layer.

Q5. How does the effective receptive field of a neuron change as we go deeper into a convolutional neural network?

  1. It grows with depth, so deep layers see global context
  2. It shrinks to a single pixel
  3. It stays fixed at exactly 3x3 pixels forever
  4. It inverts horizontally
Show answer

Answer: A. It grows with depth, so deep layers see global context

Receptive fields expand with each successive convolution and pooling layer, allowing the network to form a spatial hierarchy from local edges to object parts to full semantic objects.

Glossary

Residual Connection
A skip connection that adds the input of a layer block directly to its output: y = F(x) + x.
Degradation Problem
The phenomenon where increasing network depth beyond a certain point leads to higher training error due to optimization difficulties.
Global Average Pooling
A spatial reduction operation that computes the mean of each 2D feature map channel, producing a 1D feature vector.
Receptive Field
The spatial extent in the raw input image that contributes to the activation of a particular neuron in a feature map.
Bottleneck Block
A residual block utilizing 1x1 convolutions to compress and restore channel dimensions around a 3x3 convolution.
VGG Block
A sequence of multiple 3x3 convolutions followed by a single 2x2 Max Pooling layer.
Feature Hierarchy
The progression in deep vision networks where early layers detect edges, middle layers detect textures/parts, and late layers detect objects.
Pointwise Convolution
A 1x1 convolution used for cross-channel linear combination and dimension reduction without spatial transformation.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.