Deep Learning β€Ί Convolutional Networks and Vision β€Ί Day 211

Day 211: Convolutions

Day 211 of 365 β€” Convolutions

Master the fundamental mechanics of 2D convolutions in computer vision: derive spatial weight sharing, sliding kernel cross-correlation, padding and stride arithmetic, multi-channel feature maps, Sobel edge detectors, and PyTorch nn.Conv2d implementation.

Course
Deep Learning
Category
Convolutional Networks and Vision
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-211-convolutions

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-211-convolutions
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

In Weeks 29 and 30, we built fully connected neural networks (MLPs). When processing tabular data or low-resolution flattened vectors (such as 28x28 MNIST images), MLPs perform reasonably well.

However, consider applying a standard Fully Connected layer to a modern high-resolution color photograph from a smartphone camera ($1920 \times 1080 \times 3 \approx 6.2 million inputs$):

To solve this, computer vision pioneers introduced Convolutional Neural Networks (CNNs). By enforcing Local Connectivity and Spatial Weight Sharing, convolutions reduce parameter counts by orders of magnitude while establishing Translation Equivariance: a vertical edge detector learned in the top-left corner of an image operates identically in the bottom-right corner.

Today, you will master the mathematical derivation, spatial dimension arithmetic, and PyTorch implementation of 2D Convolutions (nn.Conv2d).


The idea in plain language

Diagram: a dense layer compared with a convolutional layer on image input

Diagram: the three assumptions a convolution makes about image data

Imagine holding a small magnifying glass (a $3 \times 3$ Kernel) over a giant printed mosaic painting:


Historical background

Diagram: the progression from receptive fields to learned convolutional filters

  1. 1959–1962 (Hubel & Wiesel - Receptive Fields in the Visual Cortex): David Hubel and Torsten Wiesel discovered that neurons in the mammalian visual cortex (area V1) do not respond to whole visual scenes, but rather to local oriented edges and slits within localized receptive fields (Nobel Prize, 1981).
  2. 1980 (Fukushima - The Neocognitron): Kunihiko Fukushima introduced the Neocognitron, introducing alternating layers of simple cells (convolution) and complex cells (pooling).
  3. 1989–1998 (Yann LeCun - LeNet-5): Yann LeCun and collaborators integrated backpropagation with 2D convolutions and weight sharing, creating LeNet-5 to automatically read handwritten bank checks.
  4. 2012 (AlexNet): Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton scaled deep CNNs on GPUs to win the ImageNet challenge by a historic margin, inaugurating the modern Deep Learning revolution.

What it is β€” and what it is not

What a 2D Convolution IS:

What it is NOT:


Why it was created and what problems it solves

Diagram: the three limitations of dense layers on images that convolution addresses

2D Convolution operation showing sliding 3x3 kernel over 5x5 input feature map computing dot product and generating output feature map

Convolutions eliminate three fundamental bottlenecks of fully connected vision models:

  1. The Parameter Explosion: Replacing millions of unconstrained weights with small $3 \times 3$ shared kernels ($9 weights$ per channel).
  2. Loss of 2D Spatial Structure: Preserving pixel neighborhood relationships rather than flattening images into arbitrary 1D vectors.
  3. Translation Vulnerability: Guaranteeing that localized visual patterns (edges, corners, textures) are recognized regardless of where they appear in the frame.

How it works

Let us dissect the mathematics of 2D convolutions, spatial dimension arithmetic, and channel propagation.


1. Mathematical 2D Convolution vs Cross-Correlation

Diagram: mathematical convolution compared with the cross-correlation frameworks implement

In deep learning frameworks (PyTorch, TensorFlow, JAX), the operation implemented under the name β€œconvolution” is technically Discrete 2D Cross-Correlation.

For a single-channel 2D input image X and a kernel K of size k_h x k_w:

Y[i, j] = sum_{m=0}^{k_h - 1} sum_{n=0}^{k_w - 1} X[i + m, j + n] * K[m, n]

The Difference from Formal Mathematics:


2. Multi-Channel Input and Output Arithmetic

Modern images and hidden layers are 3D volumes: (C_in, H_in, W_in).

Padding and Stride effects on feature map spatial resolution showing valid padding, same padding, and downsampling with stride 2

To map from C_in input channels to C_out output channels:

  1. We construct C_out distinct 3D filter kernels.
  2. Each filter kernel has dimensions (C_in, k_h, k_w).
  3. To compute a single output channel c_out:
    • Compute 2D convolutions across all C_in input channels with the corresponding kernel slice.
    • Sum the results across all input channels:
      Y[c_out, i, j] = sum_{c_in=0}^{C_in - 1} (X[c_in] * K[c_out, c_in])[i, j] + bias[c_out]

Parameter Count Formula:

Total Parameters = (C_out * C_in * k_h * k_w) + (C_out if bias else 0)

3. Spatial Dimension Formula (Padding & Stride)

Diagram: how kernel size, padding and stride determine output spatial dimensions

When sliding a kernel of size K across an input of height H_in with padding P (zero-padding added to both sides) and stride S (step size):

H_out = floor((H_in - K + 2 * P) / S) + 1
W_out = floor((W_in - K + 2 * P) / S) + 1

Common Padding Conventions:


4. Classical Hand-Crafted Spatial Filters

Before deep learning, computer vision scientists engineered explicit convolution kernels to detect visual features:

Horizontal Sobel Filter (Detects Horizontal Edges):

K_sobel_h = [[-1, -2, -1],
             [ 0,  0,  0],
             [ 1,  2,  1]]

Vertical Sobel Filter (Detects Vertical Edges):

K_sobel_v = [[-1,  0,  1],
             [-2,  0,  2],
             [-1,  0,  1]]

Gaussian Blur Filter (Noise Reduction):

K_gaussian = (1/16) * [[1, 2, 1],
                       [2, 4, 2],
                       [1, 2, 1]]

In a Convolutional Neural Network, the network learns these edge detectors automatically in its early layers through gradient descent!


5. Dimensional Variations: 1D, 2D, and 3D Convolutions

The sliding window principle extends naturally across different spatial and temporal dimensionalities:


6. Transposed Convolutions and Upsampling

In generative models, semantic segmentation networks (like U-Net), and autoencoders, we frequently need to increase the spatial resolution of a feature map (e.g. from 16x16 up to 32x32).


7. Depthwise Separable Convolutions (MobileNet)

Diagram: standard, depthwise separable and transposed convolutions compared

Standard convolutions perform spatial filtering and channel cross-talk simultaneously in a single heavy step.


An everyday analogy

Diagram: one stamp applied across a surface as an analogy for weight sharing

Think of a cookie cutter stamping dough:


Examples in practice

Let us build and inspect a 2D convolution in PyTorch, applying hand-crafted edge detectors and learnable layers:

import torch
import torch.nn as nn
import torch.nn.functional as F

# 1. Instantiate a standard PyTorch Conv2d layer
conv = nn.Conv2d(
    in_channels=3,     # RGB Image
    out_channels=16,   # 16 distinct feature maps
    kernel_size=3,     # 3x3 spatial filter
    stride=1,          # Step size of 1 pixel
    padding=1          # 'Same' padding: preserves spatial size
)

# 2. Forward pass with batch of 4 images (4, 3, 64, 64)
x = torch.randn(4, 3, 64, 64)
out = conv(x)
print(f"Input Shape:  {x.shape}")
print(f"Output Shape: {out.shape}")  # (4, 16, 64, 64)

# 3. Apply a custom Sobel Edge Detector via F.conv2d
sobel_v = torch.tensor([
    [-1.0, 0.0, 1.0],
    [-2.0, 0.0, 2.0],
    [-1.0, 0.0, 1.0]
]).view(1, 1, 3, 3)

# Grayscale test patch
gray_img = torch.zeros(1, 1, 6, 6)
gray_img[:, :, :, 3:] = 1.0 # Sharp vertical boundary in the middle

edges = F.conv2d(gray_img, sobel_v, padding=1)
print(f"Detected Vertical Edge Magnitudes:\n{edges[0, 0]}")

Implications: security, privacy, performance, scalability, and cost

Diagram: the parameter, compute, memory and inductive bias implications of convolution

  1. Receptive Field Growth & Feature Hierarchies:
    • Layer 1 ($3 \times 3$) sees $3 \times 3$ pixel patches (lines, edges).
    • Layer 2 ($3 \times 3$) sees $5 \times 5$ pixel patches (corners, junctions).
    • Layer 5 sees $11 \times 11$ patches (eyes, noses, wheels).
    • Deep CNNs compose simple edges into complex semantic concepts hierarchically.
  2. GPU Compute Optimizations (im2col & Winograd):
    • Naive nested 6-deep loops for convolutions are slow.
    • Modern GPU libraries (cuDNN) convert 2D convolutions into single matrix multiplications via im2col (GEMM) or use Winograd Minimal Filtering Algorithms to achieve 4x faster execution for $3 \times 3$ kernels.

Alternatives: free, open source, and commercial

Vision OperatorMechanismParametersBest Suited For
Standard 2D Conv (Conv2d)Dense $C_in \times C_out$ 3D spatial dot productMediumGeneral Computer Vision
Depthwise Separable ConvSpatial Conv per channel + $1 \times 1$ Pointwise ConvVery Low (8x fewer)Mobile architectures (MobileNet)
Dilated / Atrous ConvSpaces kernel elements with holesMediumSemantic Segmentation (DeepLab)
Vision Transformer (ViT)Non-local Self-Attention over flattened image patchesHighLarge-scale pre-training datasets

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    SPATIAL OPERATOR TAXONOMY                           β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Operator              β”‚ Connectivity Scope β”‚ Parameter Sharing         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Fully Connected (FC)  β”‚ Global (All-to-All)β”‚ None (Independent weights)β”‚
β”‚ 2D Convolution (Conv) β”‚ Local Neighborhood β”‚ Strict Spatial Sharing    β”‚
β”‚ Max Pooling (Pool)    β”‚ Local Neighborhood β”‚ Parameter-Free (Fixed max)β”‚
β”‚ Self-Attention (ViT)  β”‚ Global Dense Patch β”‚ Dynamic Data-Dependent    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when convolution is the right operation

When to USE 2D Convolutions:

When NOT to use:


The AI connection

Knowledge check

  1. Why does a $3 \times 3$ convolution have significantly fewer parameters than a fully connected layer on high-resolution images?
  2. What is the output spatial size of a $64 \times 64$ feature map passed through a convolution with $K=5$, $S=1$, and $P=0$?
  3. How does PyTorch handle multi-channel inputs when mapping from 3 RGB channels to 32 output channels?
  4. What is the difference between Translation Equivariance and Translation Invariance?
  5. How does the im2col algorithm accelerate convolution operations on modern GPUs?

Hands-on exercise

In this lab, you will build and test custom 2D convolution engines: implement a pure NumPy multi-channel custom_conv2d engine from scratch supporting arbitrary strides and padding, verify mathematical bitwise equivalence against PyTorch’s native nn.Conv2d, apply classical Sobel edge detection filters, and benchmark spatial dimension output formulas.

Expected output

[2D Convolution Engine Verification Suite]
Testing Custom 2D Multi-Channel Convolution (Input: 2x3x32x32 -> Kernel: 8x3x3x3):
  Applying Stride = 1, Padding = 1 (Same Padding):
  Output Spatial Dimensions: (2, 8, 32, 32) [DIMENSION MATCH]
  Max Absolute Difference vs PyTorch nn.Conv2d: 1.1921e-07 [BITWISE EQUIVALENT]
Testing Sobel Edge Detection Filter on Synthetic Step Edge:
  Horizontal Edge Response Peak: 4.0000
  Vertical Edge Response Peak:   0.0000 [DIRECTIONAL ISOLATION VERIFIED]
Testing Strided Downsampling (Input: 32x32, K=3, S=2, P=1):
  Output Spatial Dimensions: (2, 8, 16, 16) [EXACT HALVING MATCH]
Test Suite: 4 passed in 0.26s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a $1 \times 1$ Pointwise Convolution (nn.Conv2d(in_c, out_c, kernel_size=1)) and verify how it acts as a cross-channel linear projection without changing spatial height and width.
  2. Implement Depthwise Separable Convolution (Depthwise conv followed by Pointwise conv) and measure the parameter reduction compared to standard Conv2d.

Extension challenge

Implement Dilated / Atrous Convolution:

Quiz

Q1. What is the primary architectural advantage of a 2D Convolutional layer over a Fully Connected Linear layer when processing 2D image data?

  1. Convolutional layers only work on grayscale images
  2. Convolutional layers do not require activation functions
  3. Weight sharing and local connectivity
  4. Linear layers require GPU hardware while convolutions only run on CPU
Show answer

Answer: C. Weight sharing and local connectivity

A fully connected layer on a 1000x1000 RGB image would require billions of parameters, ignoring spatial locality. Convolutions use small shared kernels (e.g. 3x3), enforcing local connectivity and translation equivariance.

Q2. If an input feature map has dimensions (C_in=3, H=32, W=32), what is the spatial resolution of the output feature map after passing through a convolutional layer with Kernel=3, Stride=2, and Padding=1?

  1. (C_out, 8, 8)
  2. (C_out, 30, 30)
  3. (C_out, 32, 32)
  4. (C_out, 16, 16)
Show answer

Answer: D. (C_out, 16, 16)

Using H_out = floor((H_in - K + 2*P) / S) + 1: floor((32 - 3 + 2(1)) / 2) + 1 = floor(31 / 2) + 1 = 15 + 1 = 16.

Q3. How many learnable weight parameters (excluding bias) are contained in a standard PyTorch nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3)?

  1. 64 * 128 * 3 * 3 = 73,728
  2. 64 * 128 = 8,192
  3. 128 * 3 * 3 = 1,152
  4. 64 * 3 * 3 = 576
Show answer

Answer: A. 64 * 128 * 3 * 3 = 73,728

Each of the 128 output channels requires a 3D kernel of shape (in_channels, kernel_height, kernel_width) = (64, 3, 3). Total weights = 128 * 64 * 3 * 3 = 73,728.

Q4. What is the key difference between mathematical convolution and the cross-correlation operation implemented in PyTorch nn.Conv2d?

  1. Cross-correlation uses division instead of multiplication
  2. True convolution flips the kernel; PyTorch does not
  3. PyTorch convolution only works on 1D vectors
  4. Mathematical convolution cannot compute gradients
Show answer

Answer: B. True convolution flips the kernel; PyTorch does not

True convolution flips the filter kernel by 180 degrees. Deep learning libraries implement cross-correlation without flipping for efficiency; the learned weights simply adapt to the un-flipped orientation.

Q5. What property describes how a convolutional feature map shifts predictably when the input image is translated spatially?

  1. Scale Invariance
  2. Translation Invariance
  3. Translation equivariance
  4. Rotational Symmetry
Show answer

Answer: C. Translation equivariance

Convolutions are equivariant to translation: translating an input shifts its feature map activations identically. Pooling layers and deep classification heads subsequently introduce translation invariance.

Glossary

Convolution
A mathematical operation on two functions producing a third function expressing how the shape of one is modified by the other.
Kernel
A small matrix of learnable weights (e.g. 3x3) that slides over an input tensor to extract local spatial features.
Feature Map
The 2D output tensor generated by convolving a specific filter kernel across an input feature volume.
Stride
The step size (in pixels) by which the convolution kernel moves across the input spatial dimensions.
Padding
Adding extra border pixels (typically zeros) around the input tensor perimeter to control output spatial resolution.
Receptive Field
The specific spatial region in the input image that directly influences the activation of a particular neuron in a feature map.
Translation Equivariance
A mathematical property where transforming an input shifts the resulting output representation by the exact same transformation.
Sobel Filter
A classical discrete differentiation filter kernel used to compute horizontal or vertical image gradient approximations.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.