Deep Learning β€Ί Convolutional Networks and Vision β€Ί Day 216

Day 216: Beyond Classification: Detection and Segmentation

Day 216 of 365 β€” Beyond Classification: Detection and Segmentation

Master dense spatial Computer Vision beyond classification: Object Detection (bounding boxes, IoU, NMS, mAP, YOLO vs Faster R-CNN), Semantic Segmentation (U-Net encoder-decoder skip architecture, Dice loss), and Instance Segmentation (Mask R-CNN, SAM).

Course
Deep Learning
Category
Convolutional Networks and Vision
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-216-beyond-classification-detection-and-segmentation

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-216-beyond-classification-detection-and-segmentation
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Image classification answers a simple global question: β€œWhat is the primary category depicted in this image?”

However, most real-world visual applications require far richer spatial understanding:

Today, you will master the principles, loss functions, and PyTorch architectures of Dense Visual Prediction: Object Detection with IoU and NMS, Semantic Segmentation with U-Net, and Foundation Segmentation models (Segment Anything Model - SAM).


The idea in plain language

Diagram: classification, detection and segmentation compared by output structure

Imagine inspecting a busy city street photograph:


Historical background

Visual taxonomy of computer vision tasks comparing image classification object detection semantic segmentation and instance segmentation

  1. 2014–2015 (R-CNN to Faster R-CNN): Ross Girshick and Kaiming He introduced Region-based CNNs, culminating in Faster R-CNN’s Region Proposal Network (RPN) for end-to-end two-stage detection.
  2. 2015 (U-Net & Ronneberger et al.): Introduced the U-shaped encoder-decoder architecture with lateral skip connections, sweeping the ISBI cell tracking challenge and becoming the universal standard for biomedical segmentation and diffusion model backbones.
  3. 2016 (YOLO & Joseph Redmon): Framed object detection as a single regression problem directly from full image pixels to bounding box coordinates, achieving 45 FPS real-time detection.
  4. 2023 (SAM - Segment Anything Model / Meta AI): Introduced promptable visual foundation models capable of zero-shot segmentation of any visual entity.

What it is β€” and what it is not

What Dense Prediction IS:

What it is NOT:


Why it was created and what problems it solves

Dense spatial prediction solves three major challenges:

  1. Multi-Object Scenes: Classifies and localizes dozens of overlapping objects in a single image.
  2. Precision Spatial Contours: Produces pixel-accurate masks required for robotics, medical oncology, and augmented reality.
  3. Scale Variation: Feature Pyramid Networks (FPN) detect objects ranging from 10 pixels to 1000 pixels in diameter.

How it works

Let us dissect the mathematical formulations of Intersection over Union (IoU), Non-Maximum Suppression (NMS), the U-Net architecture, and Dice Loss.


1. Intersection over Union (IoU) Arithmetic

Diagram: how intersection over union measures spatial overlap

Given two bounding boxes $A = [x_1^A, y_1^A, x_2^A, y_2^A]$ and $B = [x_1^B, y_1^B, x_2^B, y_2^B]$:

  1. Calculate the intersection rectangle coordinates:

    x_inter_1 = max(x_1^A, x_1^B)
    y_inter_1 = max(y_1^A, y_1^B)
    x_inter_2 = min(x_2^A, x_2^B)
    y_inter_2 = min(y_2^A, y_2^B)
  2. Compute intersection area:

    W_inter = max(0, x_inter_2 - x_inter_1)
    H_inter = max(0, y_inter_2 - y_inter_1)
    Area_inter = W_inter * H_inter
  3. Compute union area:

    Area_A = (x_2^A - x_1^A) * (y_2^A - y_1^A)
    Area_B = (x_2^B - x_1^B) * (y_2^B - y_1^B)
    Area_union = Area_A + Area_B - Area_inter
  4. Compute IoU:

    IoU = Area_inter / Area_union

2. Non-Maximum Suppression (NMS) Algorithm

Diagram: how non-maximum suppression reduces overlapping detections

Dense object detectors predict hundreds of candidate bounding boxes per object. NMS filters duplicates:

  1. Sort all predicted candidate boxes by confidence score in descending order.
  2. Select the box $B_max$ with the highest confidence; add it to the final detection list.
  3. Calculate the IoU between $B_max$ and all remaining candidate boxes.
  4. Remove any candidate box whose $IoU(B_max, B) > Threshold_NMS$ (typically $0.5$).
  5. Repeat until no candidate boxes remain.

3. The U-Net Segmentation Architecture

Diagram: the three stages of a U-Net segmentation architecture

UNet architecture diagram showing contracting encoder path bottleneck expanding decoder path and lateral skip connections

The U-Net architecture consists of three modular stages:

  1. Contracting Encoder Path:

    • Repeated blocks of two $3 \times 3$ convolutions followed by $2 \times 2$ Max Pooling (stride 2).
    • Halves spatial dimensions while doubling channel depth ($64 \rightarrow 128 \rightarrow 256 \rightarrow 512$), extracting rich semantic features.
  2. Bottleneck:

    • Deepest latent representation ($1024 channels$ at $1/16$ original spatial resolution).
  3. Expansive Decoder Path:

    • Repeated blocks of $2 \times 2$ Transposed Convolutions (or Bilinear Upsampling) doubling spatial resolution.
    • Lateral Skip Connections: Concatenates high-resolution spatial feature maps directly from the corresponding encoder stage:
      # Decoder step
      x_upsampled = self.upconv(x_deep)
      x_combined = torch.cat([x_upsampled, x_encoder_skip], dim=1)
      out = self.conv_block(x_combined)
  4. Final $1 \times 1$ Output Convolution: Projects multi-channel features to $C$ class logits per pixel $(N, C, H, W)$.


4. Dice Loss & Jaccard Loss for Imbalanced Masks

Diagram: pixel-wise cross-entropy compared with overlap-based segmentation losses

In medical imaging and satellite segmentation, background pixels often occupy $98%+$ of the image. Standard Cross-Entropy is overwhelmed by background true negatives.

SΓΈrensen-Dice Coefficient:

Dice = (2 * |X intersect Y|) / (|X| + |Y|)

For continuous predicted probability map $\hat(p) \in [0, 1]$ and binary ground truth mask $y \in (0, 1)$:

Dice Loss = 1 - ((2 * sum(p_hat * y) + epsilon) / (sum(p_hat) + sum(y) + epsilon))

where $\epsilon = 1.0$ (Laplacian smoothing) prevents division by zero.


5. Two-Stage vs Single-Stage Object Detectors

Diagram: single-stage detectors compared with two-stage detectors

The evolution of modern object detection divided into two distinct philosophical families:


6. Feature Pyramid Networks (FPN) for Multi-Scale Vision

Diagram: how a feature pyramid handles objects at multiple scales

A standard deep CNN downsamples spatial resolution by $32\times$, meaning a small 16-pixel object becomes a single sub-pixel feature vector by the final layer:


7. DETR: Transformer-Based Detection with Hungarian Matching

In 2020, Nicolas Carion et al. (Meta AI) revolutionized object detection with DETR (DEtection TRansformer):


8. The Segment Anything Model (SAM / Meta AI)

Diagram: task-specific segmentation models compared with a promptable foundation model

In 2023, Meta introduced SAM (Segment Anything Model), creating the first promptable visual foundation model for image segmentation:

  1. Heavy Image Encoder (MAE Pretrained ViT): Computes a dense $1024 \times 1024$ image embedding once (~50 ms on GPU).
  2. Flexible Prompt Encoder: Encodes arbitrary user prompts in real time (< 5 ms): single point clicks, bounding box prompts, or free-form text queries.
  3. Lightweight Mask Decoder: Rapidly predicts high-resolution binary segmentation masks and ambiguity confidence scores, revolutionizing medical annotation, robotics, and interactive VFX.

An everyday analogy

Diagram: a size label compared with a full set of measurements

Think of custom tailoring a bespoke suit:


Examples in practice

Let us build and verify a complete, modular U-Net Segmentation Network in PyTorch:

import torch
import torch.nn as nn

class DoubleConv(nn.Module):
    def __init__(self, in_channels: int, out_channels: int):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True)
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.block(x)

class MiniUNet(nn.Module):
    def __init__(self, in_channels: int = 3, num_classes: int = 1):
        super().__init__()
        # Encoder
        self.enc1 = DoubleConv(in_channels, 32)
        self.pool1 = nn.MaxPool2d(2)
        self.enc2 = DoubleConv(32, 64)
        self.pool2 = nn.MaxPool2d(2)

        # Bottleneck
        self.bottleneck = DoubleConv(64, 128)

        # Decoder
        self.up2 = nn.ConvTranspose2d(128, 64, kernel_size=2, stride=2)
        self.dec2 = DoubleConv(128, 64) # 64 from up + 64 from enc2 skip

        self.up1 = nn.ConvTranspose2d(64, 32, kernel_size=2, stride=2)
        self.dec1 = DoubleConv(64, 32)  # 32 from up + 32 from enc1 skip

        # Final 1x1 projection
        self.final_conv = nn.Conv2d(32, num_classes, kernel_size=1)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Contracting Path
        e1 = self.enc1(x)       # (B, 32, H, W)
        e2 = self.enc2(self.pool1(e1)) # (B, 64, H/2, W/2)

        # Bottleneck
        b = self.bottleneck(self.pool2(e2)) # (B, 128, H/4, W/4)

        # Expansive Path with Lateral Skips
        d2 = self.up2(b)
        d2 = torch.cat([d2, e2], dim=1) # Concatenate skip
        d2 = self.dec2(d2)

        d1 = self.up1(d2)
        d1 = torch.cat([d1, e1], dim=1) # Concatenate skip
        d1 = self.dec1(d1)

        return self.final_conv(d1)

# Verify forward pass
model = MiniUNet(in_channels=3, num_classes=2)
x = torch.randn(2, 3, 64, 64)
mask_logits = model(x)
print(f"Input Image: {x.shape} -> Output Pixel Mask Logits: {mask_logits.shape}")

Implications: security, privacy, performance, scalability, and cost

  1. Memory Footprint of Dense Predictions:
    • In semantic segmentation, intermediate activation tensors retain large spatial resolutions throughout the decoder path, consuming significantly more VRAM than classification models. Use gradient checkpointing (torch.utils.checkpoint) when training high-resolution $1024 \times 1024$ segmentation networks.
  2. Real-Time Edge Inference (YOLOv8 & TensorRT):
    • Single-stage anchor-free detectors (such as Ultralytics YOLOv8) can process 1080p video streams at over 120 FPS on an NVIDIA Jetson Orin edge device using FP16 TensorRT engines.

Alternatives: free, open source, and commercial

Dense Vision ModelTaskLatencyBenchmark Standard
YOLOv8 / YOLOv10Real-time Object Detection< 5 msReal-time industrial edge inspection
Faster R-CNNTwo-stage Object Detection~30 msHigh-accuracy research baseline
U-Net / U-Net++Semantic Segmentation~15 msMedical imaging & Diffusion models
Mask R-CNNInstance Segmentation~40 msInstance-level multi-task learning
SAM (Segment Anything)Zero-Shot Foundation Seg~100 msUniversal promptable annotation

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                     COMPUTER VISION TASK SPECTRUM                      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Task               β”‚ Output Representation β”‚ Core Evaluation Metric    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Classification     β”‚ (N, C) Logits         β”‚ Top-1 / Top-5 Accuracy    β”‚
β”‚ Object Detection   β”‚ List of [x1, y1, x2, y2, c, p] | mAP@0.5:0.95 (IoU)   β”‚
β”‚ Semantic Seg       β”‚ (N, C, H, W) Pixel mapβ”‚ Mean IoU / Dice Score     β”‚
β”‚ Instance Seg       β”‚ List of (Mask, Class) β”‚ Mask AP@0.5:0.95          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: choosing between classification, detection and segmentation

When to USE Detection / Segmentation:

When to stick with Classification:


The AI connection

Knowledge check

  1. What is the formula for Intersection over Union (IoU)?
  2. How does Non-Maximum Suppression (NMS) prevent duplicate bounding box predictions?
  3. Why does U-Net concatenate encoder features with decoder features?
  4. What is the fundamental difference between Semantic Segmentation and Instance Segmentation?
  5. Why is Dice Loss mathematically superior to Cross-Entropy on severely imbalanced medical masks?

Hands-on exercise

In this lab, you will build and test a modular dense prediction library in PyTorch: implement vectorized Intersection over Union (IoU) and Non-Maximum Suppression (NMS) algorithms from mathematical first principles, construct a MiniUNet encoder-decoder architecture with lateral skip connections, compute soft Dice Loss for imbalanced segmentation masks, and verify spatial output dimensions.

Expected output

[Dense Spatial Vision Verification Suite]
Testing Intersection over Union (IoU) Vectorized Calculator:
  Perfect Overlap (Box A == Box B): 1.0000 [EXACT MATCH]
  Partial Overlap (50% area match): 0.5000 [ACCURATE RATIO]
  Disjoint Boxes (Zero overlap):     0.0000 [ZERO VERIFIED]
Testing Non-Maximum Suppression (NMS) Filter:
  Candidate Boxes: 10 overlapping proposals
  Suppression Output: 2 distinct detected objects [DUPLICATES ELIMINATED]
Testing MiniUNet Segmentation Forward Pass:
  Input Image Shape: (2, 3, 64, 64) -> Output Mask Logits: (2, 2, 64, 64) [SPATIAL MATCH]
Testing Soft Dice Loss on Imbalanced Synthetic Mask:
  Dice Loss (Perfect Prediction): 0.0000
  Dice Loss (Inverted Prediction): 0.9998 [LOSS CALIBRATION VERIFIED]
Test Suite: 4 passed in 0.32s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Focal Loss for object detection ($FL(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t)$ with $\gamma = 2.0$) to downweight easy background true negatives.
  2. Implement Mean Average Precision (mAP@0.5) by integrating the precision-recall curve across IoU threshold 0.5.

Extension challenge

Implement a Promptable Segmentation Engine (Mini-SAM):

Quiz

Q1. What is Intersection over Union (IoU), and what does an IoU of 1.0 indicate between a predicted bounding box and a ground-truth bounding box?

  1. IoU is the learning rate divided by batch size
  2. IoU is the ratio of image width to image height
  3. IoU measures the number of classes detected
  4. Overlap divided by union; 1.0 means perfect alignment
Show answer

Answer: D. Overlap divided by union; 1.0 means perfect alignment

IoU = Area(Box_A intersect Box_B) / Area(Box_A union Box_B). It ranges from 0.0 (no overlap) to 1.0 (exact match).

Q2. Why does the U-Net segmentation architecture concatenate high-resolution feature maps from the contracting encoder directly to the expanding decoder via lateral skip connections?

  1. Skip connections restore the spatial detail the encoder lost
  2. Skip connections reduce the number of channels to zero
  3. Skip connections convert RGB images to grayscale
  4. Skip connections eliminate the need for training labels
Show answer

Answer: A. Skip connections restore the spatial detail the encoder lost

Pooling layers discard spatial coordinates while extracting high-level semantics. U-Net skip connections recover exact boundary locations by passing uncompressed spatial features to the decoder.

Q3. What is the role of Non-Maximum Suppression (NMS) in object detection algorithms like YOLO and Faster R-CNN?

  1. It suppresses negative loss gradients
  2. It drops overlapping boxes for the same object
  3. It darkens the brightest pixels in an image
  4. It suppresses CPU utilization
Show answer

Answer: B. It drops overlapping boxes for the same object

Dense object detectors predict dozens of candidate boxes for a single object. NMS iteratively retains the highest-confidence box and removes all neighboring boxes overlapping with IoU > threshold.

Q4. What is the difference between Semantic Segmentation and Instance Segmentation?

  1. Instance segmentation uses no neural networks
  2. Semantic segmentation is only for text
  3. Semantic labels pixels; instance separates each object
  4. Semantic segmentation produces 3D point clouds
Show answer

Answer: C. Semantic labels pixels; instance separates each object

If three sheep stand in a pasture, semantic segmentation colors all sheep pixels yellow as one blob. Instance segmentation assigns distinct masks: Sheep #1, Sheep #2, and Sheep #3.

Q5. Why is the Dice Loss (SΓΈrensen-Dice coefficient) commonly used instead of standard Cross-Entropy loss in medical image segmentation tasks?

  1. Dice loss replaces convolution with random guessing
  2. Dice loss works without GPU memory
  3. Dice loss can only be computed on integers
  4. Dice measures overlap, so it survives extreme imbalance
Show answer

Answer: D. Dice measures overlap, so it survives extreme imbalance

Standard Cross-Entropy is overwhelmed by the 99% background pixels. Dice Loss measures 2 * |A intersect B| / (|A| + |B|), penalizing false negatives on small lesion regions.

Glossary

Object Detection
Locating instances of semantic objects in images and predicting their categorical class and bounding box coordinates.
Semantic Segmentation
Predicting a categorical class label for every individual pixel in an image.
Instance Segmentation
Detecting individual object entities and generating a pixel-level binary mask for each specific instance.
Intersection over Union
Evaluation metric measuring the overlap ratio between predicted and ground-truth bounding boxes or masks.
Non-Maximum Suppression
Post-processing algorithm that eliminates duplicate overlapping bounding boxes for the same detected object.
Mean Average Precision
The standard benchmark metric for object detection, averaging Precision-Recall curve areas across IoU thresholds and classes.
U-Net
An encoder-decoder architecture with lateral skip connections connecting contracting stages directly to expansive stages.
Dice Loss
A loss function based on the Dice-SΓΈrensen coefficient measuring spatial overlap between predicted and target segmentation masks.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.