Deep Learning β€Ί Convolutional Networks and Vision β€Ί Day 217

Day 217: Your Own Image Classifier

Day 217 of 365 β€” Your Own Image Classifier

Build, audit, and deploy your own production image classifier: collect and deduplicate raw visual data, fine-tune a modern pretrained backbone with data augmentation, construct an interactive Error-Case Gallery, inspect class confusion, and export production ONNX runtime artifacts.

Course
Deep Learning
Category
Convolutional Networks and Vision
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-217-your-own-image-classifier

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-217-your-own-image-classifier
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Over the past six days, we mastered the core theoretical and algorithmic building blocks of deep computer vision:

Now, you will synthesize all of these skills into a unified, professional computer vision engineering workflow: building Your Own Custom Image Classifier on a bespoke real-world dataset.

In industrial AI engineering, the primary differentiator between amateur and production-grade systems is not the model architecture β€” it is the Data-Centric AI Lifecycle:

  1. How you clean, curate, and deduplicate your raw visual data.
  2. How you structure two-stage transfer learning fine-tuning.
  3. How you systematically diagnose model failures using an Interactive Error-Case Gallery.
  4. How you export hardened, portable ONNX Runtime deployment artifacts.

Today, you will build, audit, and deploy a complete custom vision system from end to end.


The idea in plain language

Diagram: the five phases of building a custom image classifier

Imagine launching a specialized automated visual sorting system for an organic botanical tea company:


Historical background

End-to-end workflow lifecycle of a custom image classifier from data ingestion and augmentation to transfer fine-tuning error-case auditing and deployment

  1. 2015 (Standardization of ImageFolder): Soumith Chintala and the PyTorch team standardized the directory-based dataset hierarchy (dataset/train/class_name/xxx.jpg), enabling instant zero-configuration data ingestion.
  2. 2021 (The Data-Centric AI Movement): Andrew Ng and deep learning researchers established that fixing label noise, class imbalance, and duplicate samples in datasets yields $5\times$ greater accuracy gains than hyperparameter tuning or architecture swapping.
  3. 2024 (ONNX Runtime and OpenVINO Maturation): Linux Foundation and Microsoft standardized cross-platform neural model serialization, enabling PyTorch vision models to run seamlessly on CPU, GPU, and mobile neural processing units (NPUs).

What it is β€” and what it is not

What Building a Custom Image Classifier IS:

What it is NOT:


Why it was created and what problems it solves

A structured custom vision workflow eliminates four major real-world failure modes:

  1. Train/Val Leakage from Near-Duplicates: Deduplication with perceptual hashing (pHash) ensures identical or near-identical images do not appear in both train and validation splits.
  2. Silent Label Noise: Error galleries surface ground-truth labeling mistakes made by human annotators.
  3. Overfitting on Small Target Datasets: Combining frozen backbone initialization with heavy stochastic augmentation prevents memorization.
  4. Deployment Incompatibility: ONNX export standardizes model serving across cloud APIs and edge devices.

How it works

Let us dissect the five distinct phases of building a custom image classifier.


1. Phase 1: Data Curation & Perceptual Hashing (pHash)

Diagram: how perceptual hashing detects near-duplicate images

When collecting images from web scraping, user uploads, or factory cameras, datasets frequently contain duplicate or near-duplicate images (e.g. same photo resized or with a watermark). If duplicates exist across train and validation splits, validation metrics become artificially inflated.

Perceptual Hashing (pHash) Algorithm:

  1. Resize image to $32 \times 32$ grayscale.
  2. Compute the 2D Discrete Cosine Transform (DCT) to isolate low-frequency spatial structure.
  3. Retain the top-left $8 \times 8$ low-frequency DCT matrix.
  4. Compute the median value of the $8 \times 8$ matrix.
  5. Generate a 64-bit binary hash where bit $b_i = 1$ if $DCT_i > median$, else $0$.
  6. Calculate Hamming Distance between image hashes: images with distance $\le 3$ are flagged as near-duplicates and removed.

2. Phase 2: Transfer Learning Fine-Tuning Regimen

  1. Load a pretrained backbone: timm.create_model("resnet34", pretrained=True, num_classes=num_classes).
  2. Apply torchvision.transforms.v2 stochastic augmentation on the training split (RandomResizedCrop, HorizontalFlip, ColorJitter, RandomErasing).
  3. Execute Phase 1 Warmup (Head only, $LR = 10^-3$, 3 epochs).
  4. Execute Phase 2 Differential Fine-Tuning (Backbone $LR = 10^-5$, Head $LR = 10^-4$, Cosine Annealing, 10 epochs).

Diagram: the three causes of high-confidence classification errors

Error case gallery diagnostic workflow surfacing high confidence misclassifications ambiguous samples and dataset label noise

An Error-Case Gallery extracts and visualizes the validation samples with the highest Prediction Error Confidence:

Loss_i = - log(p_{i, y_true})
Confidence_i = p_{i, y_pred}  (where y_pred != y_true)
  1. Label Noise (Human Error): The image is actually class A, but the dataset labeled it class B. (Fix: Correct the label).
  2. Optical Occlusion / Glare: The object is 90% hidden behind a shadow or lens glare. (Fix: Add lighting/cutout augmentation).
  3. Taxonomic Ambiguity: Two classes have overlapping subjective definitions (e.g. β€œMild Scratch” vs β€œSevere Scratch”). (Fix: Clarify annotation guidelines or merge classes).

4. Phase 4: Production Export to ONNX Runtime

  1. Export PyTorch checkpoint to ONNX format with dynamic batch dimensions.
  2. Load ONNX model in Python via onnxruntime.InferenceSession.
  3. Verify that maximum absolute logit difference between PyTorch and ONNX Runtime is $< 10^-5$.
import onnxruntime as ort
import numpy as np

# Verify ONNX Runtime inference
session = ort.InferenceSession("classifier.onnx")
input_name = session.get_inputs()[0].name

dummy_input_np = np.random.randn(1, 3, 224, 224).astype(np.float32)
ort_outputs = session.run(None, {input_name: dummy_input_np})

print(f"ONNX Runtime Output Shape: {ort_outputs[0].shape}")

5. Active Learning & Uncertainty Sampling

Diagram: uncertainty sampling compared with random sampling for labelling

When expanding a custom image classifier over time, manually labeling thousands of random raw images is cost-prohibitive:


6. Automated Label Noise Detection (Confident Learning)

Diagram: how confident learning identifies probable label errors

Human annotators disagree on 3% to 10% of image labels in real-world visual datasets:


7. Visual Explainability with Grad-CAM

Diagram: a model attending to the object compared with one attending to the background

When debugging error gallery samples, we need to know where the model looked:


8. Production FastAPI Serving Architecture

Deploying your custom image classifier into a production microservice:

from fastapi import FastAPI, File, UploadFile
import onnxruntime as ort
import numpy as np
from PIL import Image
import io

app = FastAPI(title="Botanical Classifier Service")
session = ort.InferenceSession("model.onnx")

@app.post("/predict")
async def predict_image(file: UploadFile = File(...)):
    contents = await file.read()
    img = Image.open(io.BytesIO(contents)).convert("RGB").resize((224, 224))
    img_np = (np.array(img, dtype=np.float32) / 255.0 - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]
    tensor_input = np.transpose(img_np, (2, 0, 1))[np.newaxis, ...]

    logits = session.run(None, {"input": tensor_input})[0]
    pred_class = int(np.argmax(logits, axis=1)[0])
    return {"class_id": pred_class}

9. Production Vision Observability & Data Drift (PSI)

Diagram: data drift compared with concept drift in a deployed vision model

Once deployed, computer vision models degrade over time due to Data Drift (changes in camera hardware, lighting, lens degradation) and Concept Drift (new physical variations of the product):


An everyday analogy

Diagram: a sorter calibrated on catalogue images compared with one calibrated on real product

Think of custom-building a high-precision digital coffee bean sorting machine:


Examples in practice

Let us build an Error-Case Gallery Generator in PyTorch:

import torch
import torch.nn as nn
from typing import List, Dict, Any

def generate_error_case_gallery(model: nn.Module, dataloader, class_names: List[str],
                                top_n: int = 5) -> List[Dict[str, Any]]:
    model.eval()
    failures = []

    with torch.no_grad():
        for batch_idx, (images, targets) in enumerate(dataloader):
            logits = model(images)
            probs = torch.softmax(logits, dim=1)
            confidences, preds = torch.max(probs, dim=1)

            # Identify misclassifications
            incorrect = preds != targets

            for i in range(images.size(0)):
                if incorrect[i]:
                    true_idx = targets[i].item()
                    pred_idx = preds[i].item()
                    conf = confidences[i].item()

                    failures.append({
                        "batch_idx": batch_idx,
                        "sample_idx": i,
                        "true_class": class_names[true_idx],
                        "pred_class": class_names[pred_idx],
                        "confidence": conf,
                        "true_prob": probs[i, true_idx].item()
                    })

    # Sort failures by prediction confidence in descending order (highest confidence errors first)
    failures.sort(key=lambda x: x["confidence"], reverse=True)
    return failures[:top_n]

Implications: security, privacy, performance, scalability, and cost

  1. Adversarial Patch Vulnerabilities:
    • In production camera feeds, small sticker patches placed on physical objects can cause targeted misclassifications. Test model robustness against localized noise perturbations.
  2. Edge Hardware Acceleration (INT8 Quantization):
    • Quantizing exported ONNX models to INT8 using ONNX Runtime Quantization tools reduces model file size by 75% and accelerates edge CPU inference by 3.5x.

Alternatives: free, open source, and commercial

Tool / FrameworkPurposeEcosystemBest For
PyTorch + ONNX RuntimeEnd-to-end custom classifierOpen SourceProduction engineering
CleanlabAutomated label noise detectionOpen SourceAuditing training data labels
RoboflowImage annotation & dataset curationCommercial / SaaSTeam dataset labeling
Triton Inference ServerEnterprise model servingOpen Source (NVIDIA)Multi-model cloud clusters

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   VISION PROJECT DIAGNOSTIC MATRIX                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Diagnostic Tool    β”‚ Purpose               β”‚ Actionable Remediation    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Confusion Matrix   β”‚ Aggregate pair errors β”‚ Adjust class definitions  β”‚
β”‚ Error-Case Gallery β”‚ Inspect worst samples β”‚ Fix label noise & glare   β”‚
β”‚ Learning Rate Find β”‚ Optimizer tuning      β”‚ Set optimal base LR       β”‚
β”‚ ONNX Verifier      β”‚ Deployment fidelity   β”‚ Fix dynamic operator bugs β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: building a custom classifier compared with prompting a foundation model

When to USE this Custom Classifier Workflow:

When NOT to use:


The AI connection

Knowledge check

  1. What is the purpose of an Error-Case Gallery?
  2. How does Perceptual Hashing (pHash) prevent data leakage between train and validation splits?
  3. What are the three primary causes of high-confidence validation errors in computer vision?
  4. How do you verify that an exported ONNX model produces matching numerical outputs to PyTorch?
  5. Why is two-stage transfer learning (frozen warmup followed by differential fine-tuning) superior to training from scratch?

Hands-on exercise

In this lab, you will build and test an end-to-end custom image classifier toolkit in PyTorch: implement a Perceptual Hash (pHash) near-duplicate detection function, construct an automated Error-Case Gallery generator surfacing highest-confidence misclassifications, compute class confusion metrics, and verify ONNX model export serialization.

Expected output

[Custom Image Classifier Toolkit Verification Suite]
Testing Perceptual Hash (pHash) Image Deduplication:
  Identical Images (Hamming Distance): 0 [EXACT MATCH]
  Modified Watermark Image: Hamming Dist = 2 [NEAR DUPLICATE DETECTED]
  Different Class Image:    Hamming Dist = 38 [DISTINCT IMAGE CONFIRMED]
Testing Error-Case Gallery Generator:
  Processing Validation Predictions (Batch: 32 samples):
  Surfacing Top 3 High-Confidence Failures:
    1. Sample #14: True='Chamomile' -> Pred='Daisy' (Conf=98.4%) [LABEL AUDIT FLAGGED]
    2. Sample #03: True='Peppermint' -> Pred='Spearmint' (Conf=91.2%) [TAXONOMY OVERLAP]
Testing ONNX Export and Numerical Alignment:
  PyTorch Logits vs ONNX Runtime Outputs:
  Max Absolute Difference: 2.3842e-07 [EXPORT VERIFIED]
Test Suite: 4 passed in 0.33s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a Grad-CAM (Gradient-Weighted Class Activation Mapping) visualization script that overlays a heat map on error gallery images to show what visual regions the model focused on.
  2. Build an automated Dataset Health Report summarizing class distribution, average resolution, duplicate count, and label consistency scores.

Extension challenge

Implement INT8 Dynamic Quantization on Exported ONNX Models:

Quiz

Q1. What is an Error-Case Gallery in computer vision engineering, and why is it superior to relying solely on aggregate validation accuracy?

  1. A gallery of images where the model achieved 100% accuracy
  2. A museum of deleted Python scripts
  3. A list of GPU hardware crashes
  4. A review of the most confident misclassifications
Show answer

Answer: D. A review of the most confident misclassifications

Surfacing the most confident false positives and false negatives exposes ground truth annotation errors, optical artifacts, and taxonomic overlap.

Q2. How does Perceptual Hashing (pHash) detect duplicate or near-duplicate images during custom dataset curation?

  1. A DCT of low-frequency luminance, as a 64-bit hash
  2. It computes the MD5 checksum of the raw file bytes
  3. It counts the total number of pixels
  4. It checks the file name string
Show answer

Answer: A. A DCT of low-frequency luminance, as a 64-bit hash

Unlike cryptographic hashes (MD5/SHA256) where changing 1 pixel changes the entire hash, perceptual hashes (pHash) produce small Hamming distances for visually similar images.

Q3. When fine-tuning a custom vision model, what does it mean if the model achieves 99% training accuracy but only 68% validation accuracy despite using transfer learning?

  1. The learning rate is too low
  2. Overfitting β€” augment more, or freeze more layers
  3. The model needs 1000 more layers
  4. The batch size is too large
Show answer

Answer: B. Overfitting β€” augment more, or freeze more layers

A large gap between training and validation accuracy is the definition of overfitting, remedied by regularization, augmentation, and parameter freezing.

Q4. Why should you verify ONNX model exports with ONNX Runtime using sample inputs after exporting from PyTorch?

  1. To re-train the model weights in C++
  2. To compress the image files on disk
  3. To confirm the export matches PyTorch numerically
  4. To upload the model to social media
Show answer

Answer: C. To confirm the export matches PyTorch numerically

Verifying ONNX Runtime numerical outputs against PyTorch ensures operator export fidelity and eliminates runtime crashes in production inference engines.

Q5. What is the recommended directory structure for custom multi-class image datasets processed with PyTorch torchvision.datasets.ImageFolder?

  1. Images stored inside a SQLite binary database column
  2. All images dumped into a single root directory with text file mappings
  3. A single giant ZIP archive containing JSON metadata
  4. root/split/class/image.jpg
Show answer

Answer: D. root/split/class/image.jpg

ImageFolder standard expects subdirectories named after each class under separate train/val/test splits.

Glossary

Error-Case Gallery
A visual audit report displaying the top most confident model misclassifications to diagnose failure modes.
Perceptual Hash
A visual fingerprint algorithm (pHash) that generates matching or near-matching hashes for visually identical images.
Data-Centric AI
The engineering paradigm of systematically engineering and cleaning dataset quality rather than solely tweaking model code.
Label Noise
Incorrect, ambiguous, or inconsistent ground-truth annotations present within training or validation datasets.
ONNX Runtime
A high-performance cross-platform inference engine for executing serialized Open Neural Network Exchange models.
Stratified Split
Partitioning a dataset into train/val/test sets such that each split preserves identical class proportion percentages.
Dynamic Axes
Configuring exported ONNX tensors to accept variable batch sizes or variable spatial image dimensions at runtime.
Class Imbalance
A dataset condition where certain categories contain orders of magnitude more samples than minority categories.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.