Deep Learning β€Ί Training at Scale β€Ί Day 232

Day 232: GPUs and AI Hardware

Day 232 of 365 β€” GPUs and AI Hardware

Master AI hardware architecture: compare GPUs, TPUs, and CPUs, understand Streaming Multiprocessors, Tensor Cores, High Bandwidth Memory (HBM), compute Arithmetic Intensity via the Roofline Model, diagnose memory-bandwidth vs compute bottlenecks in transformer training and inference, and write PyTorch GPU benchmarking harnesses.

Course
Deep Learning
Category
Training at Scale
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-232-gpus-and-ai-hardware

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-232-gpus-and-ai-hardware
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

The deep learning revolution was not sparked by a new mathematical discovery; the backpropagation algorithm was published in 1986.

What made modern AI possible was Hardware Acceleration:

As an AI practitioner, writing high-performance code requires understanding the physical machine underneath your PyTorch scripts:

Today, you will master GPU microarchitecture, Streaming Multiprocessors, Tensor Cores, High Bandwidth Memory (HBM), and the foundational Roofline Model.


The idea in plain language

Diagram: sequential processing compared with massively parallel arithmetic

Think of moving 1,000,000 gallons of water across a city:


Historical background

Diagram: the historical progression from graphics hardware to AI accelerators

GPU Streaming Multiprocessor and Tensor Core architecture diagram showing SIMT execution warp schedulers shared memory and HBM interconnect

  1. 2006 (NVIDIA & CUDA): Ian Buck and Jensen Huang released CUDA (Compute Unified Device Architecture), transforming GPUs from fixed-function 3D rendering chips into general-purpose parallel computing supercomputers.
  2. 2017 (Volta V100 & Tensor Cores): NVIDIA introduced dedicated mixed-precision matrix multiply hardware (Tensor Cores), delivering a 5\times leap in deep learning training throughput.
  3. 2022 (Hopper H100 & Transformer Engine): Added FP8 Tensor Cores, asynchronous distributed shared memory, and DPX dynamic programming instructions.
  4. 2024 (Blackwell B200): Introduced 208-billion transistor dual-die architectures delivering 20 PFLOPS of FP4 compute for frontier reasoning models.

What it is β€” and what it is not

What AI Hardware Acceleration IS:

What it is NOT:


Why it was created and what problems it solves

Deep learning architectures are fundamentally composed of two mathematical primitives:

  1. General Matrix Multiplications (GEMMs): Dense linear projections (Q = X W_Q, FFN = X W_1), representing 95\%+ of all training FLOPs.
  2. Elementwise Operations: Activations (GELU/ReLU), Normalizations (LayerNorm/RMSNorm), and Residual additions.

GPUs and TPUs were engineered specifically to execute dense GEMMs at theoretical hardware limits.


How it works

Let us dissect the Streaming Multiprocessor architecture, memory tiers, and the Roofline Model.


1. The Streaming Multiprocessor (SM) Engine

Diagram: the hierarchy of execution units inside an accelerator

An NVIDIA H100 GPU contains 132 Streaming Multiprocessors (SMs):


2. The GPU Memory Hierarchy

Diagram: the four-tier accelerator memory hierarchy from device memory to registers

Data moves through four distinct speed and capacity tiers:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                        GPU MEMORY HIERARCHY                            β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Memory Tier        β”‚ Location        β”‚ Typical Capacity β”‚ Bandwidth    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Registers          β”‚ Inside Core     β”‚ ~64 KB per SM    β”‚ ~30 TB/s     β”‚
β”‚ SRAM Shared / L1   β”‚ Inside SM       β”‚ 256 KB per SM    β”‚ ~19 TB/s     β”‚
β”‚ L2 Cache           β”‚ On-Chip Shared  β”‚ 50 MB            β”‚ ~5.4 TB/s    β”‚
β”‚ HBM3e VRAM         β”‚ Off-Chip Interp β”‚ 80 GB - 141 GB   β”‚ ~3.35 TB/s   β”‚
β”‚ Host CPU RAM (PCIe)β”‚ Motherboard     β”‚ 512 GB - 2 TB    β”‚ 64 GB/s (Gen5β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The FlashAttention Insight (Dao et al., 2022):

Standard attention writes the N \times N attention matrix S = Q K^\top / \sqrt{d_k} to slow HBM memory and reads it back for Softmax. FlashAttention tiles Q, K, V into small blocks that fit entirely in 256 KB SRAM Shared Memory, computing Softmax incrementally on-chip, speeding up attention 3\times!


3. The Roofline Model: Memory-Bound vs. Compute-Bound

Diagram: memory-bandwidth-bound operations compared with compute-bound operations

The Roofline Model diagram plotting Arithmetic Intensity vs Attainable Performance showing Memory Bound vs Compute Bound regimes

The Roofline Model defines the fundamental physical limit of computational throughput:

1. Arithmetic Intensity (I):

I = \frac{\text{Total Arithmetic Operations (FLOPs)}}{\text{Total Memory Traffic Transferred across HBM (Bytes)}}

2. Machine Balance (\beta):

\beta = \frac{\text{Peak Compute Performance (TFLOPs/s)}}{\text{Peak Memory Bandwidth (TB/s)}}

3. Operational Regimes:


4. Specialized AI Accelerators: Google TPUs and Apple Neural Engines

Beyond NVIDIA GPUs, two specialized hardware architectures dominate AI:

  1. Google TPUs (Tensor Processing Units):
    • Built around Systolic Arrays: 2D grid of arithmetic processing units where intermediate matrix results flow directly from cell to cell without writing to registers, maximizing energy efficiency.
  2. Apple Neural Engine (ANE):
    • Unified memory architecture (UMA) where CPU, GPU, and NPU share the exact same high-bandwidth LPDDR5 memory pool, enabling zero-copy model execution on MacBooks and iPhones.

5. Model FLOPs Utilization (MFU / Chowdhery et al., 2022)

Diagram: why achieved throughput falls short of peak hardware specifications

In high-performance cluster engineering, peak hardware specifications are never reached 100% of the time:


Diagram: the three tiers of accelerator interconnect by bandwidth

When scaling training across hundreds of GPUs, the inter-GPU communication fabric becomes the ultimate bottleneck:

  1. PCIe Gen5: 64 GB/s bidirectional host-to-device bandwidth (too slow for all-reduce tensor parallelism).
  2. NVLink 4 / 5: High-speed direct GPU-to-GPU interconnect providing 900 GB/s to 1,800 GB/s bandwidth inside an 8-GPU node.
  3. NVSwitch Fabric: A dedicated crossbar routing switch allowing all 8 GPUs in a node to communicate simultaneously at full bi-directional NVLink wire speed.
  4. InfiniBand / RoCE (RDMA over Converged Ethernet): 400 Gbps to 800 Gbps per node inter-chassis networking using Remote Direct Memory Access (bypassing the host CPU entirely).

7. The CUDA Thread Hierarchy & Warp Divergence

How are parallel operations dispatched to the physical hardware?


8. OpenAI Triton: Python-Based GPU Kernel Programming (Tillet et al., 2019)

Diagram: low-level kernel programming compared with a high-level kernel language

Writing bare-metal CUDA C++ requires manually orchestrating register allocation, thread synchronization barriers (__syncthreads()), and shared memory bank conflicts:


An everyday analogy

Diagram: a single skilled worker compared with many simple workers in parallel

Think of an industrial commercial bakery:


Examples in practice

Let us build a GPU Hardware Benchmark and Arithmetic Intensity Profiler in PyTorch:

import torch
import time

def benchmark_matrix_multiplication(size: int = 4096, dtype=torch.float16,
                                    num_repeats: int = 50) -> dict:
    # Benchmarks Matrix Multiplication throughput (TFLOPs/s) and Arithmetic Intensity.
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    
    # Allocate square matrices A and B: (size x size)
    A = torch.randn(size, size, device=device, dtype=dtype)
    B = torch.randn(size, size, device=device, dtype=dtype)

    # Warmup GPU
    for _ in range(10):
        _ = torch.matmul(A, B)
    if device.type == "cuda":
        torch.cuda.synchronize()

    # Time execution
    start_time = time.perf_counter()
    for _ in range(num_repeats):
        C = torch.matmul(A, B)
    if device.type == "cuda":
        torch.cuda.synchronize()
    elapsed_time = (time.perf_counter() - start_time) / num_repeats

    # Compute FLOPs: 2 * N^3 for GEMM
    total_flops = 2 * (size ** 3)
    tflops_per_sec = (total_flops / elapsed_time) / 1e12

    # Memory Traffic: 3 matrices of size * size * bytes_per_elem
    elem_bytes = 2 if dtype in (torch.float16, torch.bfloat16) else 4
    total_bytes = 3 * (size ** 2) * elem_bytes
    arithmetic_intensity = total_flops / total_bytes

    return {
        "matrix_size": size,
        "dtype": str(dtype),
        "latency_ms": round(elapsed_time * 1000, 3),
        "tflops_per_sec": round(tflops_per_sec, 2),
        "arithmetic_intensity": round(arithmetic_intensity, 2)
    }

Implications: security, privacy, performance, scalability, and cost

Diagram: the cost, supply, efficiency and lock-in implications of AI hardware

  1. GPU Power & Thermal Throttling:
    • Modern GPUs draw up to 700 Watts under full Tensor Core load. If datacenter cooling is inadequate, GPUs throttle clock speeds by 30\%+, silently degrading distributed training throughput.
  2. CUDA Side-Channel & Memory Leakage:
    • Multi-tenant GPU sharing without virtualization (MIG / vGPU) can leak sensitive tensor memory traces across processes. Use NVIDIA Multi-Instance GPU (MIG) for strict physical memory isolation.

Alternatives: free, open source, and commercial

Hardware PlatformArchitecturePrecision TypesPeak ThroughputTarget Use
NVIDIA H100 SXMHopper SM + Tensor CoreFP8 / FP16 / BF16 / FP321,000 TFLOPs (FP16)Premier LLM Pretraining
Google TPU v5pSystolic Array 3D MeshBF16 / INT8 / FP32459 TFLOPs (BF16)Large-Scale Pod Training
Apple M3/M4 MaxUnified Memory GPU/ANEFP16 / BF16 / INT8100+ GB Unified RAMLocal Edge Prototyping
AMD Instinct MI300XCDNA 3 + 192GB HBM3FP8 / FP16 / BF161,300 TFLOPs (FP16)High-VRAM LLM Serving

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   AI HARDWARE ARCHITECTURE COMPARISON                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Architecture       β”‚ Execution Model β”‚ Memory Strategy  β”‚ Best Workloadβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ CPU (x86 / ARM)    β”‚ Latency-Focused β”‚ Massive L3 Cache β”‚ Sequential   β”‚
β”‚ GPU (NVIDIA / AMD) β”‚ SIMT Throughput β”‚ High Bandwidth   β”‚ GEMM / Train β”‚
β”‚ TPU (Google)       β”‚ Systolic Array  β”‚ 3D Torus Interconβ”‚ Dense Batch  β”‚
β”‚ NPU / ANE (Apple)  β”‚ Fixed-Function  β”‚ Unified Memory   β”‚ Edge On-Dev  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when accelerated hardware is necessary and when a processor suffices

When to Optimize for GPU Hardware Architecture:

When NOT to use GPUs:


The AI connection

Knowledge check

  1. What is the fundamental difference between CUDA SIMT cores and Tensor Cores?
  2. What is Arithmetic Intensity and how does it determine whether a kernel is memory-bound or compute-bound?
  3. Why does single-token autoregressive generation have low arithmetic intensity?
  4. How does SRAM Shared Memory enable kernel fusion in FlashAttention?
  5. What is the difference between a GPU SIMT engine and a TPU Systolic Array?

Hands-on exercise

In this lab, you will build and test a modular hardware benchmarking suite in PyTorch: implement matrix multiplication FLOP calculations, measure execution latency across variable matrix sizes, compute Arithmetic Intensity across FP32 and FP16 data types, and determine the machine balance ridge point for hardware acceleration.

Expected output

[AI Hardware Performance Verification Suite]
Testing Matrix Multiply Benchmark (Size=512):
  Precision: torch.float32 | FLOPs: 268.4M | Memory: 3.15 MB
  Arithmetic Intensity: 85.33 FLOPs/Byte [MEASURED]
Testing Half-Precision Benchmark (Size=512, FP16):
  Precision: torch.float16 | FLOPs: 268.4M | Memory: 1.57 MB
  Arithmetic Intensity: 170.67 FLOPs/Byte (2x Data Density) [VERIFIED]
Testing Machine Balance Calculator:
  Calculated Machine Balance: 125.0 FLOPs/Byte (Threshold for Compute Bound)
Test Suite: 3 passed in 0.41s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a Roofline Model Plotter in Python using Matplotlib that plots the theoretical performance envelope for your local hardware.
  2. Benchmark an Elementwise LayerNorm vs a Dense Matrix Multiplication and prove that LayerNorm is memory-bandwidth bound.

Extension challenge

Implement a Simple Triton Fused GELU Kernel:

Quiz

Q1. What is the fundamental architectural difference between a CPU and a modern AI GPU (e.g. NVIDIA H100)?

  1. A CPU uses electricity while a GPU uses optical light
  2. A CPU optimises latency; a GPU optimises throughput
  3. A GPU cannot perform addition
  4. A CPU only runs Python
Show answer

Answer: B. A CPU optimises latency; a GPU optimises throughput

CPUs optimize for sequential latency and branch prediction; GPUs optimize for massive parallel arithmetic throughput.

Q2. What is Arithmetic Intensity (I) in the Roofline Model?

  1. The number of layers in the neural network
  2. The clock speed of the processor in Gigahertz
  3. FLOPs performed per byte moved from memory
  4. The temperature of the cooling liquid
Show answer

Answer: C. FLOPs performed per byte moved from memory

Arithmetic intensity measures how much computation is performed per byte of memory bandwidth consumed.

Q3. Why is single-token autoregressive generation in LLMs almost always Memory-Bandwidth Bound rather than Compute Bound?

  1. Because Python executes single threads
  2. Because GPUs disable Tensor Cores during generation
  3. Because tokenizers use too much disk space
  4. Every token reloads all the weights from memory
Show answer

Answer: D. Every token reloads all the weights from memory

Batch-1 generation reads massive weight matrices from memory for minimal compute, saturating memory bandwidth while leaving ALUs mostly idle.

Q4. How do Tensor Cores achieve an order-of-magnitude speedup over standard CUDA cores during deep learning training?

  1. Fused matrix multiply-accumulate per cycle
  2. They skip multiplying zeros
  3. They reduce the number of tokens in the prompt
  4. They run at 100 GHz clock speed
Show answer

Answer: A. Fused matrix multiply-accumulate per cycle

Tensor Cores compute entire matrix multiply-accumulate tiles in single hardware clock cycles using mixed precision.

Q5. What role does SRAM Shared Memory / L1 Cache play in modern GPU optimizations like FlashAttention?

  1. It stores operating system kernel logs
  2. Fast on-chip memory inside each SM
  3. It replaces the hard drive
  4. It encrypts network packets
Show answer

Answer: B. Fast on-chip memory inside each SM

SRAM on-chip shared memory enables kernel fusion and block tiling, bypassing the HBM memory bandwidth bottleneck.

Glossary

Streaming Multiprocessor (SM)
The fundamental compute building block of an NVIDIA GPU containing warp schedulers, CUDA cores, Tensor Cores, and SRAM shared memory.
Tensor Core
A specialized hardware execution unit executing fused mixed-precision matrix multiply-accumulate operations (D = A * B + C) in a single clock cycle.
Arithmetic Intensity
The ratio of arithmetic operations (FLOPs) to memory traffic (Bytes) transferred across High Bandwidth Memory: I = FLOPs / Byte.
Roofline Model
A visual performance model that plots attainable computational throughput against arithmetic intensity, bounding performance by memory bandwidth and peak compute ceilings.
High Bandwidth Memory (HBM3e)
3D-stacked DRAM on the GPU interposer providing multi-terabyte-per-second memory bandwidth.
SRAM Shared Memory
Ultra-low-latency on-chip memory directly integrated into each SM, delivering ~19 TB/s aggregate bandwidth.
Memory-Bound Operation
An algorithm whose execution speed is limited by memory bandwidth transfer rates rather than processor arithmetic capacity.
Compute-Bound Operation
An algorithm whose execution speed is limited by raw processor arithmetic throughput rather than memory data transfer.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.