Deep Learning βΊ Training at Scale βΊ Day 232
Day 232: GPUs and AI Hardware
Master AI hardware architecture: compare GPUs, TPUs, and CPUs, understand Streaming Multiprocessors, Tensor Cores, High Bandwidth Memory (HBM), compute Arithmetic Intensity via the Roofline Model, diagnose memory-bandwidth vs compute bottlenecks in transformer training and inference, and write PyTorch GPU benchmarking harnesses.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-232-gpus-and-ai-hardware
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/deep-learning/day-232-gpus-and-ai-hardware - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Compare CPU latency-oriented architecture with GPU high-throughput SIMT (Single Instruction Multiple Threads) parallelism.
- Analyze the GPU memory hierarchy: Registers, SRAM Shared Memory, L2 Cache, and HBM3e High Bandwidth Memory.
- Explain how Tensor Cores execute fused matrix multiply-accumulate operations (D = A x B + C) in hardware.
- Apply the Roofline Model to calculate Arithmetic Intensity (FLOPs/Byte) and identify memory-bound vs compute-bound operations.
- Benchmark tensor operation throughput (TFLOPs/s and GB/s) in PyTorch across different precision modes.
Prerequisites
Why this matters
The deep learning revolution was not sparked by a new mathematical discovery; the backpropagation algorithm was published in 1986.
What made modern AI possible was Hardware Acceleration:
- In 2012, Alex Krizhevsky trained AlexNet on two gaming GPUs (NVIDIA GTX 580s with 3 GB VRAM), cutting training time from months on CPUs to 6 days.
- In 2024, training a frontier LLM requires clusters of 100,000 NVIDIA H100/B200 GPUs, consuming 100 Megawatts of power and executing
10^{25}floating-point operations.
As an AI practitioner, writing high-performance code requires understanding the physical machine underneath your PyTorch scripts:
- Why does doubling your batch size sometimes make your code run
4\timesfaster? - Why is single-token LLM generation bottlenecked by memory bandwidth rather than compute?
- How does FlashAttention achieve a
3\timesspeedup without changing a single mathematical output?
Today, you will master GPU microarchitecture, Streaming Multiprocessors, Tensor Cores, High Bandwidth Memory (HBM), and the foundational Roofline Model.
The idea in plain language
Think of moving 1,000,000 gallons of water across a city:
- CPU (The Ferrari Delivery Courier): A Ferrari can drive 200 mph with zero delay (Ultra-Low Latency). But it can only carry 2 jugs of water in its trunk (16-64 Cores). Excellent for rushing an urgent legal document across town, but useless for draining a lake.
- GPU (The Freight Train Pipeline): A massive freight train with 10,000 tanker cars moves at a modest 40 mph (High Latency per Thread). But in a single trip, it moves all 1,000,000 gallons simultaneously (Massive Parallel SIMT Throughput).
- The Bottleneck (The Hose vs The Pump): If your water pump can process 1,000 gallons/second (Tensor Core Compute), but your pipe can only supply 50 gallons/second (Memory Bandwidth), your expensive pump sits idle
95\%of the time (Memory-Bound Execution).
Historical background
- 2006 (NVIDIA & CUDA): Ian Buck and Jensen Huang released CUDA (Compute Unified Device Architecture), transforming GPUs from fixed-function 3D rendering chips into general-purpose parallel computing supercomputers.
- 2017 (Volta V100 & Tensor Cores): NVIDIA introduced dedicated mixed-precision matrix multiply hardware (Tensor Cores), delivering a
5\timesleap in deep learning training throughput. - 2022 (Hopper H100 & Transformer Engine): Added FP8 Tensor Cores, asynchronous distributed shared memory, and DPX dynamic programming instructions.
- 2024 (Blackwell B200): Introduced 208-billion transistor dual-die architectures delivering 20 PFLOPS of FP4 compute for frontier reasoning models.
What it is β and what it is not
What AI Hardware Acceleration IS:
- Massive Parallelism via SIMT (Single Instruction, Multiple Threads): Executing identical mathematical operations across thousands of independent data points simultaneously.
- A Deep Memory Hierarchy: Moving data through a high-speed hierarchy from HBM VRAM
\rightarrowL2 Cache\rightarrowSRAM Shared Memory\rightarrowRegisters.
What it is NOT:
- Not Magic Faster Clocks: GPU clock frequencies (1.5β2.0 GHz) are actually slower than high-end CPUs (4.5β5.5 GHz). The speed comes entirely from massive concurrency and specialized matrix ALUs.
Why it was created and what problems it solves
Deep learning architectures are fundamentally composed of two mathematical primitives:
- General Matrix Multiplications (GEMMs): Dense linear projections (
Q = X W_Q,FFN = X W_1), representing95\%+of all training FLOPs. - Elementwise Operations: Activations (GELU/ReLU), Normalizations (LayerNorm/RMSNorm), and Residual additions.
GPUs and TPUs were engineered specifically to execute dense GEMMs at theoretical hardware limits.
How it works
Let us dissect the Streaming Multiprocessor architecture, memory tiers, and the Roofline Model.
1. The Streaming Multiprocessor (SM) Engine
An NVIDIA H100 GPU contains 132 Streaming Multiprocessors (SMs):
- Warp Schedulers: Group threads into Warps of 32 threads. All 32 threads in a warp execute the exact same instruction at the same cycle.
- CUDA Cores: Standard 32-bit ALUs for scalar arithmetic, indexing, and non-linear activations.
- Tensor Cores (4th Generation): Specialized matrix engines that execute:
whereD = A \times B + CA, Bare16 \times 16mixed-precision matrices (FP16/BF16/FP8) andC, Dare FP32 accumulation matrices, completing an entire16 \times 16 \times 16matrix multiply-accumulate in a single clock cycle!
2. The GPU Memory Hierarchy
Data moves through four distinct speed and capacity tiers:
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β GPU MEMORY HIERARCHY β
ββββββββββββββββββββββ¬ββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββ€
β Memory Tier β Location β Typical Capacity β Bandwidth β
ββββββββββββββββββββββΌββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββ€
β Registers β Inside Core β ~64 KB per SM β ~30 TB/s β
β SRAM Shared / L1 β Inside SM β 256 KB per SM β ~19 TB/s β
β L2 Cache β On-Chip Shared β 50 MB β ~5.4 TB/s β
β HBM3e VRAM β Off-Chip Interp β 80 GB - 141 GB β ~3.35 TB/s β
β Host CPU RAM (PCIe)β Motherboard β 512 GB - 2 TB β 64 GB/s (Gen5β
ββββββββββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββ
The FlashAttention Insight (Dao et al., 2022):
Standard attention writes the N \times N attention matrix S = Q K^\top / \sqrt{d_k} to slow HBM memory and reads it back for Softmax. FlashAttention tiles Q, K, V into small blocks that fit entirely in 256 KB SRAM Shared Memory, computing Softmax incrementally on-chip, speeding up attention 3\times!
3. The Roofline Model: Memory-Bound vs. Compute-Bound
The Roofline Model defines the fundamental physical limit of computational throughput:
1. Arithmetic Intensity (I):
I = \frac{\text{Total Arithmetic Operations (FLOPs)}}{\text{Total Memory Traffic Transferred across HBM (Bytes)}}
2. Machine Balance (\beta):
\beta = \frac{\text{Peak Compute Performance (TFLOPs/s)}}{\text{Peak Memory Bandwidth (TB/s)}}
- For an NVIDIA H100 (FP16 Tensor Core: 1,000 TFLOPs/s, HBM3: 3.35 TB/s):
\beta = \frac{1000 \times 10^{12}}{3.35 \times 10^{12}} \approx 298.5 \text{ FLOPs / Byte}
3. Operational Regimes:
-
Memory-Bandwidth Bound (
I < \beta):\text{Attainable Performance} = I \times \text{Peak Memory Bandwidth}The processor is starved for data. Examples: LayerNorm, Dropout, Softmax, Batch-1 Autoregressive Generation.
-
Compute-Bound (
I \ge \beta):\text{Attainable Performance} = \text{Peak Compute TFLOPs}The memory bus keeps up, and Tensor Cores run at maximum speed. Examples: Large-batch matrix multiplications (
B \ge 32), deep CNN convolutions.
4. Specialized AI Accelerators: Google TPUs and Apple Neural Engines
Beyond NVIDIA GPUs, two specialized hardware architectures dominate AI:
- Google TPUs (Tensor Processing Units):
- Built around Systolic Arrays: 2D grid of arithmetic processing units where intermediate matrix results flow directly from cell to cell without writing to registers, maximizing energy efficiency.
- Apple Neural Engine (ANE):
- Unified memory architecture (UMA) where CPU, GPU, and NPU share the exact same high-bandwidth LPDDR5 memory pool, enabling zero-copy model execution on MacBooks and iPhones.
5. Model FLOPs Utilization (MFU / Chowdhery et al., 2022)
In high-performance cluster engineering, peak hardware specifications are never reached 100% of the time:
-
Model FLOPs Utilization (MFU): The ratio of theoretical floating-point operations required for the forward/backward pass divided by the hardwareβs theoretical peak capacity during the elapsed wall-clock training time:
\text{MFU} = \frac{\text{Theoretical FLOPs per Step} / \text{Elapsed Time}}{\text{Number of GPUs} \times \text{Peak GPU TFLOPs/s}} -
State-of-the-Art Benchmarks: Naive PyTorch training achieves
20\%to30\%MFU. Megatron-LM with FlashAttention and tensor parallelism achieves50\%to60\%MFU (world-class efficiency on H100 clusters).
6. Interconnect Topologies: NVLink, NVSwitch, and InfiniBand
When scaling training across hundreds of GPUs, the inter-GPU communication fabric becomes the ultimate bottleneck:
- PCIe Gen5:
64GB/s bidirectional host-to-device bandwidth (too slow for all-reduce tensor parallelism). - NVLink 4 / 5: High-speed direct GPU-to-GPU interconnect providing
900GB/s to1,800GB/s bandwidth inside an 8-GPU node. - NVSwitch Fabric: A dedicated crossbar routing switch allowing all 8 GPUs in a node to communicate simultaneously at full bi-directional NVLink wire speed.
- InfiniBand / RoCE (RDMA over Converged Ethernet):
400Gbps to800Gbps per node inter-chassis networking using Remote Direct Memory Access (bypassing the host CPU entirely).
7. The CUDA Thread Hierarchy & Warp Divergence
How are parallel operations dispatched to the physical hardware?
- Grid
\rightarrowThread Blocks\rightarrowWarps\rightarrowThreads:- A kernel is launched as a 3D Grid of Thread Blocks.
- Each Thread Block is assigned to a single Streaming Multiprocessor (SM).
- The SM divides the block into Warps of 32 threads.
- Warp Divergence: If threads within the same warp take different branches of an
if-elsecondition, the SM must serialize both execution paths, cutting arithmetic throughput in half!
8. OpenAI Triton: Python-Based GPU Kernel Programming (Tillet et al., 2019)
Writing bare-metal CUDA C++ requires manually orchestrating register allocation, thread synchronization barriers (__syncthreads()), and shared memory bank conflicts:
- OpenAI Triton: A Python-based programming language and optimizing compiler for writing custom GPU primitives.
- Operates at the Block level rather than the single-thread level, automatically handling memory coalescing, block tiling, and Tensor Core scheduling with performance matching hand-tuned CUDA C++!
- Hardware Profiling: Tools like NVIDIA Nsight Systems and the PyTorch Profiler reveal SM active occupancy, kernel execution timelines, and memory copy overheads.
An everyday analogy
Think of an industrial commercial bakery:
- Tensor Cores (The High-Speed Oven): An automated oven that can bake 1,000 loaves of bread in 10 minutes (Peak Compute Capacity).
- HBM VRAM (The Loading Dock): The delivery truck delivering raw flour and yeast.
- The Factory Bottleneck:
- If the delivery truck arrives with only 1 bag of flour per hour (Low Arithmetic Intensity), the massive oven sits cold and empty (
98\%idle). - If you load 100 bags of flour into the on-site kitchen pantry (SRAM Shared Memory Tiling), the oven runs continuously at full capacity (Compute-Bound Efficiency).
- If the delivery truck arrives with only 1 bag of flour per hour (Low Arithmetic Intensity), the massive oven sits cold and empty (
Examples in practice
Let us build a GPU Hardware Benchmark and Arithmetic Intensity Profiler in PyTorch:
import torch
import time
def benchmark_matrix_multiplication(size: int = 4096, dtype=torch.float16,
num_repeats: int = 50) -> dict:
# Benchmarks Matrix Multiplication throughput (TFLOPs/s) and Arithmetic Intensity.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Allocate square matrices A and B: (size x size)
A = torch.randn(size, size, device=device, dtype=dtype)
B = torch.randn(size, size, device=device, dtype=dtype)
# Warmup GPU
for _ in range(10):
_ = torch.matmul(A, B)
if device.type == "cuda":
torch.cuda.synchronize()
# Time execution
start_time = time.perf_counter()
for _ in range(num_repeats):
C = torch.matmul(A, B)
if device.type == "cuda":
torch.cuda.synchronize()
elapsed_time = (time.perf_counter() - start_time) / num_repeats
# Compute FLOPs: 2 * N^3 for GEMM
total_flops = 2 * (size ** 3)
tflops_per_sec = (total_flops / elapsed_time) / 1e12
# Memory Traffic: 3 matrices of size * size * bytes_per_elem
elem_bytes = 2 if dtype in (torch.float16, torch.bfloat16) else 4
total_bytes = 3 * (size ** 2) * elem_bytes
arithmetic_intensity = total_flops / total_bytes
return {
"matrix_size": size,
"dtype": str(dtype),
"latency_ms": round(elapsed_time * 1000, 3),
"tflops_per_sec": round(tflops_per_sec, 2),
"arithmetic_intensity": round(arithmetic_intensity, 2)
}
Implications: security, privacy, performance, scalability, and cost
- GPU Power & Thermal Throttling:
- Modern GPUs draw up to 700 Watts under full Tensor Core load. If datacenter cooling is inadequate, GPUs throttle clock speeds by
30\%+, silently degrading distributed training throughput.
- Modern GPUs draw up to 700 Watts under full Tensor Core load. If datacenter cooling is inadequate, GPUs throttle clock speeds by
- CUDA Side-Channel & Memory Leakage:
- Multi-tenant GPU sharing without virtualization (MIG / vGPU) can leak sensitive tensor memory traces across processes. Use NVIDIA Multi-Instance GPU (MIG) for strict physical memory isolation.
Alternatives: free, open source, and commercial
| Hardware Platform | Architecture | Precision Types | Peak Throughput | Target Use |
|---|---|---|---|---|
| NVIDIA H100 SXM | Hopper SM + Tensor Core | FP8 / FP16 / BF16 / FP32 | 1,000 TFLOPs (FP16) | Premier LLM Pretraining |
| Google TPU v5p | Systolic Array 3D Mesh | BF16 / INT8 / FP32 | 459 TFLOPs (BF16) | Large-Scale Pod Training |
| Apple M3/M4 Max | Unified Memory GPU/ANE | FP16 / BF16 / INT8 | 100+ GB Unified RAM | Local Edge Prototyping |
| AMD Instinct MI300X | CDNA 3 + 192GB HBM3 | FP8 / FP16 / BF16 | 1,300 TFLOPs (FP16) | High-VRAM LLM Serving |
Comparison with related concepts
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β AI HARDWARE ARCHITECTURE COMPARISON β
ββββββββββββββββββββββ¬ββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββ€
β Architecture β Execution Model β Memory Strategy β Best Workloadβ
ββββββββββββββββββββββΌββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββ€
β CPU (x86 / ARM) β Latency-Focused β Massive L3 Cache β Sequential β
β GPU (NVIDIA / AMD) β SIMT Throughput β High Bandwidth β GEMM / Train β
β TPU (Google) β Systolic Array β 3D Torus Interconβ Dense Batch β
β NPU / ANE (Apple) β Fixed-Function β Unified Memory β Edge On-Dev β
ββββββββββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββ
When to use it β and when not to
When to Optimize for GPU Hardware Architecture:
- Training deep neural networks, pretraining LLMs, high-throughput model serving, and developing custom fused CUDA/Triton kernels.
When NOT to use GPUs:
- Low-volume single-sample CPU inference where PCI-e bus transfer latency exceeds the computation time.
The AI connection
- Hardware, not algorithms, unlocked deep learning. Backpropagation was published in 1986; what changed was the arrival of cheap parallel arithmetic.
- The roofline model explains most performance surprises β whether a kernel is starved for data or saturating the arithmetic units decides what optimisation helps.
- Memory bandwidth is the scarce resource, which is why fused kernels and on-chip tiling produce larger wins than faster maths.
- Interconnect topology bounds cluster scaling, so the network between accelerators is as much a model-design constraint as the accelerators themselves.
- Utilisation is usually far below peak, and knowing the gap is what separates a real capacity estimate from a marketing number.
Knowledge check
- What is the fundamental difference between CUDA SIMT cores and Tensor Cores?
- What is Arithmetic Intensity and how does it determine whether a kernel is memory-bound or compute-bound?
- Why does single-token autoregressive generation have low arithmetic intensity?
- How does SRAM Shared Memory enable kernel fusion in FlashAttention?
- What is the difference between a GPU SIMT engine and a TPU Systolic Array?
Hands-on exercise
In this lab, you will build and test a modular hardware benchmarking suite in PyTorch: implement matrix multiplication FLOP calculations, measure execution latency across variable matrix sizes, compute Arithmetic Intensity across FP32 and FP16 data types, and determine the machine balance ridge point for hardware acceleration.
Expected output
[AI Hardware Performance Verification Suite]
Testing Matrix Multiply Benchmark (Size=512):
Precision: torch.float32 | FLOPs: 268.4M | Memory: 3.15 MB
Arithmetic Intensity: 85.33 FLOPs/Byte [MEASURED]
Testing Half-Precision Benchmark (Size=512, FP16):
Precision: torch.float16 | FLOPs: 268.4M | Memory: 1.57 MB
Arithmetic Intensity: 170.67 FLOPs/Byte (2x Data Density) [VERIFIED]
Testing Machine Balance Calculator:
Calculated Machine Balance: 125.0 FLOPs/Byte (Threshold for Compute Bound)
Test Suite: 3 passed in 0.41s
Validate your work
Run the automated test runner:
./tests/run_tests.sh
Troubleshooting
- When benchmarking on GPU, always call
torch.cuda.synchronize()before and after timing blocks to ensure asynchronous CUDA streams complete.
Common mistakes
- Measuring GPU Latency Without Synchronization: PyTorch CUDA operations are non-blocking; timing without
torch.cuda.synchronize()measures Python launch overhead rather than GPU kernel execution.
Practice assignment
- Implement a Roofline Model Plotter in Python using Matplotlib that plots the theoretical performance envelope for your local hardware.
- Benchmark an Elementwise LayerNorm vs a Dense Matrix Multiplication and prove that LayerNorm is memory-bandwidth bound.
Extension challenge
Implement a Simple Triton Fused GELU Kernel:
- Write a Triton kernel that fuses linear projection and GELU activation in SRAM shared memory.
- Benchmark memory traffic reduction against native PyTorch.
Quiz
Q1. What is the fundamental architectural difference between a CPU and a modern AI GPU (e.g. NVIDIA H100)?
- A CPU uses electricity while a GPU uses optical light
- A CPU optimises latency; a GPU optimises throughput
- A GPU cannot perform addition
- A CPU only runs Python
Show answer
Answer: B. A CPU optimises latency; a GPU optimises throughput
CPUs optimize for sequential latency and branch prediction; GPUs optimize for massive parallel arithmetic throughput.
Q2. What is Arithmetic Intensity (I) in the Roofline Model?
- The number of layers in the neural network
- The clock speed of the processor in Gigahertz
- FLOPs performed per byte moved from memory
- The temperature of the cooling liquid
Show answer
Answer: C. FLOPs performed per byte moved from memory
Arithmetic intensity measures how much computation is performed per byte of memory bandwidth consumed.
Q3. Why is single-token autoregressive generation in LLMs almost always Memory-Bandwidth Bound rather than Compute Bound?
- Because Python executes single threads
- Because GPUs disable Tensor Cores during generation
- Because tokenizers use too much disk space
- Every token reloads all the weights from memory
Show answer
Answer: D. Every token reloads all the weights from memory
Batch-1 generation reads massive weight matrices from memory for minimal compute, saturating memory bandwidth while leaving ALUs mostly idle.
Q4. How do Tensor Cores achieve an order-of-magnitude speedup over standard CUDA cores during deep learning training?
- Fused matrix multiply-accumulate per cycle
- They skip multiplying zeros
- They reduce the number of tokens in the prompt
- They run at 100 GHz clock speed
Show answer
Answer: A. Fused matrix multiply-accumulate per cycle
Tensor Cores compute entire matrix multiply-accumulate tiles in single hardware clock cycles using mixed precision.
Q5. What role does SRAM Shared Memory / L1 Cache play in modern GPU optimizations like FlashAttention?
- It stores operating system kernel logs
- Fast on-chip memory inside each SM
- It replaces the hard drive
- It encrypts network packets
Show answer
Answer: B. Fast on-chip memory inside each SM
SRAM on-chip shared memory enables kernel fusion and block tiling, bypassing the HBM memory bandwidth bottleneck.
Glossary
- Streaming Multiprocessor (SM)
- The fundamental compute building block of an NVIDIA GPU containing warp schedulers, CUDA cores, Tensor Cores, and SRAM shared memory.
- Tensor Core
- A specialized hardware execution unit executing fused mixed-precision matrix multiply-accumulate operations (D = A * B + C) in a single clock cycle.
- Arithmetic Intensity
- The ratio of arithmetic operations (FLOPs) to memory traffic (Bytes) transferred across High Bandwidth Memory: I = FLOPs / Byte.
- Roofline Model
- A visual performance model that plots attainable computational throughput against arithmetic intensity, bounding performance by memory bandwidth and peak compute ceilings.
- High Bandwidth Memory (HBM3e)
- 3D-stacked DRAM on the GPU interposer providing multi-terabyte-per-second memory bandwidth.
- SRAM Shared Memory
- Ultra-low-latency on-chip memory directly integrated into each SM, delivering ~19 TB/s aggregate bandwidth.
- Memory-Bound Operation
- An algorithm whose execution speed is limited by memory bandwidth transfer rates rather than processor arithmetic capacity.
- Compute-Bound Operation
- An algorithm whose execution speed is limited by raw processor arithmetic throughput rather than memory data transfer.
Sources and further reading
- Roofline: An Insightful Visual Performance Model for Multicore Architectures β Communications of the ACM 2009 (accessed 2026-08-29)
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness β NeurIPS 2022 (accessed 2026-08-29)
- NVIDIA Hopper Architecture In-Depth β NVIDIA Developer Blog (accessed 2026-08-29)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.