Deep Learning › Training at Scale › Day 232
Hands-on lab — Day 232: GPUs and AI Hardware
- ← Back to the Day 232 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-232-gpus-and-ai-hardware/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/gpus_and_ai_hardware_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/gpus_and_ai_hardware_lib.py examples/test_gpus_and_ai_hardware_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/gpus_and_ai_hardware_lib.py starter/test_gpus_and_ai_hardware_lib.py tests/run_tests.sh tests/test_gpus_and_ai_hardware_lib.py troubleshooting.md
Lab README
Lab: Day 232 -- GPUs and AI Hardware
Lesson
Day number: 232 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: AI Hardware, GPU Architecture, and the Roofline Model.
Purpose
Build and test a modular hardware performance analysis and Arithmetic Intensity calculation suite in PyTorch. Implement GEMM FLOP formulas, compute memory traffic across precision modes, and determine the machine balance ridge point for hardware acceleration.
Learning objectives
- Calculate GEMM floating-point operation counts and memory transfer footprints.
- Compute Arithmetic Intensity across FP32 and FP16 numerical representations.
- Calculate hardware machine balance to diagnose memory-bandwidth vs compute bounds.
- Verify hardware performance calculations against theoretical limits.
Prerequisites
- Day 231 (Fine-Tuning a Small Transformer).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/gpus_and_ai_hardware_lib.py: Student scaffold file.examples/gpus_and_ai_hardware_lib.py: Complete reference implementation.tests/test_gpus_and_ai_hardware_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/gpus_and_ai_hardware_lib.py
What the commands do
- Calculates Arithmetic Intensity across FP16 and FP32 precision modes.
- Computes machine balance ridge points.
- Runs unit test assertions.
Expected output
Hardware Demo: FP16 Intensity = 170.67 FLOPs/Byte, Balance = 125.0
Validation steps
- Verify
compute_gemm_intensitycalculates $2 \times N^3$ FLOPs. - Confirm FP16 yields $2\times$ higher arithmetic intensity than FP32 for identical matrix dimensions.
- Confirm
calculate_machine_balancecorrectly divides peak TFLOPs by peak bandwidth. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Division by zero in intensity: Ensure matrix size $N > 0$ and
dtype_bytes > 0.
Security notes
All performance benchmarks execute locally in process memory.
Extension exercises
- Implement a Roofline Model visualization plot using Matplotlib.
- Profile PyTorch matrix multiplication using
torch.profiler.
Navigation
- Lesson title: GPUs and AI Hardware
- Day number: 232 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-232-gpus-and-ai-hardware
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-232-gpus-and-ai-hardwarewhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 232
- `FP16 Intensity`: Arithmetic intensity in FLOPs per Byte for FP16 GEMM.
- `Balance`: Calculated hardware machine balance ridge point.
examples-run.txt
Hardware Demo: FP16 Intensity = 170.67 FLOPs/Byte, Balance = 125.0
measured-values.txt
FP16 Intensity: 170.67 FLOPs/Byte
Balance: 125.0
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_gpus_and_ai_hardware_lib.py::test_gemm_intensity_fp16 PASSED [ 33%]
tests/test_gpus_and_ai_hardware_lib.py::test_gemm_intensity_fp32 PASSED [ 66%]
tests/test_gpus_and_ai_hardware_lib.py::test_machine_balance PASSED [100%]
============================== 3 passed in 0.41s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/gpus_and_ai_hardware_lib.py (1085 bytes)
import torch
from typing import Dict, Any
def compute_gemm_intensity(size: int, dtype_bytes: int = 2) -> Dict[str, Any]:
total_flops = 2 * (size ** 3)
total_bytes = 3 * (size ** 2) * dtype_bytes
intensity = total_flops / total_bytes
return {
"size": size,
"dtype_bytes": dtype_bytes,
"total_flops": total_flops,
"total_bytes": total_bytes,
"arithmetic_intensity": round(intensity, 2)
}
def calculate_machine_balance(peak_tflops: float, peak_bandwidth_tbs: float) -> float:
# Peak FLOPs / Peak Bandwidth (TFLOPs / TB/s = FLOPs / Byte)
return round(peak_tflops / peak_bandwidth_tbs, 2)
def run_hardware_demo():
res_fp32 = compute_gemm_intensity(size=512, dtype_bytes=4)
res_fp16 = compute_gemm_intensity(size=512, dtype_bytes=2)
balance = calculate_machine_balance(peak_tflops=100.0, peak_bandwidth_tbs=0.8)
print(f"Hardware Demo: FP16 Intensity = {res_fp16['arithmetic_intensity']} FLOPs/Byte, Balance = {balance}")
return res_fp16, balance
if __name__ == "__main__":
run_hardware_demo()
examples/test_gpus_and_ai_hardware_lib.py (704 bytes)
import pytest
from examples.gpus_and_ai_hardware_lib import compute_gemm_intensity, calculate_machine_balance
def test_gemm_intensity_fp16():
res = compute_gemm_intensity(size=512, dtype_bytes=2)
assert res["total_flops"] == 2 * (512 ** 3)
assert res["total_bytes"] == 3 * (512 ** 2) * 2
assert res["arithmetic_intensity"] == round((2 * 512) / 6, 2) # ~170.67
def test_gemm_intensity_fp32():
res = compute_gemm_intensity(size=512, dtype_bytes=4)
assert res["arithmetic_intensity"] == round((2 * 512) / 12, 2) # ~85.33
def test_machine_balance():
balance = calculate_machine_balance(peak_tflops=1000.0, peak_bandwidth_tbs=3.35)
assert balance == round(1000.0 / 3.35, 2)
metadata.yml (433 bytes)
lesson_id: D232
day: 232
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/gpus_and_ai_hardware_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/gpus_and_ai_hardware_lib.py (372 bytes)
import torch
from typing import Dict, Any
def compute_gemm_intensity(size: int, dtype_bytes: int = 2) -> Dict[str, Any]:
# TODO: Calculate total FLOPs, total bytes transferred, and arithmetic intensity for GEMM
pass
def calculate_machine_balance(peak_tflops: float, peak_bandwidth_tbs: float) -> float:
# TODO: Calculate machine balance ridge point
pass
starter/test_gpus_and_ai_hardware_lib.py (704 bytes)
import pytest
from examples.gpus_and_ai_hardware_lib import compute_gemm_intensity, calculate_machine_balance
def test_gemm_intensity_fp16():
res = compute_gemm_intensity(size=512, dtype_bytes=2)
assert res["total_flops"] == 2 * (512 ** 3)
assert res["total_bytes"] == 3 * (512 ** 2) * 2
assert res["arithmetic_intensity"] == round((2 * 512) / 6, 2) # ~170.67
def test_gemm_intensity_fp32():
res = compute_gemm_intensity(size=512, dtype_bytes=4)
assert res["arithmetic_intensity"] == round((2 * 512) / 12, 2) # ~85.33
def test_machine_balance():
balance = calculate_machine_balance(peak_tflops=1000.0, peak_bandwidth_tbs=3.35)
assert balance == round(1000.0 / 3.35, 2)
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 232 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_gpus_and_ai_hardware_lib.py (704 bytes)
import pytest
from examples.gpus_and_ai_hardware_lib import compute_gemm_intensity, calculate_machine_balance
def test_gemm_intensity_fp16():
res = compute_gemm_intensity(size=512, dtype_bytes=2)
assert res["total_flops"] == 2 * (512 ** 3)
assert res["total_bytes"] == 3 * (512 ** 2) * 2
assert res["arithmetic_intensity"] == round((2 * 512) / 6, 2) # ~170.67
def test_gemm_intensity_fp32():
res = compute_gemm_intensity(size=512, dtype_bytes=4)
assert res["arithmetic_intensity"] == round((2 * 512) / 12, 2) # ~85.33
def test_machine_balance():
balance = calculate_machine_balance(peak_tflops=1000.0, peak_bandwidth_tbs=3.35)
assert balance == round(1000.0 / 3.35, 2)
Troubleshooting
Troubleshooting: Day 232 - GPUs and AI Hardware
Common Issues
- Intensity calculation discrepancy:
- Cause: Forgetting the factor of 2 in GEMM FLOPs ($2 N^3$ due to multiply and add).
- Fix: Use $2 \times N^3$.
Security notes
Security & Privacy: Day 232 - GPUs and AI Hardware
Security Guidance
- All hardware benchmark calculations execute locally in memory.