Deep Learning › Training at Scale › Day 232

Hands-on lab — Day 232: GPUs and AI Hardware

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/gpus_and_ai_hardware_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/gpus_and_ai_hardware_lib.py
examples/test_gpus_and_ai_hardware_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/gpus_and_ai_hardware_lib.py
starter/test_gpus_and_ai_hardware_lib.py
tests/run_tests.sh
tests/test_gpus_and_ai_hardware_lib.py
troubleshooting.md

Lab README

Lab: Day 232 -- GPUs and AI Hardware

Lesson

Day number: 232 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: AI Hardware, GPU Architecture, and the Roofline Model.

Purpose

Build and test a modular hardware performance analysis and Arithmetic Intensity calculation suite in PyTorch. Implement GEMM FLOP formulas, compute memory traffic across precision modes, and determine the machine balance ridge point for hardware acceleration.

Learning objectives

  • Calculate GEMM floating-point operation counts and memory transfer footprints.
  • Compute Arithmetic Intensity across FP32 and FP16 numerical representations.
  • Calculate hardware machine balance to diagnose memory-bandwidth vs compute bounds.
  • Verify hardware performance calculations against theoretical limits.

Prerequisites

  • Day 231 (Fine-Tuning a Small Transformer).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/gpus_and_ai_hardware_lib.py: Student scaffold file.
  • examples/gpus_and_ai_hardware_lib.py: Complete reference implementation.
  • tests/test_gpus_and_ai_hardware_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/gpus_and_ai_hardware_lib.py

What the commands do

  • Calculates Arithmetic Intensity across FP16 and FP32 precision modes.
  • Computes machine balance ridge points.
  • Runs unit test assertions.

Expected output

Hardware Demo: FP16 Intensity = 170.67 FLOPs/Byte, Balance = 125.0

Validation steps

  1. Verify compute_gemm_intensity calculates $2 \times N^3$ FLOPs.
  2. Confirm FP16 yields $2\times$ higher arithmetic intensity than FP32 for identical matrix dimensions.
  3. Confirm calculate_machine_balance correctly divides peak TFLOPs by peak bandwidth.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Division by zero in intensity: Ensure matrix size $N > 0$ and dtype_bytes > 0.

Security notes

All performance benchmarks execute locally in process memory.

Extension exercises

  1. Implement a Roofline Model visualization plot using Matplotlib.
  2. Profile PyTorch matrix multiplication using torch.profiler.
  • Lesson title: GPUs and AI Hardware
  • Day number: 232 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-232-gpus-and-ai-hardware
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-232-gpus-and-ai-hardware when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 232

- `FP16 Intensity`: Arithmetic intensity in FLOPs per Byte for FP16 GEMM.
- `Balance`: Calculated hardware machine balance ridge point.

examples-run.txt

Hardware Demo: FP16 Intensity = 170.67 FLOPs/Byte, Balance = 125.0

measured-values.txt

FP16 Intensity: 170.67 FLOPs/Byte
Balance: 125.0

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_gpus_and_ai_hardware_lib.py::test_gemm_intensity_fp16 PASSED    [ 33%]
tests/test_gpus_and_ai_hardware_lib.py::test_gemm_intensity_fp32 PASSED    [ 66%]
tests/test_gpus_and_ai_hardware_lib.py::test_machine_balance PASSED        [100%]

============================== 3 passed in 0.41s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/gpus_and_ai_hardware_lib.py (1085 bytes)
import torch
from typing import Dict, Any

def compute_gemm_intensity(size: int, dtype_bytes: int = 2) -> Dict[str, Any]:
    total_flops = 2 * (size ** 3)
    total_bytes = 3 * (size ** 2) * dtype_bytes
    intensity = total_flops / total_bytes

    return {
        "size": size,
        "dtype_bytes": dtype_bytes,
        "total_flops": total_flops,
        "total_bytes": total_bytes,
        "arithmetic_intensity": round(intensity, 2)
    }

def calculate_machine_balance(peak_tflops: float, peak_bandwidth_tbs: float) -> float:
    # Peak FLOPs / Peak Bandwidth (TFLOPs / TB/s = FLOPs / Byte)
    return round(peak_tflops / peak_bandwidth_tbs, 2)

def run_hardware_demo():
    res_fp32 = compute_gemm_intensity(size=512, dtype_bytes=4)
    res_fp16 = compute_gemm_intensity(size=512, dtype_bytes=2)
    balance = calculate_machine_balance(peak_tflops=100.0, peak_bandwidth_tbs=0.8)

    print(f"Hardware Demo: FP16 Intensity = {res_fp16['arithmetic_intensity']} FLOPs/Byte, Balance = {balance}")
    return res_fp16, balance

if __name__ == "__main__":
    run_hardware_demo()
examples/test_gpus_and_ai_hardware_lib.py (704 bytes)
import pytest
from examples.gpus_and_ai_hardware_lib import compute_gemm_intensity, calculate_machine_balance

def test_gemm_intensity_fp16():
    res = compute_gemm_intensity(size=512, dtype_bytes=2)
    assert res["total_flops"] == 2 * (512 ** 3)
    assert res["total_bytes"] == 3 * (512 ** 2) * 2
    assert res["arithmetic_intensity"] == round((2 * 512) / 6, 2) # ~170.67

def test_gemm_intensity_fp32():
    res = compute_gemm_intensity(size=512, dtype_bytes=4)
    assert res["arithmetic_intensity"] == round((2 * 512) / 12, 2) # ~85.33

def test_machine_balance():
    balance = calculate_machine_balance(peak_tflops=1000.0, peak_bandwidth_tbs=3.35)
    assert balance == round(1000.0 / 3.35, 2)
metadata.yml (433 bytes)
lesson_id: D232
day: 232
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/gpus_and_ai_hardware_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/gpus_and_ai_hardware_lib.py (372 bytes)
import torch
from typing import Dict, Any

def compute_gemm_intensity(size: int, dtype_bytes: int = 2) -> Dict[str, Any]:
    # TODO: Calculate total FLOPs, total bytes transferred, and arithmetic intensity for GEMM
    pass

def calculate_machine_balance(peak_tflops: float, peak_bandwidth_tbs: float) -> float:
    # TODO: Calculate machine balance ridge point
    pass
starter/test_gpus_and_ai_hardware_lib.py (704 bytes)
import pytest
from examples.gpus_and_ai_hardware_lib import compute_gemm_intensity, calculate_machine_balance

def test_gemm_intensity_fp16():
    res = compute_gemm_intensity(size=512, dtype_bytes=2)
    assert res["total_flops"] == 2 * (512 ** 3)
    assert res["total_bytes"] == 3 * (512 ** 2) * 2
    assert res["arithmetic_intensity"] == round((2 * 512) / 6, 2) # ~170.67

def test_gemm_intensity_fp32():
    res = compute_gemm_intensity(size=512, dtype_bytes=4)
    assert res["arithmetic_intensity"] == round((2 * 512) / 12, 2) # ~85.33

def test_machine_balance():
    balance = calculate_machine_balance(peak_tflops=1000.0, peak_bandwidth_tbs=3.35)
    assert balance == round(1000.0 / 3.35, 2)
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 232 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_gpus_and_ai_hardware_lib.py (704 bytes)
import pytest
from examples.gpus_and_ai_hardware_lib import compute_gemm_intensity, calculate_machine_balance

def test_gemm_intensity_fp16():
    res = compute_gemm_intensity(size=512, dtype_bytes=2)
    assert res["total_flops"] == 2 * (512 ** 3)
    assert res["total_bytes"] == 3 * (512 ** 2) * 2
    assert res["arithmetic_intensity"] == round((2 * 512) / 6, 2) # ~170.67

def test_gemm_intensity_fp32():
    res = compute_gemm_intensity(size=512, dtype_bytes=4)
    assert res["arithmetic_intensity"] == round((2 * 512) / 12, 2) # ~85.33

def test_machine_balance():
    balance = calculate_machine_balance(peak_tflops=1000.0, peak_bandwidth_tbs=3.35)
    assert balance == round(1000.0 / 3.35, 2)

Troubleshooting

Troubleshooting: Day 232 - GPUs and AI Hardware

Common Issues

  1. Intensity calculation discrepancy:
    • Cause: Forgetting the factor of 2 in GEMM FLOPs ($2 N^3$ due to multiply and add).
    • Fix: Use $2 \times N^3$.

Security notes

Security & Privacy: Day 232 - GPUs and AI Hardware

Security Guidance

  • All hardware benchmark calculations execute locally in memory.