Deep Learning β€Ί Training at Scale β€Ί Day 233

Day 233: Mixed Precision and Performance

Day 233 of 365 β€” Mixed Precision and Performance

Master mixed-precision acceleration and deep learning performance engineering: understand FP32, FP16, BF16, and FP8 numerical formats, implement PyTorch Automatic Mixed Precision (AMP) with autocast and dynamic loss scaling (GradScaler), analyze training VRAM memory allocation across weights and optimizer states, optimize execution with torch.compile, and build a high-throughput benchmarking suite.

Course
Deep Learning
Category
Training at Scale
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-233-mixed-precision-and-performance

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-233-mixed-precision-and-performance
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

For decades, scientific computing relied exclusively on IEEE 754 32-bit Single Precision (FP32) and 64-bit Double Precision (FP64).

However, deep neural networks possess an astonishing mathematical property: Robust Graceful Precision Tolerance:

Today, training frontier models entirely in FP32 is obsolete. Every major AI laboratory uses Automatic Mixed Precision (AMP), Bfloat16, and FP8 Transformer Engines.

Today, you will master the bit-level anatomy of floating-point representations, dynamic loss scaling (GradScaler), VRAM memory budget calculations, and PyTorch 2.0 torch.compile kernel fusion.


The idea in plain language

Diagram: how mixed-precision training combines low and full precision

Think of writing notes during a fast-paced university lecture:

You get 100\% of the lecture content recorded in real-time with zero loss in factual accuracy!


Historical background

Diagram: the historical progression of numerical formats in deep learning training

Floating point number formats comparison showing sign exponent and mantissa bits for FP32 FP16 BF16 and FP8

  1. 2017 (Micikevicius et al. & Mixed Precision Training): NVIDIA published the foundational paper establishing that training deep networks in FP16 with FP32 master weights and loss scaling achieved identical validation accuracy to FP32 baselines.
  2. 2018 (Google TPU v2 & Bfloat16): Google created Bfloat16 (Brain Floating Point), truncating FP32 mantissa while preserving its full 8-bit exponent, eliminating loss scaling entirely.
  3. 2022 (NVIDIA Hopper H100 & FP8 Transformer Engine): Introduced native 8-bit floating-point Tensor Cores, doubling throughput over FP16.
  4. 2023 (PyTorch 2.0 & torch.compile): Replaced manual CUDA kernel engineering with JIT graph capture and automated Triton kernel fusion.

What it is β€” and what it is not

What Automatic Mixed Precision (AMP) IS:

What it is NOT:


Why it was created and what problems it solves

Mixed Precision solved the twin bottlenecks of AI scaling:

  1. Memory Bandwidth Saturation: Halving byte size instantly doubles effective memory bandwidth across the bus.
  2. GPU VRAM Capacity Limits: Enables training models twice as large on existing physical hardware.
  3. Silicon Energy Efficiency: 16-bit and 8-bit ALUs consume a fraction of the physical silicon area and electrical energy of 32-bit ALUs.

How it works

Let us dissect floating-point numerical structures, loss scaling algorithms, and VRAM memory mathematics.


1. Anatomy of Floating-Point Representations

Diagram: two sixteen-bit floating point formats compared by range and precision

A floating point number represents a real value via:

\text{Value} = (-1)^{\text{sign}} \times 2^{\text{exponent} - \text{bias}} \times (1 + \text{mantissa})
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   FLOATING POINT BITWISE COMPARISON                    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Format             β”‚ Sign Bit β”‚ Exponent (E) β”‚ Mantissa β”‚ Dynamic Rangeβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ FP32 (Single)      β”‚ 1 bit    β”‚ 8 bits       β”‚ 23 bits  β”‚ ~1e-38 to 3e38β”‚
β”‚ FP16 (Half)        β”‚ 1 bit    β”‚ 5 bits       β”‚ 10 bits  β”‚ ~6e-8 to 65504β”‚
β”‚ BF16 (Bfloat16)    β”‚ 1 bit    β”‚ 8 bits       β”‚ 7 bits   β”‚ ~1e-38 to 3e38β”‚
β”‚ FP8 (E4M3)         β”‚ 1 bit    β”‚ 4 bits       β”‚ 3 bits   β”‚ ~0.002 to 448β”‚
β”‚ FP8 (E5M2)         β”‚ 1 bit    β”‚ 5 bits       β”‚ 2 bits   β”‚ ~6e-8 to 57344β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Why BF16 Beat FP16:


2. Automatic Mixed Precision (AMP) Workflow & GradScaler

Diagram: how dynamic loss scaling prevents gradient underflow in low precision

Automatic Mixed Precision training loop flow showing FP16 forward pass loss scaling backpropagation and FP32 optimizer update

When using FP16, the GradScaler manages dynamic loss scaling:

  1. Forward Pass (autocast): Executes linear projections in FP16; outputs loss \mathcal{L} in FP32.

  2. Dynamic Loss Scaling: Multiplies loss by scale factor S (initialized to S = 65,536):

    \mathcal{L}_{\text{scaled}} = \mathcal{L} \times S
  3. Backward Pass: Computes scaled gradients g_{\text{scaled}} = g \times S in FP16 (shifted away from zero underflow).

  4. Unscaling & Inf/NaN Checking:

    g = \frac{g_{\text{scaled}}}{S}
    • If any gradient contains Inf or NaN (overflow detected), the optimizer step is skipped, and the scaler shrinks S \leftarrow S / 2.
    • If N consecutive steps succeed without overflow (e.g. N=2000), the scaler grows S \leftarrow S \times 2.
  5. Master Weight Update: Applies g to the FP32 Master Weights, and copies updated parameters back to FP16 for the next forward pass.


3. Exact GPU VRAM Breakdown for Mixed-Precision AdamW

Diagram: the four consumers of training memory per parameter

When training an N-parameter model in mixed-precision AdamW:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚             VRAM MEMORY BREAKDOWN (PER PARAMETER IN ADAMW)             β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Component          β”‚ Data Typeβ”‚ Bytes / Paramβ”‚ Example (7B Model)      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Model Parameters   β”‚ FP16/BF16β”‚ 2 Bytes      β”‚ 14 GB                   β”‚
β”‚ Gradients          β”‚ FP16/BF16β”‚ 2 Bytes      β”‚ 14 GB                   β”‚
β”‚ Master Weights     β”‚ FP32     β”‚ 4 Bytes      β”‚ 28 GB                   β”‚
β”‚ Adam First Moment  β”‚ FP32     β”‚ 4 Bytes      β”‚ 28 GB                   β”‚
β”‚ Adam Second Moment β”‚ FP32     β”‚ 4 Bytes      β”‚ 28 GB                   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ TOTAL STATIC STATE β”‚          β”‚ 16 Bytes     β”‚ 112 GB                  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

4. Kernel Fusion with torch.compile (PyTorch 2.0)

Diagram: separate elementwise kernels compared with one fused kernel

In uncompiled PyTorch, executing:

y = F.gelu(x @ W + b) + residual

launches 4 separate GPU kernels:

  1. Matrix multiplication: Reads x, W \rightarrow writes temp1 to HBM.
  2. Bias addition: Reads temp1, b \rightarrow writes temp2 to HBM.
  3. GELU activation: Reads temp2 \rightarrow writes temp3 to HBM.
  4. Residual addition: Reads temp3, residual \rightarrow writes y to HBM.

torch.compile(model) with TorchInductor:


5. Microscaling Formats (MXFP8, MXFP6, MXFP4 / OCP Standard)

As models scale past 100 billion parameters, standard FP8 encounters precision degradation in outlier activations:


6. Activation Checkpointing (Gradient Checkpointing / Chen et al., 2016)

While weights and optimizer states are static, Intermediate Forward Activations grow linearly with sequence length L and batch size B:


7. 8-Bit Optimizers: 8-bit Adam (Dettmers et al., 2021)

Diagram: how quantised optimiser state reduces training memory

Standard mixed-precision AdamW maintains 8 bytes per parameter for momentum and variance in FP32:


8. Custom Autocast Layer Registration

When implementing custom CUDA or C++ extensions in PyTorch, standard autocasting does not know how to handle your functions:


An everyday analogy

Diagram: verbatim transcription compared with abbreviated note-taking

Think of a commercial cargo airplane fleet:


Examples in practice

Let us build a Complete Mixed-Precision Training Benchmark in PyTorch:

import torch
import torch.nn as nn
import time

class BenchmarkMLP(nn.Module):
    def __init__(self, dim: int = 2048):
        super().__init__()
        self.fc1 = nn.Linear(dim, dim * 4)
        self.act = nn.GELU()
        self.fc2 = nn.Linear(dim * 4, dim)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return x + self.fc2(self.act(self.fc1(x)))

def run_mixed_precision_benchmark(batch_size: int = 64, dim: int = 2048,
                                   steps: int = 20) -> dict:
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = BenchmarkMLP(dim).to(device)
    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
    scaler = torch.amp.GradScaler("cuda", enabled=(device.type == "cuda"))

    inputs = torch.randn(batch_size, dim, device=device)
    targets = torch.randn(batch_size, dim, device=device)
    criterion = nn.MSELoss()

    # Warmup
    for _ in range(5):
        optimizer.zero_grad()
        with torch.amp.autocast(device_type=device.type, dtype=torch.float16, enabled=(device.type == "cuda")):
            outputs = model(inputs)
            loss = criterion(outputs, targets)
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

    if device.type == "cuda":
        torch.cuda.synchronize()

    # Timed benchmark loop
    start = time.perf_counter()
    for _ in range(steps):
        optimizer.zero_grad()
        with torch.amp.autocast(device_type=device.type, dtype=torch.float16, enabled=(device.type == "cuda")):
            outputs = model(inputs)
            loss = criterion(outputs, targets)
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

    if device.type == "cuda":
        torch.cuda.synchronize()
    elapsed = (time.perf_counter() - start) / steps

    return {
        "batch_size": batch_size,
        "step_latency_ms": round(elapsed * 1000, 3),
        "scaler_scale": scaler.get_scale() if device.type == "cuda" else 1.0
    }

Implications: security, privacy, performance, scalability, and cost

Diagram: the throughput, memory, reproducibility and debugging implications of mixed precision

  1. Numerical Stability & NaN Cascades:
    • In FP16, a single activation exceeding 65,504 produces Inf. In LayerNorm, \text{Inf} - \text{Inf} = \text{NaN}, which instantly infects all model weights in the next optimizer step.
    • Remedy: Always use BF16 on Ampere/Hopper GPUs, or maintain conservative gradient clipping (max_norm=1.0) in FP16.
  2. Energy Efficiency in Green AI:
    • Switching large LLM pretraining from FP32 to BF16/FP8 slashes total datacenter energy consumption by 60\%, saving megawatts of electricity and millions of dollars in utility bills.

Alternatives: free, open source, and commercial

Precision ModeHardware SupportLoss Scaler Required?Relative SpeedStability
FP32Universal (CPUs/GPUs)No1.0\times BaselineRock Solid
FP16Volta / Turing / AmpereYes (Mandatory)2.0\times - 3.0\timesModerate (Underflow risk)
BF16Ampere / Hopper / TPUNo2.0\times - 3.0\timesRock Solid (SOTA Standard)
FP8Hopper H100 / BlackwellAdaptive scaling4.0\times - 5.0\timesHigh with scaling

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   PRECISION FORMAT COMPARISON SUMMARY                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Format             β”‚ Exponent Bits   β”‚ Mantissa Bits    β”‚ Dynamic Rangeβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ FP32               β”‚ 8 bits          β”‚ 23 bits          β”‚ 10^{38}      β”‚
β”‚ FP16               β”‚ 5 bits          β”‚ 10 bits          β”‚ 10^4 (Narrow)β”‚
β”‚ BF16               β”‚ 8 bits          β”‚ 7 bits           β”‚ 10^{38} (Wide)β”‚
β”‚ FP8 (E4M3)         β”‚ 4 bits          β”‚ 3 bits           β”‚ 10^2 (Micro) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when mixed precision is appropriate and when full precision is required

When to USE Mixed Precision (BF16/FP16):

When NOT to use:


The AI connection

Knowledge check

  1. Why does BF16 eliminate the need for dynamic loss scaling compared to FP16?
  2. What happens during a GradScaler step if an Inf or NaN gradient is detected?
  3. How many bytes of static VRAM per parameter are required by mixed-precision AdamW?
  4. How does torch.compile eliminate HBM memory traffic via kernel fusion?
  5. What are the roles of E4M3 vs E5M2 formats in FP8 training?

Hands-on exercise

In this lab, you will build and test a modular mixed-precision simulation and VRAM memory profiling engine in PyTorch: implement precision bitwise representations, compute exact static AdamW VRAM allocation formulas, construct a simulated dynamic loss scaling engine with overflow detection, and verify training throughput acceleration.

Expected output

[Mixed Precision and Performance Verification Suite]
Testing VRAM Memory Budget Profiler:
  Model Size: 7.0 Billion Parameters
  Static VRAM Allocation (Mixed Precision AdamW): 112.00 GB [EXACT MATCH]
Testing Dynamic Loss Scaling Simulator:
  Initial Scale Factor: 65536.0
  Normal Step: Scale maintained (65536.0)
  Overflow Detected (Inf Gradient): Scale backoff to 32768.0 [OVERFLOW CAUGHT]
Testing Bitwise Numerical Properties:
  BF16 Exponent Bits: 8 (Matches FP32 dynamic range) [VERIFIED]
Test Suite: 3 passed in 0.42s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a VRAM Memory Budget Calculator function that accepts model parameters, sequence length, batch size, and returns peak activation + static memory estimates.
  2. Benchmark torch.compile on a 6-layer Transformer block in PyTorch.

Extension challenge

Implement FP8 Simulated Quantization in PyTorch:

Quiz

Q1. Why does Bfloat16 (BF16) avoid the gradient underflow issues that affect standard FP16 during deep neural network training?

  1. BF16 eliminates negative numbers
  2. BF16 uses integer arithmetic instead of floating point
  3. BF16 keeps FP32's exponent range, so gradients survive
  4. BF16 requires 64 bits of storage
Show answer

Answer: C. BF16 keeps FP32's exponent range, so gradients survive

BF16 preserves FP32 wide 8-bit dynamic range, eliminating the need for loss scaling during training.

Q2. What is the purpose of the GradScaler in PyTorch Automatic Mixed Precision (AMP) with FP16?

  1. It shrinks the vocabulary size
  2. It scales up the learning rate by 10x
  3. It doubles the batch size
  4. It scales the loss before backward
Show answer

Answer: D. It scales the loss before backward

GradScaler prevents small backpropagation gradients from underflowing to zero in 16-bit half precision.

Q3. How much GPU VRAM does a 7-billion parameter model require just to store training states in mixed-precision AdamW?

  1. 112 GB
  2. 7 GB
  3. 1 GB
  4. 500 MB
Show answer

Answer: A. 112 GB

Mixed-precision AdamW requires 16 bytes per parameter (2B model + 2B grad + 4B master + 4B moment 1 + 4B moment 2), totaling 112 GB for 7B parameters.

Q4. What is the primary optimization performed by torch.compile (PyTorch 2.0 TorchDynamo / Inductor)?

  1. It translates Python into Java
  2. It fuses ops into one kernel
  3. It compresses dataset text
  4. It disables backpropagation
Show answer

Answer: B. It fuses ops into one kernel

torch.compile automatically fuses elementwise kernels, keeping activations in fast SRAM and bypassing HBM bandwidth limits.

Q5. What are the two distinct FP8 numerical variants defined for deep learning (Hopper/Blackwell architecture)?

  1. FP8-CPU and FP8-GPU
  2. FP8-Left and FP8-Right
  3. E4M3 and E5M2
  4. FP8-Int and FP8-Float
Show answer

Answer: C. E4M3 and E5M2

E4M3 maximizes precision for forward passes; E5M2 maximizes dynamic range to capture gradient distributions.

Glossary

Automatic Mixed Precision (AMP)
A framework executing performance-critical matrix operations in 16-bit half precision while maintaining 32-bit master weights for stability.
GradScaler
A dynamic loss scaling engine that multiplies loss by a scale factor to prevent gradient underflow during FP16 backward passes.
Bfloat16 (BF16)
A 16-bit floating point format with 1 sign bit, 8 exponent bits (matching FP32 range), and 7 mantissa bits.
FP8 (E4M3 / E5M2)
An 8-bit floating point format delivering double the throughput of 16-bit on modern NVIDIA Hopper and Blackwell architectures.
Kernel Fusion
Combining multiple sequential operations into a single GPU kernel to eliminate intermediate memory reads and writes across HBM.
torch.compile
The PyTorch 2.0 JIT compiler utilizing TorchDynamo and TorchInductor to optimize PyTorch code into high-speed Triton/C++ kernels.
Master Weights
A full-precision FP32 copy of model parameters maintained by the optimizer to accumulate small parameter updates.
Gradient Underflow
The numerical failure when small backpropagation gradients drop below the minimum representable value of FP16 and vanish to zero.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.