Deep Learning β€Ί Training at Scale β€Ί Day 234

Day 234: Distributed Training Concepts

Day 234 of 365 β€” Distributed Training Concepts

Master distributed training architectures and scaling paradigms: compare DDP vs DP, analyze Ring-AllReduce collective communications, understand ZeRO memory stages (ZeRO-1, ZeRO-2, ZeRO-3 / PyTorch FSDP), master 3D Parallelism (Tensor Parallelism Megatron-LM, Pipeline Parallelism 1F1B schedule, and Sequence Parallelism), diagnose communication-to-computation overlap, and simulate distributed rank synchronization in PyTorch.

Course
Deep Learning
Category
Training at Scale
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-234-distributed-training-concepts

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-234-distributed-training-concepts
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

A single modern GPU (such as an NVIDIA H100 with 80 GB VRAM) is an engineering marvel.

Yet, a 70-Billion parameter LLaMA model in mixed precision requires over 1,120 GB of VRAM just for static training states (weights, gradients, optimizer moments), completely blowing past the physical memory limit of any single chip on Earth.

To train frontier models, we must distribute computation across clusters of thousands of GPUs:

Today, you will master the principles of Distributed Data Parallelism (DDP), the Ring-AllReduce algorithm, ZeRO Memory Sharding (FSDP), and 3D Parallelism.


The idea in plain language

Diagram: the four steps of synchronous data-parallel training

Think of constructing a 100-story skyscraper:


Historical background

Diagram: the historical progression of distributed deep learning strategies

Taxonomy of 3D Parallelism showing Data Parallelism Tensor Parallelism and Pipeline Parallelism across multi-GPU clusters

  1. 2017 (Baidu & Ring-AllReduce): Andrew Gibiansky and Baidu Research brought Ring-AllReduce from High Performance Computing (HPC) into deep learning, demonstrating linear scaling across 128 GPUs.
  2. 2019 (NVIDIA & Megatron-LM): Shoeybi et al. published Tensor Parallelism, enabling intra-node matrix slicing with minimal communication overhead.
  3. 2019 (Microsoft DeepSpeed & ZeRO): Rajbhandari et al. introduced the Zero Redundancy Optimizer, proving that memory redundancy could be eliminated without the complexity of pipeline partitioning.
  4. 2023 (Meta AI & PyTorch FSDP): Productionized Fully Sharded Data Parallel natively in PyTorch core, making trillion-parameter distributed scaling accessible to all engineers.

What it is β€” and what it is not

What Distributed Training IS:

What it is NOT:


Why it was created and what problems it solves

Distributed training solves two fundamental scaling barriers:

  1. The Throughput Barrier (Dataset Scale): Pretraining an LLM on 15 Trillion tokens on 1 GPU would take 150 years. With 1,024 GPUs in Data Parallelism, training completes in 50 days.
  2. The Memory Capacity Barrier (Model Size): Storing a 405B parameter model requires 6.5 TB of VRAM. Distributed sharding (FSDP / Tensor Parallelism) splits the model across hundreds of chips.

How it works

Let us dissect collective communications, DDP mechanics, ZeRO stages, and 3D parallelism.


1. The Ring-AllReduce Algorithm (Baidu, 2017)

Diagram: centralised gradient averaging compared with ring-based averaging

How do P GPUs compute the average of their gradient tensors of size V without a central master bottleneck?

In Ring-AllReduce, the P GPUs are arranged in a logical unidirectional ring (0 \rightarrow 1 \rightarrow 2 \rightarrow \dots \rightarrow P-1 \rightarrow 0):

  1. Step 1: Scatter-Reduce (P - 1 steps):
    • Each GPU divides its gradient tensor into P equal chunks.
    • At each step, GPU i sends chunk k to GPU i+1 and receives chunk k-1 from GPU i-1, summing received data into its local chunk.
    • After P-1 steps, each GPU holds the exact global sum of one distinct chunk of the gradient tensor!
  2. Step 2: All-Gather (P - 1 steps):
    • Each GPU sends its fully reduced chunk around the ring.
    • After P-1 steps, every GPU holds the complete, globally reduced gradient tensor!

Bandwidth Optimality:

\text{Total Data Transferred per GPU} = 2 \times \left(\frac{P - 1}{P}\right) \times V

2. PyTorch DistributedDataParallel (DDP)

Diagram: how gradient communication overlaps with the backward pass

In DDP:


3. ZeRO Memory Sharding & PyTorch FSDP

Diagram: full model replication compared with sharded parameters and optimiser state

ZeRO memory sharding stages comparison showing Baseline DDP ZeRO 1 ZeRO 2 and ZeRO 3 Fully Sharded Data Parallel

In standard DDP, every GPU redundantly holds identical model parameters, gradients, and optimizer states (16 \text{ Bytes / Parameter}).

ZeRO (Zero Redundancy Optimizer) eliminates this redundancy in three progressive stages:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   ZERO / FSDP SHARDING STAGES                          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Stage        β”‚ Sharded Components     β”‚ Memory / Paramβ”‚ Comm Overhead  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Baseline DDP β”‚ None (Full Redundancy) β”‚ 16 Bytes      β”‚ 0% Baseline    β”‚
β”‚ ZeRO-1 (Pos) β”‚ Optimizer States Only  β”‚ 4 + 12/P Bytesβ”‚ 0% Overhead    β”‚
β”‚ ZeRO-2 (Pos+gβ”‚ Optimizer + Gradients  β”‚ 2 + 14/P Bytesβ”‚ 0% Overhead    β”‚
β”‚ ZeRO-3 / FSDPβ”‚ Optim + Grads + Params β”‚ 16/P Bytes    β”‚ +50% AllGather β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

How ZeRO-3 / FSDP Operates at Runtime:

  1. Forward Pass: Before executing layer L, the GPUs perform an All-Gather to temporarily reconstruct the full weights of layer L. The layer executes, and the full weights are immediately deleted from VRAM, keeping only the 1/P local shard!
  2. Backward Pass: Before computing gradients for layer L, another All-Gather reconstructs full weights. Gradients are computed, reduced across ranks via Reduce-Scatter, and full weights are deleted again!

4. 3D Parallelism: Megatron-LM Tensor & Pipeline Parallelism

Diagram: data, tensor and pipeline parallelism as three combined axes

For models exceeding 100B parameters, combining DP, TP, and PP yields 3D Parallelism:

A. Tensor Parallelism (TP / Megatron-LM):

B. Pipeline Parallelism (PP / 1F1B Schedule):


5. Sequence Parallelism & DeepSpeed Ulysses (Korthikanti et al., 2022)

In standard Tensor Parallelism, LayerNorm and Dropout activations are still duplicated across all TP ranks:


6. Context Parallelism & RingAttention (Liu et al., 2023)

Diagram: how attention is computed when the sequence is split across devices

How do modern models process 1,000,000+ token context windows without crashing GPU VRAM?


7. Expert Parallelism in Mixture-of-Experts (MoE / Lepikhin et al., GShard)

Diagram: how mixture-of-experts routing activates only part of the model per token

In sparse Mixture-of-Experts models (like Mixtral 8x7B or Grok-1):


8. Production Cluster Orchestration: torchrun and NCCL Tuning

To launch distributed jobs reliably on production clusters:


An everyday analogy

Diagram: dividing work by chapter compared with dividing by task

Think of translating an encyclopedic 1,000-page book from English to Spanish:


Examples in practice

Let us build a Simulated Ring-AllReduce Collective Communication Engine in Python:

import numpy as np
from typing import List

def ring_allreduce_simulation(rank_tensors: List[np.ndarray]) -> List[np.ndarray]:
    # Simulates the Ring-AllReduce collective communication algorithm across P ranks.
    P = len(rank_tensors)
    assert P > 1, "Ring-AllReduce requires at least 2 ranks."
    tensor_len = len(rank_tensors[0])
    assert tensor_len % P == 0, f"Tensor length {tensor_len} must be divisible by {P}"
    chunk_size = tensor_len // P

    # Make working copies: list of P chunks per rank
    ranks = [[t[i*chunk_size:(i+1)*chunk_size].copy() for i in range(P)] for t in rank_tensors]

    # 1. Scatter-Reduce Phase (P - 1 steps)
    for step in range(P - 1):
        for r in range(P):
            send_chunk_idx = (r - step) % P
            recv_chunk_idx = (r - step - 1) % P
            send_val = ranks[r][send_chunk_idx]
            recv_rank = (r + 1) % P
            # Add received chunk to next rank
            ranks[recv_rank][send_chunk_idx] += send_val

    # 2. All-Gather Phase (P - 1 steps)
    for step in range(P - 1):
        for r in range(P):
            send_chunk_idx = (r - step + 1) % P
            send_val = ranks[r][send_chunk_idx]
            recv_rank = (r + 1) % P
            ranks[recv_rank][send_chunk_idx] = send_val.copy()

    # Reassemble full tensors for each rank
    return [np.concatenate(r) for r in ranks]

Implications: security, privacy, performance, scalability, and cost

Diagram: the cost, failure, efficiency and complexity implications of distributed training

  1. Network Fault Tolerance & Stragglers:
    • In synchronous DDP, if a single GPU among 1,024 throttles due to heat or a degraded optical cable, all 1,023 other GPUs stall on the All-Reduce barrier. Continuous telemetry (NCCL health checks) is essential.
  2. Gradient Inversion & Differential Privacy:
    • In federated distributed training, transmitting unencrypted raw gradients can allow adversaries to reconstruct private training images or text via gradient inversion attacks.

Alternatives: free, open source, and commercial

Distributed FrameworkPrimary TechnologyScaling RangeBest Use Case
PyTorch DDPMulti-Process All-Reduce2 - 128 GPUsModels fitting on 1 GPU
PyTorch FSDPNative ZeRO-3 Sharding8 - 1,024 GPUsModels 7B - 70B params
DeepSpeed (ZeRO)ZeRO-1/2/3 + Offload8 - 4,096 GPUsTrillion-param training
Megatron-Core3D Parallelism (TP+PP+DP)64 - 100,000 GPUsFrontier LLM Pretraining

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   DISTRIBUTED PARADIGMS COMPARISON                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Paradigm           β”‚ What is Sharded β”‚ Interconnect Req β”‚ Comm Volume  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ DDP                β”‚ Data Batch      β”‚ Low (InfiniBand) β”‚ 2 * Grads    β”‚
β”‚ FSDP / ZeRO-3      β”‚ Data + Weights  β”‚ Moderate (RoCE)  β”‚ 3 * Weights  β”‚
β”‚ Tensor Parallel (TPβ”‚ Weight Matrices β”‚ Ultra-High(NVLinkβ”‚ 2x / Layer   β”‚
β”‚ Pipeline (PP)      β”‚ Network Layers  β”‚ Low (P2P Stream) β”‚ Activations  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when distributed training is warranted and when a single device is better

When to Use Distributed Training:

When NOT to use:


The AI connection

Knowledge check

  1. What is the fundamental difference between PyTorch DP and PyTorch DDP?
  2. How does Ring-AllReduce achieve constant per-GPU communication volume independent of cluster size?
  3. What are the differences between ZeRO-1, ZeRO-2, and ZeRO-3 memory sharding?
  4. Why is Tensor Parallelism typically restricted to an 8-GPU NVLink node rather than running across nodes?
  5. How does the 1F1B schedule reduce the pipeline bubble in Pipeline Parallelism?

Hands-on exercise

In this lab, you will build and test a modular distributed training simulation engine in Python: implement Ring-AllReduce collective communication mechanics, calculate ZeRO memory sharding budgets across cluster ranks, simulate gradient aggregation across distributed processes, and verify mathematical exactness.

Expected output

[Distributed Training Architecture Verification Suite]
Testing Ring-AllReduce Simulation (4 Ranks, Tensor Length=8):
  Rank 0 Initial: [1. 1. 1. 1. 1. 1. 1. 1.]
  Rank 1 Initial: [2. 2. 2. 2. 2. 2. 2. 2.]
  Rank 2 Initial: [3. 3. 3. 3. 3. 3. 3. 3.]
  Rank 3 Initial: [4. 4. 4. 4. 4. 4. 4. 4.]
  Expected Global Sum: [10. 10. 10. 10. 10. 10. 10. 10.]
  Ring-AllReduce Output Rank 0: [10. 10. 10. 10. 10. 10. 10. 10.] [EXACT MATCH]
Testing ZeRO Memory Sharding Calculator (70B Model on 64 GPUs):
  Standard DDP: 1120.00 GB per GPU (Out of Memory)
  ZeRO-3 / FSDP: 17.50 GB per GPU (Fits easily in 80GB H100) [VERIFIED]
Test Suite: 3 passed in 0.43s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a Pipeline Parallelism 1F1B Scheduler Simulator that computes the pipeline bubble percentage for varying micro-batch counts M and stages K.
  2. Implement Column-Parallel and Row-Parallel Linear Layers in PyTorch.

Extension challenge

Implement a Toy FSDP Wrapper in PyTorch:

Quiz

Q1. Why is PyTorch DistributedDataParallel (DDP) fundamentally superior to legacy DataParallel (DP)?

  1. DP is only for audio models
  2. DDP runs without electricity
  3. One process per GPU
  4. DDP deletes optimizer states
Show answer

Answer: C. One process per GPU

DDP eliminates Python GIL bottlenecks and master-GPU single-point congestion through multi-process ring all-reduce.

Q2. What is the total data transmitted per GPU in a Ring-AllReduce across P GPUs with tensor volume V?

  1. P * V^2
  2. P * V
  3. V^2
  4. 2(P-1)V/P
Show answer

Answer: D. 2(P-1)V/P

Ring-AllReduce requires two passes (Scatter-Reduce and All-Gather), transferring 2*(P-1)/P * V bytes per GPU.

Q3. What does ZeRO-3 (Zero Redundancy Optimizer Stage 3 / PyTorch FSDP) shard across GPUs?

  1. Optimizer, gradients, parameters
  2. It only shards text tokens
  3. It only shards the loss function
  4. It deletes the test set
Show answer

Answer: A. Optimizer, gradients, parameters

ZeRO-3 shards optimizer states, gradients, and model parameters, achieving near-linear memory reduction across ranks.

Q4. Why does Megatron-LM Tensor Parallelism (TP) require high-bandwidth NVLink interconnects rather than standard Ethernet networking?

  1. Because Ethernet does not support floating point numbers
  2. TP all-reduces inside every layer, so it needs NVLink
  3. Because NVLink is cheaper than Ethernet
  4. Because GPUs cannot plug into Ethernet cables
Show answer

Answer: B. TP all-reduces inside every layer, so it needs NVLink

TP communicates multiple times per Transformer layer; without NVLink wire speed, training stalls completely on network transfer.

Q5. What is the primary problem with naive Pipeline Parallelism, and how does the 1F1B (One-Forward-One-Backward) schedule solve it?

  1. 1F1B deletes the model weights
  2. Naive pipeline parallelism causes loss to become negative
  3. 1F1B interleaves passes so GPUs stay busy
  4. Naive pipeline parallelism requires liquid helium
Show answer

Answer: C. 1F1B interleaves passes so GPUs stay busy

1F1B schedule minimizes the pipeline bubble and stabilizes activation memory by executing 1 backward pass for every forward pass.

Glossary

DistributedDataParallel (DDP)
PyTorch multi-process data parallelism running one process per GPU and overlapping gradient all-reduce with backward pass computation.
Ring-AllReduce
A bandwidth-optimal collective communication algorithm where logical rings of processors aggregate tensors in Scatter-Reduce and All-Gather phases.
ZeRO (Zero Redundancy Optimizer)
A memory optimization technology that eliminates redundant memory states across data-parallel processes by sharding optimizer states, gradients, and parameters.
Fully Sharded Data Parallel (FSDP)
PyTorch native implementation of ZeRO-3, dynamically gathering parameter shards just-in-time and discarding them immediately after layer execution.
Tensor Parallelism (TP)
Intra-node model parallelism that splits individual weight matrices (e.g. QKV projections and MLP layers) across multiple GPUs within an NVLink domain.
Pipeline Parallelism (PP)
Inter-node model parallelism that partitions consecutive Transformer layers across multiple GPUs, executing micro-batches in a 1F1B pipeline schedule.
Pipeline Bubble
The idle time experienced by pipeline parallel stages waiting for data from preceding or succeeding stages.
3D Parallelism
The orthogonal combination of Data Parallelism (DP/FSDP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) to train trillion-parameter models.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.