Deep Learning › Training at Scale › Day 234

Hands-on lab — Day 234: Distributed Training Concepts

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/distributed_training_concepts_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/distributed_training_concepts_lib.py
examples/test_distributed_training_concepts_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/distributed_training_concepts_lib.py
starter/test_distributed_training_concepts_lib.py
tests/run_tests.sh
tests/test_distributed_training_concepts_lib.py
troubleshooting.md

Lab README

Lab: Day 234 -- Distributed Training Concepts

Lesson

Day number: 234 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Distributed Training Architectures, Ring-AllReduce, and ZeRO Memory Sharding.

Purpose

Build and test a modular distributed training simulation engine in Python. Implement Ring-AllReduce collective communication mechanics and calculate ZeRO memory sharding budgets across GPU cluster ranks.

Learning objectives

  • Implement the 2-phase Ring-AllReduce algorithm (Scatter-Reduce and All-Gather).
  • Calculate per-GPU VRAM memory requirements across ZeRO-1, ZeRO-2, and ZeRO-3 / FSDP stages.
  • Analyze collective communication bandwidth efficiency.
  • Verify exact mathematical aggregation across simulated multi-GPU processes.

Prerequisites

  • Day 233 (Mixed Precision and Performance).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/distributed_training_concepts_lib.py: Student scaffold file.
  • examples/distributed_training_concepts_lib.py: Complete reference implementation.
  • tests/test_distributed_training_concepts_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/distributed_training_concepts_lib.py

What the commands do

  • Simulates Ring-AllReduce across multi-rank tensor arrays.
  • Computes ZeRO memory partition budgets.
  • Runs unit test assertions.

Expected output

Distributed Demo: Reduced Sum = [3.0, 3.0, 3.0, 3.0], 70B on 64 GPUs ZeRO-3 = 17.5 GB

Validation steps

  1. Verify simulate_ring_allreduce outputs the exact sum across all ranks.
  2. Confirm ZeRO-3 divides total 16 bytes/param by cluster GPU count $P$.
  3. Confirm ZeRO-1 shards only the 12 bytes/param optimizer states.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Length mismatch in Ring-AllReduce: Ensure tensor array length is evenly divisible by number of ranks.

Security notes

All collective communication simulations execute locally in process memory.

Extension exercises

  1. Implement a Pipeline Parallelism 1F1B bubble simulator.
  2. Build a Tensor Parallel column/row linear layer in PyTorch.

Expected output

FIELDS.md

# Expected Output Fields: Day 234

- `Reduced Sum`: The global elementwise sum across all simulated ranks.
- `70B on 64 GPUs ZeRO-3`: VRAM required in GB per GPU for a 70B parameter model with ZeRO-3.

examples-run.txt

Distributed Demo: Reduced Sum = [3.0, 3.0, 3.0, 3.0], 70B on 64 GPUs ZeRO-3 = 17.5 GB

measured-values.txt

Reduced Sum: [3.0, 3.0, 3.0, 3.0]
70B on 64 GPUs ZeRO-3: 17.5 GB

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_distributed_training_concepts_lib.py::test_ring_allreduce_exactness PASSED [ 33%]
tests/test_distributed_training_concepts_lib.py::test_ring_allreduce_4_ranks PASSED [ 66%]
tests/test_distributed_training_concepts_lib.py::test_zero_memory_sharding PASSED [100%]

============================== 3 passed in 0.43s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/distributed_training_concepts_lib.py (2250 bytes)
import numpy as np
from typing import List, Dict, Any

def simulate_ring_allreduce(rank_tensors: List[np.ndarray]) -> List[np.ndarray]:
    P = len(rank_tensors)
    if P <= 1:
        return [t.copy() for t in rank_tensors]
    
    tensor_len = len(rank_tensors[0])
    chunk_size = tensor_len // P
    
    # Split into chunks per rank
    chunks = [[t[i*chunk_size:(i+1)*chunk_size].copy() for i in range(P)] for t in rank_tensors]
    
    # 1. Scatter-Reduce (P-1 steps)
    for step in range(P - 1):
        send_data = [chunks[r][(r - step) % P].copy() for r in range(P)]
        for r in range(P):
            recv_rank = (r + 1) % P
            recv_chunk_idx = (r - step) % P
            chunks[recv_rank][recv_chunk_idx] += send_data[r]
            
    # 2. All-Gather (P-1 steps)
    for step in range(P - 1):
        send_data = [chunks[r][(r - step + 1) % P].copy() for r in range(P)]
        for r in range(P):
            recv_rank = (r + 1) % P
            recv_chunk_idx = (r - step + 1) % P
            chunks[recv_rank][recv_chunk_idx] = send_data[r].copy()
            
    return [np.concatenate(chunks[r]) for r in range(P)]

def calculate_zero_sharding_memory(num_params_billions: float, num_gpus: int) -> Dict[str, float]:
    # In decimal GB (16 bytes per param static)
    # Params: 2B, Grads: 2B, Master: 4B, Mom1: 4B, Mom2: 4B -> Optim = 12B
    N = num_params_billions
    P = num_gpus

    ddp_gb = 16.0 * N
    zero1_gb = 4.0 * N + (12.0 * N) / P
    zero2_gb = 2.0 * N + (14.0 * N) / P
    zero3_gb = (16.0 * N) / P

    return {
        "num_params_billions": N,
        "num_gpus": P,
        "ddp_per_gpu_gb": round(ddp_gb, 2),
        "zero1_per_gpu_gb": round(zero1_gb, 2),
        "zero2_per_gpu_gb": round(zero2_gb, 2),
        "zero3_fsdp_per_gpu_gb": round(zero3_gb, 2)
    }

def run_distributed_demo():
    t0 = np.array([1.0, 1.0, 1.0, 1.0])
    t1 = np.array([2.0, 2.0, 2.0, 2.0])
    reduced = simulate_ring_allreduce([t0, t1])
    zero_mem = calculate_zero_sharding_memory(70.0, 64)

    print(f"Distributed Demo: Reduced Sum = {reduced[0].tolist()}, 70B on 64 GPUs ZeRO-3 = {zero_mem['zero3_fsdp_per_gpu_gb']} GB")
    return reduced, zero_mem

if __name__ == "__main__":
    run_distributed_demo()
examples/test_distributed_training_concepts_lib.py (982 bytes)
import pytest
import numpy as np
from examples.distributed_training_concepts_lib import simulate_ring_allreduce, calculate_zero_sharding_memory

def test_ring_allreduce_exactness():
    t0 = np.array([1.0, 2.0, 3.0, 4.0])
    t1 = np.array([10.0, 20.0, 30.0, 40.0])
    res = simulate_ring_allreduce([t0, t1])
    expected = np.array([11.0, 22.0, 33.0, 44.0])
    np.testing.assert_allclose(res[0], expected)
    np.testing.assert_allclose(res[1], expected)

def test_ring_allreduce_4_ranks():
    tensors = [np.full(8, fill_value=float(i + 1)) for i in range(4)]
    res = simulate_ring_allreduce(tensors)
    # Sum of 1 + 2 + 3 + 4 = 10
    expected = np.full(8, 10.0)
    for r in range(4):
        np.testing.assert_allclose(res[r], expected)

def test_zero_memory_sharding():
    res = calculate_zero_sharding_memory(70.0, 64)
    assert res["ddp_per_gpu_gb"] == 1120.0
    assert res["zero3_fsdp_per_gpu_gb"] == 17.5
    assert res["zero1_per_gpu_gb"] < res["ddp_per_gpu_gb"]
metadata.yml (442 bytes)
lesson_id: D234
day: 234
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/distributed_training_concepts_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/distributed_training_concepts_lib.py (413 bytes)
import numpy as np
from typing import List, Dict, Any

def simulate_ring_allreduce(rank_tensors: List[np.ndarray]) -> List[np.ndarray]:
    # TODO: Implement 2-phase Ring-AllReduce (Scatter-Reduce + All-Gather)
    pass

def calculate_zero_sharding_memory(num_params_billions: float, num_gpus: int) -> Dict[str, float]:
    # TODO: Calculate per-GPU VRAM requirements for DDP, ZeRO-1, ZeRO-2, and ZeRO-3
    pass
starter/test_distributed_training_concepts_lib.py (982 bytes)
import pytest
import numpy as np
from examples.distributed_training_concepts_lib import simulate_ring_allreduce, calculate_zero_sharding_memory

def test_ring_allreduce_exactness():
    t0 = np.array([1.0, 2.0, 3.0, 4.0])
    t1 = np.array([10.0, 20.0, 30.0, 40.0])
    res = simulate_ring_allreduce([t0, t1])
    expected = np.array([11.0, 22.0, 33.0, 44.0])
    np.testing.assert_allclose(res[0], expected)
    np.testing.assert_allclose(res[1], expected)

def test_ring_allreduce_4_ranks():
    tensors = [np.full(8, fill_value=float(i + 1)) for i in range(4)]
    res = simulate_ring_allreduce(tensors)
    # Sum of 1 + 2 + 3 + 4 = 10
    expected = np.full(8, 10.0)
    for r in range(4):
        np.testing.assert_allclose(res[r], expected)

def test_zero_memory_sharding():
    res = calculate_zero_sharding_memory(70.0, 64)
    assert res["ddp_per_gpu_gb"] == 1120.0
    assert res["zero3_fsdp_per_gpu_gb"] == 17.5
    assert res["zero1_per_gpu_gb"] < res["ddp_per_gpu_gb"]
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 234 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_distributed_training_concepts_lib.py (982 bytes)
import pytest
import numpy as np
from examples.distributed_training_concepts_lib import simulate_ring_allreduce, calculate_zero_sharding_memory

def test_ring_allreduce_exactness():
    t0 = np.array([1.0, 2.0, 3.0, 4.0])
    t1 = np.array([10.0, 20.0, 30.0, 40.0])
    res = simulate_ring_allreduce([t0, t1])
    expected = np.array([11.0, 22.0, 33.0, 44.0])
    np.testing.assert_allclose(res[0], expected)
    np.testing.assert_allclose(res[1], expected)

def test_ring_allreduce_4_ranks():
    tensors = [np.full(8, fill_value=float(i + 1)) for i in range(4)]
    res = simulate_ring_allreduce(tensors)
    # Sum of 1 + 2 + 3 + 4 = 10
    expected = np.full(8, 10.0)
    for r in range(4):
        np.testing.assert_allclose(res[r], expected)

def test_zero_memory_sharding():
    res = calculate_zero_sharding_memory(70.0, 64)
    assert res["ddp_per_gpu_gb"] == 1120.0
    assert res["zero3_fsdp_per_gpu_gb"] == 17.5
    assert res["zero1_per_gpu_gb"] < res["ddp_per_gpu_gb"]

Troubleshooting

Troubleshooting: Day 234 - Distributed Training Concepts

Common Issues

  1. Ring-AllReduce division error:
    • Cause: Number of ranks does not divide tensor length.
    • Fix: Pad tensor length to a multiple of rank count $P$.

Security notes

Security & Privacy: Day 234 - Distributed Training Concepts

Security Guidance

  • All distributed communication simulations execute locally in memory.