Deep Learning › Training at Scale › Day 234
Hands-on lab — Day 234: Distributed Training Concepts
- ← Back to the Day 234 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-234-distributed-training-concepts/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/distributed_training_concepts_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/distributed_training_concepts_lib.py examples/test_distributed_training_concepts_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/distributed_training_concepts_lib.py starter/test_distributed_training_concepts_lib.py tests/run_tests.sh tests/test_distributed_training_concepts_lib.py troubleshooting.md
Lab README
Lab: Day 234 -- Distributed Training Concepts
Lesson
Day number: 234 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Distributed Training Architectures, Ring-AllReduce, and ZeRO Memory Sharding.
Purpose
Build and test a modular distributed training simulation engine in Python. Implement Ring-AllReduce collective communication mechanics and calculate ZeRO memory sharding budgets across GPU cluster ranks.
Learning objectives
- Implement the 2-phase Ring-AllReduce algorithm (Scatter-Reduce and All-Gather).
- Calculate per-GPU VRAM memory requirements across ZeRO-1, ZeRO-2, and ZeRO-3 / FSDP stages.
- Analyze collective communication bandwidth efficiency.
- Verify exact mathematical aggregation across simulated multi-GPU processes.
Prerequisites
- Day 233 (Mixed Precision and Performance).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/distributed_training_concepts_lib.py: Student scaffold file.examples/distributed_training_concepts_lib.py: Complete reference implementation.tests/test_distributed_training_concepts_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/distributed_training_concepts_lib.py
What the commands do
- Simulates Ring-AllReduce across multi-rank tensor arrays.
- Computes ZeRO memory partition budgets.
- Runs unit test assertions.
Expected output
Distributed Demo: Reduced Sum = [3.0, 3.0, 3.0, 3.0], 70B on 64 GPUs ZeRO-3 = 17.5 GB
Validation steps
- Verify
simulate_ring_allreduceoutputs the exact sum across all ranks. - Confirm ZeRO-3 divides total 16 bytes/param by cluster GPU count $P$.
- Confirm ZeRO-1 shards only the 12 bytes/param optimizer states.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Length mismatch in Ring-AllReduce: Ensure tensor array length is evenly divisible by number of ranks.
Security notes
All collective communication simulations execute locally in process memory.
Extension exercises
- Implement a Pipeline Parallelism 1F1B bubble simulator.
- Build a Tensor Parallel column/row linear layer in PyTorch.
Navigation
- Lesson title: Distributed Training Concepts
- Day number: 234 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-234-distributed-training-concepts
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-234-distributed-training-conceptswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 234
- `Reduced Sum`: The global elementwise sum across all simulated ranks.
- `70B on 64 GPUs ZeRO-3`: VRAM required in GB per GPU for a 70B parameter model with ZeRO-3.
examples-run.txt
Distributed Demo: Reduced Sum = [3.0, 3.0, 3.0, 3.0], 70B on 64 GPUs ZeRO-3 = 17.5 GB
measured-values.txt
Reduced Sum: [3.0, 3.0, 3.0, 3.0]
70B on 64 GPUs ZeRO-3: 17.5 GB
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_distributed_training_concepts_lib.py::test_ring_allreduce_exactness PASSED [ 33%]
tests/test_distributed_training_concepts_lib.py::test_ring_allreduce_4_ranks PASSED [ 66%]
tests/test_distributed_training_concepts_lib.py::test_zero_memory_sharding PASSED [100%]
============================== 3 passed in 0.43s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/distributed_training_concepts_lib.py (2250 bytes)
import numpy as np
from typing import List, Dict, Any
def simulate_ring_allreduce(rank_tensors: List[np.ndarray]) -> List[np.ndarray]:
P = len(rank_tensors)
if P <= 1:
return [t.copy() for t in rank_tensors]
tensor_len = len(rank_tensors[0])
chunk_size = tensor_len // P
# Split into chunks per rank
chunks = [[t[i*chunk_size:(i+1)*chunk_size].copy() for i in range(P)] for t in rank_tensors]
# 1. Scatter-Reduce (P-1 steps)
for step in range(P - 1):
send_data = [chunks[r][(r - step) % P].copy() for r in range(P)]
for r in range(P):
recv_rank = (r + 1) % P
recv_chunk_idx = (r - step) % P
chunks[recv_rank][recv_chunk_idx] += send_data[r]
# 2. All-Gather (P-1 steps)
for step in range(P - 1):
send_data = [chunks[r][(r - step + 1) % P].copy() for r in range(P)]
for r in range(P):
recv_rank = (r + 1) % P
recv_chunk_idx = (r - step + 1) % P
chunks[recv_rank][recv_chunk_idx] = send_data[r].copy()
return [np.concatenate(chunks[r]) for r in range(P)]
def calculate_zero_sharding_memory(num_params_billions: float, num_gpus: int) -> Dict[str, float]:
# In decimal GB (16 bytes per param static)
# Params: 2B, Grads: 2B, Master: 4B, Mom1: 4B, Mom2: 4B -> Optim = 12B
N = num_params_billions
P = num_gpus
ddp_gb = 16.0 * N
zero1_gb = 4.0 * N + (12.0 * N) / P
zero2_gb = 2.0 * N + (14.0 * N) / P
zero3_gb = (16.0 * N) / P
return {
"num_params_billions": N,
"num_gpus": P,
"ddp_per_gpu_gb": round(ddp_gb, 2),
"zero1_per_gpu_gb": round(zero1_gb, 2),
"zero2_per_gpu_gb": round(zero2_gb, 2),
"zero3_fsdp_per_gpu_gb": round(zero3_gb, 2)
}
def run_distributed_demo():
t0 = np.array([1.0, 1.0, 1.0, 1.0])
t1 = np.array([2.0, 2.0, 2.0, 2.0])
reduced = simulate_ring_allreduce([t0, t1])
zero_mem = calculate_zero_sharding_memory(70.0, 64)
print(f"Distributed Demo: Reduced Sum = {reduced[0].tolist()}, 70B on 64 GPUs ZeRO-3 = {zero_mem['zero3_fsdp_per_gpu_gb']} GB")
return reduced, zero_mem
if __name__ == "__main__":
run_distributed_demo()
examples/test_distributed_training_concepts_lib.py (982 bytes)
import pytest
import numpy as np
from examples.distributed_training_concepts_lib import simulate_ring_allreduce, calculate_zero_sharding_memory
def test_ring_allreduce_exactness():
t0 = np.array([1.0, 2.0, 3.0, 4.0])
t1 = np.array([10.0, 20.0, 30.0, 40.0])
res = simulate_ring_allreduce([t0, t1])
expected = np.array([11.0, 22.0, 33.0, 44.0])
np.testing.assert_allclose(res[0], expected)
np.testing.assert_allclose(res[1], expected)
def test_ring_allreduce_4_ranks():
tensors = [np.full(8, fill_value=float(i + 1)) for i in range(4)]
res = simulate_ring_allreduce(tensors)
# Sum of 1 + 2 + 3 + 4 = 10
expected = np.full(8, 10.0)
for r in range(4):
np.testing.assert_allclose(res[r], expected)
def test_zero_memory_sharding():
res = calculate_zero_sharding_memory(70.0, 64)
assert res["ddp_per_gpu_gb"] == 1120.0
assert res["zero3_fsdp_per_gpu_gb"] == 17.5
assert res["zero1_per_gpu_gb"] < res["ddp_per_gpu_gb"]
metadata.yml (442 bytes)
lesson_id: D234
day: 234
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/distributed_training_concepts_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/distributed_training_concepts_lib.py (413 bytes)
import numpy as np
from typing import List, Dict, Any
def simulate_ring_allreduce(rank_tensors: List[np.ndarray]) -> List[np.ndarray]:
# TODO: Implement 2-phase Ring-AllReduce (Scatter-Reduce + All-Gather)
pass
def calculate_zero_sharding_memory(num_params_billions: float, num_gpus: int) -> Dict[str, float]:
# TODO: Calculate per-GPU VRAM requirements for DDP, ZeRO-1, ZeRO-2, and ZeRO-3
pass
starter/test_distributed_training_concepts_lib.py (982 bytes)
import pytest
import numpy as np
from examples.distributed_training_concepts_lib import simulate_ring_allreduce, calculate_zero_sharding_memory
def test_ring_allreduce_exactness():
t0 = np.array([1.0, 2.0, 3.0, 4.0])
t1 = np.array([10.0, 20.0, 30.0, 40.0])
res = simulate_ring_allreduce([t0, t1])
expected = np.array([11.0, 22.0, 33.0, 44.0])
np.testing.assert_allclose(res[0], expected)
np.testing.assert_allclose(res[1], expected)
def test_ring_allreduce_4_ranks():
tensors = [np.full(8, fill_value=float(i + 1)) for i in range(4)]
res = simulate_ring_allreduce(tensors)
# Sum of 1 + 2 + 3 + 4 = 10
expected = np.full(8, 10.0)
for r in range(4):
np.testing.assert_allclose(res[r], expected)
def test_zero_memory_sharding():
res = calculate_zero_sharding_memory(70.0, 64)
assert res["ddp_per_gpu_gb"] == 1120.0
assert res["zero3_fsdp_per_gpu_gb"] == 17.5
assert res["zero1_per_gpu_gb"] < res["ddp_per_gpu_gb"]
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 234 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_distributed_training_concepts_lib.py (982 bytes)
import pytest
import numpy as np
from examples.distributed_training_concepts_lib import simulate_ring_allreduce, calculate_zero_sharding_memory
def test_ring_allreduce_exactness():
t0 = np.array([1.0, 2.0, 3.0, 4.0])
t1 = np.array([10.0, 20.0, 30.0, 40.0])
res = simulate_ring_allreduce([t0, t1])
expected = np.array([11.0, 22.0, 33.0, 44.0])
np.testing.assert_allclose(res[0], expected)
np.testing.assert_allclose(res[1], expected)
def test_ring_allreduce_4_ranks():
tensors = [np.full(8, fill_value=float(i + 1)) for i in range(4)]
res = simulate_ring_allreduce(tensors)
# Sum of 1 + 2 + 3 + 4 = 10
expected = np.full(8, 10.0)
for r in range(4):
np.testing.assert_allclose(res[r], expected)
def test_zero_memory_sharding():
res = calculate_zero_sharding_memory(70.0, 64)
assert res["ddp_per_gpu_gb"] == 1120.0
assert res["zero3_fsdp_per_gpu_gb"] == 17.5
assert res["zero1_per_gpu_gb"] < res["ddp_per_gpu_gb"]
Troubleshooting
Troubleshooting: Day 234 - Distributed Training Concepts
Common Issues
- Ring-AllReduce division error:
- Cause: Number of ranks does not divide tensor length.
- Fix: Pad tensor length to a multiple of rank count $P$.
Security notes
Security & Privacy: Day 234 - Distributed Training Concepts
Security Guidance
- All distributed communication simulations execute locally in memory.