Deep Learning › Training at Scale › Day 237
Hands-on lab — Day 237: Scaling Laws and What They Bought Us
- ← Back to the Day 237 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-237-scaling-laws-and-what-they-bought/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/scaling_laws_and_what_they_bought_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/scaling_laws_and_what_they_bought_lib.py examples/test_scaling_laws_and_what_they_bought_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/scaling_laws_and_what_they_bought_lib.py starter/test_scaling_laws_and_what_they_bought_lib.py tests/run_tests.sh tests/test_scaling_laws_and_what_they_bought_lib.py troubleshooting.md
Lab README
Lab: Day 237 -- Scaling Laws and What They Bought Us
Lesson
Day number: 237 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Neural Scaling Laws, Compute Budgeting, and Chinchilla Optimization.
Purpose
Build and test a modular scaling law simulator and compute budget optimization engine in Python. Implement the $C = 6ND$ calculation formula, solve for Chinchilla compute-optimal parameter-token allocations, and compute predicted validation loss trajectories.
Learning objectives
- Calculate training compute FLOPs using the foundational $C \approx 6ND$ formula.
- Implement the Chinchilla compute-optimal resource allocation algorithm.
- Predict cross-entropy loss trajectories across compute scales using power-law constants.
- Analyze the economics of inference-optimal overtraining.
Prerequisites
- Day 236 (Quantization and Distillation).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard math libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/scaling_laws_and_what_they_bought_lib.py: Student scaffold file.examples/scaling_laws_and_what_they_bought_lib.py: Complete reference implementation.tests/test_scaling_laws_and_what_they_bought_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/scaling_laws_and_what_they_bought_lib.py
What the commands do
- Computes total training FLOPs for model-dataset combinations.
- Solves for Chinchilla optimal parameter and token counts.
- Runs unit test assertions.
Expected output
Scaling Laws Demo: Compute = 5.88e+21 FLOPs, Optimal Params = 7.0B, Tokens = 140.0B
Validation steps
- Verify
calculate_training_flopsmultiplies $6 \times N \times D$. - Confirm
compute_chinchilla_optimalenforces $D = 20N$. - Confirm predicted loss exceeds irreducible language entropy $E = 1.69$.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- FLOP calculation error: Ensure parameters and tokens are scaled to raw units ($10^9$).
Security notes
All scaling calculations execute locally in process memory.
Extension exercises
- Plot IsoFLOP loss contours using Matplotlib.
- Build an inference lifetime cost comparison tool.
Navigation
- Lesson title: Scaling Laws and What They Bought Us
- Day number: 237 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-237-scaling-laws-and-what-they-bought
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-237-scaling-laws-and-what-they-boughtwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 237
- `Compute`: Total training compute in floating-point operations (FLOPs).
- `Optimal Params`: Chinchilla optimal model size in billions of parameters.
- `Tokens`: Chinchilla optimal dataset size in billions of tokens.
examples-run.txt
Scaling Laws Demo: Compute = 5.88e+21 FLOPs, Optimal Params = 7.0B, Tokens = 140.0B
measured-values.txt
Compute: 5.88e+21 FLOPs
Optimal Params: 7.0B
Tokens: 140.0B
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_scaling_laws_and_what_they_bought_lib.py::test_training_flops_calculation PASSED [ 33%]
tests/test_scaling_laws_and_what_they_bought_lib.py::test_chinchilla_optimal_reconstruction PASSED [ 66%]
tests/test_scaling_laws_and_what_they_bought_lib.py::test_predicted_loss_bounds PASSED [100%]
============================== 3 passed in 0.41s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/scaling_laws_and_what_they_bought_lib.py (1239 bytes)
import math
from typing import Dict, Any
def calculate_training_flops(params_billions: float, tokens_billions: float) -> float:
N = params_billions * 1e9
D = tokens_billions * 1e9
return 6.0 * N * D
def compute_chinchilla_optimal(compute_flops: float) -> Dict[str, Any]:
# C = 6 * N * D = 6 * N * (20 * N) = 120 * N^2 -> N = sqrt(C / 120)
N_opt = math.sqrt(compute_flops / 120.0)
D_opt = 20.0 * N_opt
# Chinchilla Loss constants
E = 1.69
A, alpha = 406.4, 0.34
B, beta = 410.7, 0.28
pred_loss = E + (A / (N_opt ** alpha)) + (B / (D_opt ** beta))
return {
"compute_flops": compute_flops,
"optimal_params_billions": round(N_opt / 1e9, 2),
"optimal_tokens_billions": round(D_opt / 1e9, 2),
"predicted_loss": round(pred_loss, 4),
"token_to_param_ratio": round(D_opt / N_opt, 1)
}
def run_scaling_laws_demo():
flops = calculate_training_flops(7.0, 140.0)
alloc = compute_chinchilla_optimal(flops)
print(f"Scaling Laws Demo: Compute = {flops:.2e} FLOPs, Optimal Params = {alloc['optimal_params_billions']}B, Tokens = {alloc['optimal_tokens_billions']}B")
return flops, alloc
if __name__ == "__main__":
run_scaling_laws_demo()
examples/test_scaling_laws_and_what_they_bought_lib.py (907 bytes)
import pytest
from examples.scaling_laws_and_what_they_bought_lib import (
calculate_training_flops,
compute_chinchilla_optimal
)
def test_training_flops_calculation():
# 7B params on 140B tokens: 6 * 7e9 * 140e9 = 5.88e21 FLOPs
flops = calculate_training_flops(7.0, 140.0)
assert flops == pytest.approx(5.88e21, rel=1e-3)
def test_chinchilla_optimal_reconstruction():
flops = 5.88e21
alloc = compute_chinchilla_optimal(flops)
assert alloc["optimal_params_billions"] == pytest.approx(7.0, rel=1e-2)
assert alloc["optimal_tokens_billions"] == pytest.approx(140.0, rel=1e-2)
assert alloc["token_to_param_ratio"] == 20.0
def test_predicted_loss_bounds():
flops = 5.88e21
alloc = compute_chinchilla_optimal(flops)
# Loss should be greater than irreducible entropy E (1.69)
assert alloc["predicted_loss"] > 1.69
assert alloc["predicted_loss"] < 4.0
metadata.yml (446 bytes)
lesson_id: D237
day: 237
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/scaling_laws_and_what_they_bought_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/scaling_laws_and_what_they_bought_lib.py (339 bytes)
import math
from typing import Dict, Any
def calculate_training_flops(params_billions: float, tokens_billions: float) -> float:
# TODO: Calculate training FLOPs (C = 6 * N * D)
pass
def compute_chinchilla_optimal(compute_flops: float) -> Dict[str, Any]:
# TODO: Calculate optimal N and D using Chinchilla 20:1 ratio
pass
starter/test_scaling_laws_and_what_they_bought_lib.py (907 bytes)
import pytest
from examples.scaling_laws_and_what_they_bought_lib import (
calculate_training_flops,
compute_chinchilla_optimal
)
def test_training_flops_calculation():
# 7B params on 140B tokens: 6 * 7e9 * 140e9 = 5.88e21 FLOPs
flops = calculate_training_flops(7.0, 140.0)
assert flops == pytest.approx(5.88e21, rel=1e-3)
def test_chinchilla_optimal_reconstruction():
flops = 5.88e21
alloc = compute_chinchilla_optimal(flops)
assert alloc["optimal_params_billions"] == pytest.approx(7.0, rel=1e-2)
assert alloc["optimal_tokens_billions"] == pytest.approx(140.0, rel=1e-2)
assert alloc["token_to_param_ratio"] == 20.0
def test_predicted_loss_bounds():
flops = 5.88e21
alloc = compute_chinchilla_optimal(flops)
# Loss should be greater than irreducible entropy E (1.69)
assert alloc["predicted_loss"] > 1.69
assert alloc["predicted_loss"] < 4.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 237 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_scaling_laws_and_what_they_bought_lib.py (907 bytes)
import pytest
from examples.scaling_laws_and_what_they_bought_lib import (
calculate_training_flops,
compute_chinchilla_optimal
)
def test_training_flops_calculation():
# 7B params on 140B tokens: 6 * 7e9 * 140e9 = 5.88e21 FLOPs
flops = calculate_training_flops(7.0, 140.0)
assert flops == pytest.approx(5.88e21, rel=1e-3)
def test_chinchilla_optimal_reconstruction():
flops = 5.88e21
alloc = compute_chinchilla_optimal(flops)
assert alloc["optimal_params_billions"] == pytest.approx(7.0, rel=1e-2)
assert alloc["optimal_tokens_billions"] == pytest.approx(140.0, rel=1e-2)
assert alloc["token_to_param_ratio"] == 20.0
def test_predicted_loss_bounds():
flops = 5.88e21
alloc = compute_chinchilla_optimal(flops)
# Loss should be greater than irreducible entropy E (1.69)
assert alloc["predicted_loss"] > 1.69
assert alloc["predicted_loss"] < 4.0
Troubleshooting
Troubleshooting: Day 237 - Scaling Laws and What They Bought Us
Common Issues
- Exponentiation overflow in power law:
- Cause: Passing parameters in billions directly to power function.
- Fix: Use raw parameter counts $N$ and $D$.
Security notes
Security & Privacy: Day 237 - Scaling Laws and What They Bought Us
Security Guidance
- All scaling law computations execute locally in memory.