Deep Learning › Training at Scale › Day 237

Hands-on lab — Day 237: Scaling Laws and What They Bought Us

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/scaling_laws_and_what_they_bought_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/scaling_laws_and_what_they_bought_lib.py
examples/test_scaling_laws_and_what_they_bought_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/scaling_laws_and_what_they_bought_lib.py
starter/test_scaling_laws_and_what_they_bought_lib.py
tests/run_tests.sh
tests/test_scaling_laws_and_what_they_bought_lib.py
troubleshooting.md

Lab README

Lab: Day 237 -- Scaling Laws and What They Bought Us

Lesson

Day number: 237 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Neural Scaling Laws, Compute Budgeting, and Chinchilla Optimization.

Purpose

Build and test a modular scaling law simulator and compute budget optimization engine in Python. Implement the $C = 6ND$ calculation formula, solve for Chinchilla compute-optimal parameter-token allocations, and compute predicted validation loss trajectories.

Learning objectives

  • Calculate training compute FLOPs using the foundational $C \approx 6ND$ formula.
  • Implement the Chinchilla compute-optimal resource allocation algorithm.
  • Predict cross-entropy loss trajectories across compute scales using power-law constants.
  • Analyze the economics of inference-optimal overtraining.

Prerequisites

  • Day 236 (Quantization and Distillation).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard math libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/scaling_laws_and_what_they_bought_lib.py: Student scaffold file.
  • examples/scaling_laws_and_what_they_bought_lib.py: Complete reference implementation.
  • tests/test_scaling_laws_and_what_they_bought_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/scaling_laws_and_what_they_bought_lib.py

What the commands do

  • Computes total training FLOPs for model-dataset combinations.
  • Solves for Chinchilla optimal parameter and token counts.
  • Runs unit test assertions.

Expected output

Scaling Laws Demo: Compute = 5.88e+21 FLOPs, Optimal Params = 7.0B, Tokens = 140.0B

Validation steps

  1. Verify calculate_training_flops multiplies $6 \times N \times D$.
  2. Confirm compute_chinchilla_optimal enforces $D = 20N$.
  3. Confirm predicted loss exceeds irreducible language entropy $E = 1.69$.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • FLOP calculation error: Ensure parameters and tokens are scaled to raw units ($10^9$).

Security notes

All scaling calculations execute locally in process memory.

Extension exercises

  1. Plot IsoFLOP loss contours using Matplotlib.
  2. Build an inference lifetime cost comparison tool.

Expected output

FIELDS.md

# Expected Output Fields: Day 237

- `Compute`: Total training compute in floating-point operations (FLOPs).
- `Optimal Params`: Chinchilla optimal model size in billions of parameters.
- `Tokens`: Chinchilla optimal dataset size in billions of tokens.

examples-run.txt

Scaling Laws Demo: Compute = 5.88e+21 FLOPs, Optimal Params = 7.0B, Tokens = 140.0B

measured-values.txt

Compute: 5.88e+21 FLOPs
Optimal Params: 7.0B
Tokens: 140.0B

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_scaling_laws_and_what_they_bought_lib.py::test_training_flops_calculation PASSED [ 33%]
tests/test_scaling_laws_and_what_they_bought_lib.py::test_chinchilla_optimal_reconstruction PASSED [ 66%]
tests/test_scaling_laws_and_what_they_bought_lib.py::test_predicted_loss_bounds PASSED [100%]

============================== 3 passed in 0.41s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/scaling_laws_and_what_they_bought_lib.py (1239 bytes)
import math
from typing import Dict, Any

def calculate_training_flops(params_billions: float, tokens_billions: float) -> float:
    N = params_billions * 1e9
    D = tokens_billions * 1e9
    return 6.0 * N * D

def compute_chinchilla_optimal(compute_flops: float) -> Dict[str, Any]:
    # C = 6 * N * D = 6 * N * (20 * N) = 120 * N^2 -> N = sqrt(C / 120)
    N_opt = math.sqrt(compute_flops / 120.0)
    D_opt = 20.0 * N_opt

    # Chinchilla Loss constants
    E = 1.69
    A, alpha = 406.4, 0.34
    B, beta = 410.7, 0.28
    pred_loss = E + (A / (N_opt ** alpha)) + (B / (D_opt ** beta))

    return {
        "compute_flops": compute_flops,
        "optimal_params_billions": round(N_opt / 1e9, 2),
        "optimal_tokens_billions": round(D_opt / 1e9, 2),
        "predicted_loss": round(pred_loss, 4),
        "token_to_param_ratio": round(D_opt / N_opt, 1)
    }

def run_scaling_laws_demo():
    flops = calculate_training_flops(7.0, 140.0)
    alloc = compute_chinchilla_optimal(flops)

    print(f"Scaling Laws Demo: Compute = {flops:.2e} FLOPs, Optimal Params = {alloc['optimal_params_billions']}B, Tokens = {alloc['optimal_tokens_billions']}B")
    return flops, alloc

if __name__ == "__main__":
    run_scaling_laws_demo()
examples/test_scaling_laws_and_what_they_bought_lib.py (907 bytes)
import pytest
from examples.scaling_laws_and_what_they_bought_lib import (
    calculate_training_flops,
    compute_chinchilla_optimal
)

def test_training_flops_calculation():
    # 7B params on 140B tokens: 6 * 7e9 * 140e9 = 5.88e21 FLOPs
    flops = calculate_training_flops(7.0, 140.0)
    assert flops == pytest.approx(5.88e21, rel=1e-3)

def test_chinchilla_optimal_reconstruction():
    flops = 5.88e21
    alloc = compute_chinchilla_optimal(flops)
    assert alloc["optimal_params_billions"] == pytest.approx(7.0, rel=1e-2)
    assert alloc["optimal_tokens_billions"] == pytest.approx(140.0, rel=1e-2)
    assert alloc["token_to_param_ratio"] == 20.0

def test_predicted_loss_bounds():
    flops = 5.88e21
    alloc = compute_chinchilla_optimal(flops)
    # Loss should be greater than irreducible entropy E (1.69)
    assert alloc["predicted_loss"] > 1.69
    assert alloc["predicted_loss"] < 4.0
metadata.yml (446 bytes)
lesson_id: D237
day: 237
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/scaling_laws_and_what_they_bought_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/scaling_laws_and_what_they_bought_lib.py (339 bytes)
import math
from typing import Dict, Any

def calculate_training_flops(params_billions: float, tokens_billions: float) -> float:
    # TODO: Calculate training FLOPs (C = 6 * N * D)
    pass

def compute_chinchilla_optimal(compute_flops: float) -> Dict[str, Any]:
    # TODO: Calculate optimal N and D using Chinchilla 20:1 ratio
    pass
starter/test_scaling_laws_and_what_they_bought_lib.py (907 bytes)
import pytest
from examples.scaling_laws_and_what_they_bought_lib import (
    calculate_training_flops,
    compute_chinchilla_optimal
)

def test_training_flops_calculation():
    # 7B params on 140B tokens: 6 * 7e9 * 140e9 = 5.88e21 FLOPs
    flops = calculate_training_flops(7.0, 140.0)
    assert flops == pytest.approx(5.88e21, rel=1e-3)

def test_chinchilla_optimal_reconstruction():
    flops = 5.88e21
    alloc = compute_chinchilla_optimal(flops)
    assert alloc["optimal_params_billions"] == pytest.approx(7.0, rel=1e-2)
    assert alloc["optimal_tokens_billions"] == pytest.approx(140.0, rel=1e-2)
    assert alloc["token_to_param_ratio"] == 20.0

def test_predicted_loss_bounds():
    flops = 5.88e21
    alloc = compute_chinchilla_optimal(flops)
    # Loss should be greater than irreducible entropy E (1.69)
    assert alloc["predicted_loss"] > 1.69
    assert alloc["predicted_loss"] < 4.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 237 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_scaling_laws_and_what_they_bought_lib.py (907 bytes)
import pytest
from examples.scaling_laws_and_what_they_bought_lib import (
    calculate_training_flops,
    compute_chinchilla_optimal
)

def test_training_flops_calculation():
    # 7B params on 140B tokens: 6 * 7e9 * 140e9 = 5.88e21 FLOPs
    flops = calculate_training_flops(7.0, 140.0)
    assert flops == pytest.approx(5.88e21, rel=1e-3)

def test_chinchilla_optimal_reconstruction():
    flops = 5.88e21
    alloc = compute_chinchilla_optimal(flops)
    assert alloc["optimal_params_billions"] == pytest.approx(7.0, rel=1e-2)
    assert alloc["optimal_tokens_billions"] == pytest.approx(140.0, rel=1e-2)
    assert alloc["token_to_param_ratio"] == 20.0

def test_predicted_loss_bounds():
    flops = 5.88e21
    alloc = compute_chinchilla_optimal(flops)
    # Loss should be greater than irreducible entropy E (1.69)
    assert alloc["predicted_loss"] > 1.69
    assert alloc["predicted_loss"] < 4.0

Troubleshooting

Troubleshooting: Day 237 - Scaling Laws and What They Bought Us

Common Issues

  1. Exponentiation overflow in power law:
    • Cause: Passing parameters in billions directly to power function.
    • Fix: Use raw parameter counts $N$ and $D$.

Security notes

Security & Privacy: Day 237 - Scaling Laws and What They Bought Us

Security Guidance

  • All scaling law computations execute locally in memory.