Deep Learning › Training at Scale › Day 237

Day 237: Scaling Laws and What They Bought Us

Day 237 of 365 — Scaling Laws and What They Bought Us

Master empirical Neural Scaling Laws and compute budgeting: analyze the power-law formulation L(N, D) = E + A/N^alpha + B/D^beta, calculate training FLOPs (C ≈ 6ND), compare OpenAI Kaplan et al. with DeepMind Chinchilla (Hoffmann et al.), derive the compute-optimal 20 tokens-per-parameter ratio, analyze inference-optimal overtraining (LLaMA), understand test-time compute scaling (reasoning models), and build a complete scaling law simulation in Python.

Course
Deep Learning
Category
Training at Scale
Reading time
≈ 35 min
Practical time
≈ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-237-scaling-laws-and-what-they-bought

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/deep-learning/day-237-scaling-laws-and-what-they-bought
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Before 2020, training a 100\text{ Million} dollar AI model would have been considered financial insanity.

How could a laboratory risk 50\text{ Million} dollars of cloud compute without knowing whether the resulting model would work, hallucinate wildly, or fail to learn?

The breakthrough that unlocked hundreds of billions of dollars in AI investment was the discovery of Empirical Neural Scaling Laws:

Today, you will master the mathematics of the Kaplan vs. Chinchilla scaling laws, the foundational C \approx 6ND compute rule, inference-optimal overtraining (LLaMA), and the new frontier of Test-Time Reasoning Scaling.


The idea in plain language

Diagram: how scaling laws predict the performance of a large training run

Think of civil aerospace engineering:

Scaling laws are the Reynolds numbers of artificial intelligence.


Historical background

Diagram: the historical progression of neural scaling law research

Neural Scaling Laws comparison plotting cross-entropy loss against compute FLOPs comparing Kaplan and Chinchilla

  1. 2020 (Kaplan et al. & OpenAI): Published Scaling Laws for Neural Language Models, proving power-law scaling over 6 orders of magnitude in compute. However, they concluded that parameters should scale 5.5\times faster than tokens.
  2. 2020 (OpenAI & GPT-3): Based on Kaplan’s laws, OpenAI trained GPT-3 (175 Billion parameters on only 300 Billion tokens).
  3. 2022 (Hoffmann et al. & DeepMind Chinchilla): Re-evaluated scaling with IsoFLOP profiles, proving Kaplan was wrong: parameters and tokens should scale in equal 1:1 proportion (D \approx 20N). Chinchilla (70B on 1.4T tokens) outperformed the 280B Gopher model using identical compute!
  4. 2023 (Touvron et al. & Meta LLaMA): Pioneered Inference-Optimal Overtraining, training small 7B/13B models on 1.4T to 2T tokens to maximize edge serving efficiency.
  5. 2024–Present (OpenAI o1/o3 & Test-Time Compute): Discovered that scaling inference reasoning tokens (Chain-of-Thought and search) produces an entirely new power-law scaling curve.

What it is — and what it is not

What Neural Scaling Laws ARE:

What it is NOT:


Why it was created and what problems it solves

Scaling laws transformed machine learning from a craft into an exact predictive science:

  1. Capital Risk Elimination: Eliminates the risk of catastrophic multi-million dollar training failures.
  2. Optimal Hardware Sizing: Prevents training models that are too large on too little data (GPT-3/Gopher) or too small on excess compute.
  3. Architecture Comparison: Allows comparing architectures (e.g. Mamba vs Transformer vs MoE) at small scales and knowing with certainty which will win at trillion-parameter scales.

How it works

Let us dissect the mathematical formulation of scaling laws, compute budget derivation, and inference economics.


1. The Power-Law Formulation (Hoffmann et al., 2022)

The cross-entropy loss L is modeled as a function of parameter count N (non-embedding parameters) and dataset token count D:

L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}
┌────────────────────────────────────────────────────────────────────────┐
│                   SCALING LAW PARAMETER DEFINITIONS                    │
├──────────────┬────────────────────────┬────────────────────────────────┤
│ Symbol       │ DeepMind Estimate      │ Physical Interpretation        │
├──────────────┼────────────────────────┼────────────────────────────────┤
│ E            │ 1.69                   │ Irreducible Loss of Language   │
│ A            │ 406.4                  │ Model Size Scaling Constant    │
│ alpha (α)    │ 0.34                   │ Model Size Power-Law Exponent  │
│ B            │ 410.7                  │ Data Size Scaling Constant     │
│ beta (β)     │ 0.28                   │ Data Size Power-Law Exponent   │
└──────────────┴────────────────────────┴────────────────────────────────┤

2. The C \approx 6ND Compute Derivation

Compute optimal token and parameter allocation curves showing parameter count N and token count D as function of total FLOPs

How many floating point operations (FLOPs) are required to train a Transformer?

For an autoregressive Transformer with N parameters trained on D tokens:

  1. Forward Pass: Computing GEMMs (y = x W) requires 2 FLOPs per parameter per token (1 multiply + 1 addition).
  2. Backward Pass:
    • Computing gradient with respect to inputs \frac{\partial \mathcal{L}}{\partial x}: 2 FLOPs per parameter per token.
    • Computing gradient with respect to weights \frac{\partial \mathcal{L}}{\partial W}: 2 FLOPs per parameter per token.
    • Total Backward Pass: 4 FLOPs per parameter per token.
\text{Total Training Compute (FLOPs)}: \quad C \approx 2ND + 4ND = 6ND

(When using activation checkpointing, recomputing forward activations adds another 2ND, bringing total compute to C \approx 8ND.)


3. Compute-Optimal Allocation (The Chinchilla Rule)

Diagram: how a fixed compute budget is divided between model size and training data

To minimize loss L(N, D) subject to a fixed compute budget C = 6ND, we solve the constrained optimization problem via Lagrange multipliers:

N_{\text{opt}} \propto C^a \quad \text{and} \quad D_{\text{opt}} \propto C^b

Because \alpha \approx 0.34 and \beta \approx 0.28 are close in value:

a = \frac{\beta}{\alpha + \beta} \approx 0.45 \approx 0.50 \quad \text{and} \quad b = \frac{\alpha}{\alpha + \beta} \approx 0.55 \approx 0.50

The Chinchilla Heuristic:

D_{\text{opt}} \approx 20 \times N_{\text{opt}}

4. Inference-Optimal Overtraining (The LLaMA Philosophy)

Diagram: compute-optimal training compared with inference-optimal overtraining

Why did Meta train LLaMA-3 8B on 15 Trillion tokens (D \approx 1,875 N), violating Chinchilla by almost 100\times?

┌────────────────────────────────────────────────────────────────────────┐
│                   TRAINING-OPTIMAL VS. INFERENCE-OPTIMAL               │
├────────────────────┬────────────────────────┬──────────────────────────┤
│ Factor             │ Chinchilla Optimal     │ LLaMA-Style Overtraining │
├────────────────────┼────────────────────────┼──────────────────────────┤
│ Metric Optimized   │ Training FLOPs Only    │ Total Lifetime Cost      │
│ Target Ratio       │ D = 20 * N             │ D = 500 * N to 2000 * N  │
│ Model Size         │ Larger (e.g. 70B)      │ Smaller (e.g. 8B)        │
│ Training Cost      │ Minimal                │ 5x - 10x Higher          │
│ Inference Latency  │ High (Memory-bound)    │ 8x Faster                │
│ Inference Cost     │ High (Requires H100)   │ Extremely Cheap (Edge)   │
└────────────────────┴────────────────────────┴──────────────────────────┘

5. Are Emergent Abilities a Mirage? (Schaeffer et al., 2023)

Diagram: how a discontinuous metric makes smooth improvement look abrupt

For years, researchers believed that language models acquired sudden, unpredictable “Emergent Abilities” (such as multi-digit arithmetic or symbolic translation) that only appeared once a model exceeded 100 billion parameters:


6. Multi-Epoch Training & The Data-Constrained Frontier (Muennighoff et al., 2023)

Diagram: how data scarcity changes the compute-optimal training strategy

What happens when you run out of unique text data and must repeat training tokens for multiple epochs?


7. Test-Time Compute Scaling: The Reasoning Era (2024+)

In 2024, pretraining hit the Data Wall (humanity has generated roughly \sim 10^{14} high-quality text tokens on the public internet).

The new scaling frontier is Test-Time / Inference Compute Scaling (OpenAI o1/o3):


An everyday analogy

Diagram: extrapolating from small prototypes compared with building the full-scale version blind

Think of training an Olympic marathon runner:

A well-trained runner who thinks strategically beats a stronger runner who sprints blindly.


Examples in practice

Let us build a Chinchilla Compute Budget & Loss Predictor in Python:

import math
from typing import Dict, Any

def compute_chinchilla_optimal_allocation(compute_flops: float) -> Dict[str, Any]:
    # Calculates Chinchilla compute-optimal parameter count N and token count D for a given FLOP budget.
    N_opt = math.sqrt(compute_flops / 120.0)
    D_opt = 20.0 * N_opt
    
    # Predict validation loss using Chinchilla parametric constants
    E = 1.69
    A, alpha = 406.4, 0.34
    B, beta = 410.7, 0.28
    
    loss = E + (A / (N_opt ** alpha)) + (B / (D_opt ** beta))
    
    return {
        "compute_flops": f"{compute_flops:.2e}",
        "optimal_params_billions": round(N_opt / 1e9, 2),
        "optimal_tokens_billions": round(D_opt / 1e9, 2),
        "predicted_loss": round(loss, 4),
        "token_to_param_ratio": round(D_opt / N_opt, 1)
    }

def calculate_training_flops(params_billions: float, tokens_billions: float) -> float:
    # Calculates total training FLOPs: C = 6 * N * D
    N = params_billions * 1e9
    D = tokens_billions * 1e9
    return 6.0 * N * D

Implications: security, privacy, performance, scalability, and cost

Diagram: the planning, concentration, environmental and forecasting implications of scaling laws

  1. The Synthetic Data Bottleneck & Model Collapse:
    • As pretraining exhausts human internet data, labs train models on AI-generated synthetic data. If synthetic data is unfiltered, errors compound exponentially over scaling generations (Model Autophagy Disorder / Model Collapse).
  2. Geopolitical & Energy Constraints:
    • Frontier clusters reaching 10^{26} FLOPs require dedicated multi-gigawatt nuclear or hydro-electric power substations, turning datacenter energy availability into the primary geopolitical constraint on AI advancement.

Alternatives: free, open source, and commercial

Model ParadigmScaling StrategyParametersDataset ScalePrimary Advantage
Chinchilla 70BCompute-Optimal70 Billion1.4 TrillionMinimal Training FLOPs
LLaMA-3 8BInference-Overtrained8 Billion15 TrillionUltra-Low Serving Latency
Mixtral 8x7BSparse MoE Scaling47B (13B Active)15 TrillionHigh Capacity, Fast Forward
OpenAI o1 / o3Test-Time ReasoningDense / MoESFT + RL TreeUnprecedented Logic & Math

Diagram: predictable loss curves compared with unpredictable downstream capability

┌────────────────────────────────────────────────────────────────────────┐
│                   SCALING PARADIGM COMPARISON                          │
├────────────────────┬─────────────────┬──────────────────┬──────────────┤
│ Metric             │ Kaplan (2020)   │ Chinchilla (2022)│ LLaMA (2023) │
├────────────────────┼─────────────────┼──────────────────┼──────────────┤
│ Token/Param Ratio  │ ~1.7 tokens / N │ ~20 tokens / N   │ ~2,000 / N   │
│ Optimal Parameter  │ N ∝ C^0.73      │ N ∝ C^0.50       │ Fixed Small  │
│ Optimal Tokens     │ D ∝ C^0.27      │ D ∝ C^0.50       │ Maximize D   │
│ Target Efficiency  │ Flawed Baseline │ Training Optimal │ Serving Opt  │
└────────────────────┴─────────────────┴──────────────────┴──────────────┘

When to use it — and when not to

Diagram: when scaling law reasoning is valid and when it does not transfer

When to Use Scaling Law Principles:

When NOT to use:


The AI connection

Knowledge check

  1. Why does an autoregressive Transformer require 6ND FLOPs for training?
  2. Why was GPT-3 considered compute-suboptimal by DeepMind Chinchilla?
  3. What is the economic rationale behind LLaMA-style inference overtraining?
  4. What does the irreducible loss constant E represent in natural language modeling?
  5. How does Test-Time Reasoning Scaling bypass the limits of pretraining data?

Hands-on exercise

In this lab, you will build and test a modular scaling law simulator and compute budget optimization engine in Python: implement the C = 6ND calculation formula, solve for Chinchilla compute-optimal parameter-token allocations, compute predicted validation loss trajectories across compute scales, and evaluate inference-optimal overtraining economics.

Expected output

[Neural Scaling Laws Verification Suite]
Testing Compute FLOP Calculator:
  Model: 7.0B Params, Dataset: 140.0B Tokens
  Training Compute: 5.88e+21 FLOPs (C = 6ND) [EXACT MATCH]
Testing Chinchilla Compute-Optimal Sizing (Budget: 5.88e+21 FLOPs):
  Optimal Parameter Count: 7.00 Billion
  Optimal Token Count: 140.00 Billion (Ratio = 20.0) [CHINCHILLA VERIFIED]
  Predicted Cross-Entropy Loss: 2.1485
Testing Inference Overtraining Economy:
  LLaMA-3 8B (15T Tokens): 7.20e+23 FLOPs | 8.8x Smaller Model Footprint than Chinchilla Baseline
Test Suite: 3 passed in 0.41s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement an IsoFLOP Contour Plotter using NumPy and Matplotlib that generates curves of validation loss across varying parameter counts for fixed FLOP budgets.
  2. Build an Inference Cost Calculator comparing lifetime cloud GPU serving costs of a Chinchilla-optimal 70B model vs an overtrained 8B model over 1 billion queries.

Extension challenge

Implement a Toy Test-Time Search Simulator:

Quiz

Q1. Why does training an autoregressive transformer require approximately 6 FLOPs per parameter per token (C = 6ND)?

  1. Two forward plus four backward
  2. Because transformers have 6 attention heads
  3. Because there are 6 days in a working week
  4. Because 6 is the atomic number of carbon
Show answer

Answer: A. Two forward plus four backward

Forward pass takes 2 FLOPs/param/token; backward pass takes 4 FLOPs/param/token (gradient of inputs + gradient of weights), totaling 6 FLOPs.

Q2. What was the revolutionary finding of the DeepMind Chinchilla paper (Hoffmann et al., 2022)?

  1. That neural networks cannot learn from text
  2. Parameters and tokens should scale equally, ~20 per param
  3. That GPUs should be replaced with quantum computers
  4. That models should only be trained on 1,000 tokens
Show answer

Answer: B. Parameters and tokens should scale equally, ~20 per param

Chinchilla proved that parameters and tokens should scale in equal 1:1 proportion (~20 tokens per parameter).

Q3. Why did Meta train LLaMA-3 8B on 15 Trillion tokens (nearly 1,875 tokens per parameter), drastically violating the Chinchilla compute-optimal rule (20 tokens/param)?

  1. Because 8B models stop learning at 1 billion tokens
  2. Because Meta ran out of disk space
  3. Chinchilla optimises training, not serving cost
  4. Because of an error in the training script
Show answer

Answer: C. Chinchilla optimises training, not serving cost

Inference-optimal overtraining trades excess training compute for permanent reductions in per-query serving costs.

Q4. What is Test-Time Compute Scaling (the reasoning model paradigm behind OpenAI o1/o3)?

  1. Running unit tests 1,000 times
  2. Buying more GPUs to install on a laptop
  3. Testing the model on a larger test dataset
  4. Spending more compute at inference time
Show answer

Answer: D. Spending more compute at inference time

Test-time compute scaling trades inference thinking tokens and tree search for dramatic gains in complex reasoning.

Q5. In the empirical power law L(N, D) = E + A/N^alpha + B/D^beta, what does the constant E represent?

  1. The irreducible entropy of natural language
  2. The energy consumed by the datacenter in Joules
  3. The learning rate of the optimizer
  4. The number of GPUs in the cluster
Show answer

Answer: A. The irreducible entropy of natural language

E represents the Bayes optimal irreducible loss / entropy of the natural language data distribution.

Glossary

Neural Scaling Laws
Empirical power-law relationships demonstrating that cross-entropy loss decreases smoothly and predictably as a function of compute, model size, and dataset tokens.
Chinchilla Optimal
The compute-optimal allocation principle where model parameter count N and dataset token count D scale in equal proportion: D approx 20 * N.
Compute Budget (C)
The total floating-point operations required to train a model, approximated by C approx 6 * N * D for dense transformers.
Irreducible Loss (E)
The fundamental theoretical minimum cross-entropy loss (entropy) of natural language text that cannot be compressed further.
Inference-Optimal Overtraining
Training a compact model on far more tokens than the Chinchilla optimum (D >> 20N) to minimize downstream inference latency and serving costs.
IsoFLOP Profile
A curve plotting validation loss across various parameter counts N for a fixed, constant compute budget C.
Test-Time Compute
Computational FLOPs expended during inference (e.g. reasoning tokens, tree search, self-correction) to solve complex tasks.
Emergent Abilities
Capabilities that appear suddenly at specific compute scales (often an artifact of non-linear evaluation metrics on smooth underlying loss curves).

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.