LLMs and Generative AI β€Ί The LLM Landscape β€Ί Day 239

Day 239: How Large Language Models Are Trained

Day 239 of 365 β€” How Large Language Models Are Trained

Master how modern Large Language Models are created at scale: understand multi-petabyte data ingestion, MinHash LSH deduplication, heuristic filtering (Gopher rules), educational quality classifiers (FineWeb-Edu), BPE token binary packing, 3D distributed training across 10,000+ GPUs, loss spike diagnostics, and what raw base models are capable of before post-training alignment.

Course
LLMs and Generative AI
Category
The LLM Landscape
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-239-how-large-language-models-are-trained

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-239-how-large-language-models-are-trained
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Welcome to Course 06: Large Language Models and Generative AI!

In Course 05, you mastered the mathematical architecture of the Transformerβ€”attention matrices, residual highways, multi-GPU sharding, and mixed precision.

Now, we zoom out to the industrial macro-scale: How does a modern frontier AI laboratory take 100 Terabytes of raw, chaotic, misspelled, multilingual internet HTML and turn it into GPT-4, Claude 3.5 Sonnet, Gemini 1.5 Pro, or LLaMA-3.1?

Building a modern foundation model is no longer just computer science; it is data mining, high-performance distributed systems engineering, and statistical mechanics:

Today, you will master the complete pretraining pipeline: web crawling, MinHash deduplication, educational quality scoring (FineWeb-Edu), binary token packing, cluster training dynamics, and the raw behavior of unaligned base models.


The idea in plain language

Diagram: the four stages of pretraining a large language model

Think of building an encyclopedia of all human knowledge:

The result is a Base Modelβ€”an omniscient text-completion engine possessing world knowledge.


Historical background

Diagram: the historical progression of language model pretraining

End-to-end LLM pretraining data-to-weights lifecycle diagram showing 5 stages from raw web crawl to base foundation model

  1. 2018 (GPT-1 & Radford et al.): Demonstrated that generative autoregressive pretraining on BookCorpus (4.5 GB of text) enabled effective zero-shot transfer to downstream tasks.
  2. 2020 (GPT-3 & Brown et al.): Scaled pretraining to 300 Billion tokens from Common Crawl, demonstrating few-shot in-context learning.
  3. 2021 (Gopher & Rae et al., DeepMind): Formalized rigorous rule-based text quality filtering heuristics (Gopher rules).
  4. 2024 (FineWeb & Hugging Face): Released the 15-Trillion token FineWeb dataset, demonstrating that educational quality classifiers (FineWeb-Edu) produce models that match or exceed proprietary datasets.
  5. 2024 (LLaMA-3 & Meta AI): Trained an 8B and 70B model on 15 Trillion tokens across a 24,000-GPU cluster with 99.3% training uptime!

What it is β€” and what it is not

What LLM Pretraining IS:

What it is NOT:


Why it was created and what problems it solves

Pretraining solves the fundamental bottleneck of artificial intelligence: The Data Annotation Bottleneck:

  1. Zero Human Labeling Required: Supervised learning requires humans to manually label millions of images or text pairs. Self-supervised pretraining uses the natural order of human text as its own label!
  2. Universal Feature Extraction: Instead of training one model for sentiment, another for translation, and another for summarization, a single pretrained foundation model serves as the parent for all downstream tasks.

How it works

Let us dissect the industrial stages of modern LLM pretraining.


1. The Pretraining Data Refinery Funnel

Diagram: the successive filtering stages that reduce raw web text to training data

Four-stage data refinement funnel diagram illustrating web text curation from raw crawl to educational synthetic tokens

Raw web text from Common Crawl is over $80%$ junk. The refinement pipeline processes text through 4 rigorous stages:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   DATA REFINEMENT STAGES (FINEWEB / LLAMA-3)           β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Stage        β”‚ Volume Retained  β”‚ Core Algorithms & Filters            β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ 1. Raw Crawl β”‚ 100% (~500 TB)   β”‚ WET extraction, language ID (FastTextβ”‚
β”‚ 2. Heuristicsβ”‚ 35%              β”‚ Gopher rules: length, symbol ratios  β”‚
β”‚ 3. MinHash LSHβ”‚ 20%             β”‚ 5-gram Jaccard deduplication         β”‚
β”‚ 4. Quality Clsβ”‚ 10% - 15% (15T) β”‚ FineWeb-Edu / DeBERTa quality score  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The Gopher Quality Heuristics:


2. MinHash Locality-Sensitive Hashing (LSH) Deduplication

Diagram: how locality-sensitive hashing finds near-duplicate documents at scale

Why can’t we just use Python hash() or SHA-256 for deduplication? Because two web pages that differ by only a single timestamp or tracking banner will produce completely different cryptographic hashes!

MinHash LSH estimates the Jaccard Similarity between two document sets $A$ and $B$:

J(A, B) = \frac{|A \cap B|}{|A \cup B|}

3. Binary Token Packing & Zero-Copy Memory Mapping

Training clusters cannot read individual .txt or .jsonl files from diskβ€”file system inode overhead would choke the storage network.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   DATASET BINARY PACKING FORMAT                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Header: [Magic Bytes: "BINA"] [Version: 1] [Total Tokens: 15,000,000,000]
β”‚ Data Stream: [Token 0 (uint32)] [Token 1 (uint32)] [Token 2 (uint32)] ...
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

4. Cluster Dynamics: Loss Spikes and Gradient Management

Diagram: how a training loss spike is detected and recovered from

When training 16,384 GPUs in parallel, hardware failures and numerical instabilities are inevitable:


5. Document Packing & Attention Mask Boundary Resetting

Diagram: how multiple documents are packed into one training sequence without attending across boundaries

In standard batching, short documents (e.g. 500 tokens) padded to an 8,192-token context window result in over $60%$ wasted computation on < pad> tokens:


6. The Megatron-LM Data Sampler Architecture

How do 16,000 GPUs read from a 15-Trillion token dataset without coordination deadlocks?


An everyday analogy

Diagram: unstructured wide reading compared with directed instruction

Think of training an apprentice master chef:

Pretraining gives the apprentice culinary knowledge; post-training teaches them customer service.


Examples in practice

Let us build a Mini-LLM Pretraining Data Curation & MinHash Deduplicator in Python:

import hashlib
import re
from typing import List, Set, Dict

class DocumentRefinery:
    def __init__(self, num_perm: int = 64):
        self.num_perm = num_perm
        self.stopwords = {"the", "and", "is", "of", "in", "to", "a", "with", "for", "on"}

    def passes_gopher_heuristics(self, text: str) -> bool:
        words = re.findall(r'\b\w+\b', text.lower())
        if len(words) < 10:
            return False
        
        # Mean word length heuristic (3 to 10 chars)
        mean_len = sum(len(w) for w in words) / len(words)
        if mean_len < 3.0 or mean_len > 10.0:
            return False
        
        # Stopword ratio heuristic (must contain common language words)
        stopword_count = sum(1 for w in words if w in self.stopwords)
        if stopword_count < 2:
            return False
        
        # Symbol ratio heuristic (< 15% special chars)
        symbols = sum(1 for c in text if not c.isalnum() and not c.isspace())
        if (symbols / max(len(text), 1)) > 0.15:
            return False
        
        return True

    def compute_minhash_signature(self, text: str, k: int = 3) -> List[int]:
        # Generate k-shingles
        words = text.lower().split()
        if len(words) < k:
            shingles = set(words)
        else:
            shingles = {" ".join(words[i:i+k]) for i in range(len(words) - k + 1)}
        
        sig = []
        for i in range(self.num_perm):
            min_val = float('inf')
            for s in shingles:
                # Hash shingle with seed i
                h = int(hashlib.md5(f"{i}_{s}".encode()).hexdigest(), 16)
                if h < min_val:
                    min_val = h
            sig.append(int(min_val % 1000000))
        return sig

Implications: security, privacy, performance, scalability, and cost

Diagram: the cost, copyright, energy and privacy implications of large-scale pretraining

  1. Personally Identifiable Information (PII) Scrubbing:
    • Pretraining corpora must be filtered with regex and NER models to remove Social Security numbers, private phone numbers, passwords, and private API keys before weights are trained.
  2. Copyright & Fair Use Legal Dynamics:
    • Web training has sparked landmark global copyright litigation over whether ingesting copyrighted books, journalism, and art without licensing constitutes fair use.

Alternatives: free, open source, and commercial

Pretraining DatasetToken VolumeRelease YearPrimary Characteristic
Common CrawlPetabytesContinuousRaw uncurated web crawl
The Pile (EleutherAI)825 Billion2020Diverse academic, code, and book mix
RedPajama-v2 (Together)30 Trillion2023Massive open-source reproduction of LLaMA
FineWeb (Hugging Face)15 Trillion2024Gold-standard open web dataset with Edu scores

Diagram: pretraining compared with fine-tuning by scale and purpose

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   TRAINING STAGE COMPARISON                            β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Dimension          β”‚ Pretraining     β”‚ Supervised Fine-Tβ”‚ RLHF / DPO   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Compute Cost       β”‚ $10M - $100M    β”‚ $1k - $50k       β”‚ $10k - $100k β”‚
β”‚ Dataset Size       β”‚ 15 Trillion Tok β”‚ 100k - 1M Pairs  β”‚ 50k Pairs    β”‚
β”‚ Goal               β”‚ World Knowledge β”‚ Instruction Followβ”‚ Human Align  β”‚
β”‚ Hardware           β”‚ 16,000 GPUs     β”‚ 8 - 64 GPUs      β”‚ 64 - 256 GPUsβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when pretraining a model from scratch is justified

When to Pretrain a Model From Scratch:

When NOT to pretrain:


The AI connection

Knowledge check

  1. What is the fundamental difference between a base foundation model and an instruction-tuned assistant?
  2. Why are Gopher heuristics applied to raw Common Crawl web data?
  3. How does MinHash LSH estimate the Jaccard similarity between two documents in sub-linear time?
  4. Why is binary memory-mapped token serialization used instead of streaming text files directly to GPUs?
  5. How do distributed training orchestrators recover from unexpected loss spikes?

Hands-on exercise

In this lab, you will build and test an industrial LLM data preprocessing pipeline in Python: implement Gopher rule-based quality heuristics, build a MinHash LSH near-duplicate detector, write a binary token array memory packer, and verify that clean text passes while spam and duplicate documents are rejected.

Expected output

[LLM Pre-Training Data Pipeline Verification Suite]
Testing Gopher Quality Heuristics:
  Sample 1 (High Quality Article): PASSED (Mean Len: 5.4, Stopword Ratio: 32%)
  Sample 2 (SEO Spam / Gibberish): REJECTED [CORRECT]
  Sample 3 (Excessive Symbol Ratio): REJECTED [CORRECT]
Testing MinHash LSH Deduplication:
  Doc A vs Doc B (Near Identical Text): Estimated Jaccard = 0.938 [DUPLICATE DETECTED]
  Doc A vs Doc C (Distinct Topic): Estimated Jaccard = 0.047 [UNIQUE DOCUMENT]
Testing Binary Token Packer (tiktoken / BPE):
  Packaged 10,000 tokens into memory-mapped uint32 binary buffer (40.0 KB)
Test Suite: 3 passed in 0.38s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement an automated PII Scrubber using regular expressions that masks email addresses, credit card numbers, and IP addresses from raw web text.
  2. Build a Synthetic Data Generator Harness that formats multi-step reasoning math problems into a structured pretraining data stream.

Extension challenge

Implement a FastText Quality Classifier Benchmark:

Quiz

Q1. What is the primary training objective of modern autoregressive Large Language Models during pretraining?

  1. Causal language modelling: predict the next token
  2. Memorizing Wikipedia word-for-word
  3. Answering user questions as a polite chatbot
  4. Translating English to French only
Show answer

Answer: A. Causal language modelling: predict the next token

Pretraining trains the model to minimize cross-entropy loss predicting next tokens over vast text corpora.

Q2. Why is MinHash Near-Duplicate Detection crucial when building an LLM pretraining dataset?

  1. Because GPUs cannot process duplicate numbers
  2. Repeated web text wastes compute and hurts reasoning
  3. Because duplicate text causes physical hard drive damage
  4. Because web crawlers only work on unique URLs
Show answer

Answer: B. Repeated web text wastes compute and hurts reasoning

Deduplication removes repetitive web boilerplate and spam, preventing severe capacity waste.

Q3. What is a "Base Model" (e.g. LLaMA-3-8B-Base) prior to instruction tuning or RLHF?

  1. A broken neural network that fails all tests
  2. A model that only runs on basic hardware
  3. A text-completion engine, not an assistant
  4. A model that only understands Python code
Show answer

Answer: C. A text-completion engine, not an assistant

Base models are pure text predictors that continue patterns rather than answering prompts conversationally.

Q4. What is the standard engineering protocol when a multi-week pretraining run encounters an irrecoverable gradient explosion / loss spike?

  1. Delete the model weights and start from epoch 0
  2. Cancel the entire project and fire the team
  3. Ignore the loss spike and continue training
  4. Roll back, skip the bad batches, and resume
Show answer

Answer: D. Roll back, skip the bad batches, and resume

Standard cluster protocol involves rolling back to clean checkpoints and skipping the corrupted data shard.

Q5. What role do educational classifiers (like FineWeb-Edu) play in modern foundation model data pipelines?

  1. Small classifiers score and filter for quality
  2. They grade student homework automatically
  3. They teach university courses
  4. They enforce copyright laws
Show answer

Answer: A. Small classifiers score and filter for quality

Educational classifiers filter web data for informational density, reasoning, and pedagogical clarity.

Glossary

Base Model
A foundation model trained purely on self-supervised next-token prediction across trillions of tokens, without instruction tuning or conversational alignment.
Causal Language Modeling (CLM)
The autoregressive pretraining objective where the model predicts token w_t conditioned only on preceding tokens w_<t.
MinHash LSH
A probabilistic hashing algorithm (Locality-Sensitive Hashing) used to rapidly find and remove near-duplicate documents across multi-terabyte web corpora.
Gopher Heuristics
A standard set of document quality filters measuring stopword ratios, mean word lengths, symbol-to-word ratios, and ellipsis counts to filter low-quality text.
FineWeb-Edu
A 5.4-trillion token pretraining dataset curated using an educational quality scoring classifier to select pedagogically rich web text.
Loss Spike
A sudden, catastrophic explosion in validation loss during pretraining, typically triggered by corrupted data batches, extreme gradient norms, or mixed-precision underflow.
Memory-Mapped Binary Tokens
Tokenized datasets serialized into flat binary arrays of 16-bit or 32-bit integers (e.g. numpy memmap) allowing zero-overhead streaming across GPU clusters.
Synthetic Data Curation
Using advanced frontier models to generate structured textbooks, mathematical reasoning steps, and verified code to enrich pretraining mixtures.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.