LLMs and Generative AI › The LLM Landscape › Day 239

Hands-on lab — Day 239: How Large Language Models Are Trained

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/how_large_language_models_are_trained_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/how_large_language_models_are_trained_lib.py
examples/test_how_large_language_models_are_trained_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/how_large_language_models_are_trained_lib.py
starter/test_how_large_language_models_are_trained_lib.py
tests/run_tests.sh
tests/test_how_large_language_models_are_trained_lib.py
troubleshooting.md

Lab README

Lab: Day 239 -- How Large Language Models Are Trained

Lesson

Day number: 239 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Industrial LLM Pre-Training, Web Data Filtering, and MinHash Deduplication.

Purpose

Build and test an industrial LLM data preprocessing pipeline in Python. Implement Gopher rule-based quality heuristics, build a MinHash LSH near-duplicate detector, and verify that clean text passes while spam and duplicate documents are rejected.

Learning objectives

  • Implement Gopher rule-based document quality filtering heuristics.
  • Build a MinHash LSH signature generator to estimate Jaccard similarity.
  • Filter out SEO spam, symbol-dense noise, and short text documents.
  • Understand binary token memory mapping principles for cluster DataLoaders.

Prerequisites

  • Day 238 (Section Project: Reproducing a Paper).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard math libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/how_large_language_models_are_trained_lib.py: Student scaffold file.
  • examples/how_large_language_models_are_trained_lib.py: Complete reference implementation.
  • tests/test_how_large_language_models_are_trained_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/how_large_language_models_are_trained_lib.py

What the commands do

  • Evaluates Gopher heuristics on clean and spam text samples.
  • Generates MinHash signatures and computes Jaccard similarity estimates.
  • Runs unit test assertions.

Expected output

Data Pipeline Demo: Clean Passed = True, Spam Passed = False, Estimated Jaccard = 0.906

Validation steps

  1. Verify passes_gopher_heuristics rejects text under 8 words or with excessive symbols.
  2. Confirm compute_minhash_signature generates num_perm integer hash values.
  3. Confirm estimate_jaccard detects near-duplicate text ($> 0.60$) and differentiates unique text ($< 0.20$).
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Noisy Jaccard estimates: Increase num_perm from 32 to 64.

Security notes

All data filtering executes locally in process memory.

Extension exercises

  1. Implement a regex-based PII scrubber removing emails and credit cards.
  2. Build an educational quality heuristic scoring function.

Expected output

FIELDS.md

# Expected Output Fields: Day 239

- `Clean Passed`: Boolean status confirming clean text passed quality heuristics.
- `Spam Passed`: Boolean status confirming spam was rejected.
- `Estimated Jaccard`: Jaccard similarity estimate from MinHash signatures.

examples-run.txt

Data Pipeline Demo: Clean Passed = True, Spam Passed = False, Estimated Jaccard = 0.906

measured-values.txt

Clean Passed: True
Spam Passed: False
Estimated Jaccard: 0.906

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_how_large_language_models_are_trained_lib.py::test_gopher_heuristics PASSED [ 50%]
tests/test_how_large_language_models_are_trained_lib.py::test_minhash_duplicate_detection PASSED [100%]

============================== 2 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/how_large_language_models_are_trained_lib.py (2447 bytes)
import re
import hashlib
import numpy as np
from typing import List, Set, Dict

class DocumentRefinery:
    def __init__(self, num_perm: int = 32):
        self.num_perm = num_perm
        self.stopwords = {"the", "and", "is", "of", "in", "to", "a", "with", "for", "on"}

    def passes_gopher_heuristics(self, text: str) -> bool:
        words = re.findall(r'\b\w+\b', text.lower())
        if len(words) < 8:
            return False

        mean_len = sum(len(w) for w in words) / len(words)
        if mean_len < 3.0 or mean_len > 10.0:
            return False

        stopword_count = sum(1 for w in words if w in self.stopwords)
        if stopword_count < 2:
            return False

        symbols = sum(1 for c in text if not c.isalnum() and not c.isspace())
        if (symbols / max(len(text), 1)) > 0.20:
            return False

        return True

    def compute_minhash_signature(self, text: str, k: int = 3) -> List[int]:
        words = text.lower().split()
        if len(words) < k:
            shingles = set(words)
        else:
            shingles = {" ".join(words[i:i+k]) for i in range(len(words) - k + 1)}

        sig = []
        for i in range(self.num_perm):
            min_val = float('inf')
            for s in shingles:
                h = int(hashlib.md5(f"{i}_{s}".encode()).hexdigest(), 16)
                if h < min_val:
                    min_val = h
            sig.append(int(min_val % 1000000))
        return sig

    def estimate_jaccard(self, sig_a: List[int], sig_b: List[int]) -> float:
        matches = sum(1 for a, b in zip(sig_a, sig_b) if a == b)
        return float(matches / len(sig_a))

def run_pipeline_demo():
    refinery = DocumentRefinery(num_perm=64)
    doc_clean = "The deep learning model is trained on massive datasets to predict the next word in the sequence."
    doc_spam = "BUY NOW $$$ CLICK HERE 12345 !@#$ !!!"

    passed_clean = refinery.passes_gopher_heuristics(doc_clean)
    passed_spam = refinery.passes_gopher_heuristics(doc_spam)

    sig_a = refinery.compute_minhash_signature(doc_clean)
    sig_b = refinery.compute_minhash_signature(doc_clean + " Extra minor addition.")
    jaccard = refinery.estimate_jaccard(sig_a, sig_b)

    print(f"Data Pipeline Demo: Clean Passed = {passed_clean}, Spam Passed = {passed_spam}, Estimated Jaccard = {jaccard:.3f}")
    return passed_clean, passed_spam, jaccard

if __name__ == "__main__":
    run_pipeline_demo()
examples/test_how_large_language_models_are_trained_lib.py (1295 bytes)
import pytest
from examples.how_large_language_models_are_trained_lib import DocumentRefinery

def test_gopher_heuristics():
    refinery = DocumentRefinery()
    doc_valid = "Artificial intelligence and language models are transforming modern computer science with deep learning."
    doc_short = "Hi there"
    doc_symbols = "#$$%^^&&! @@@!!! 1234"

    assert refinery.passes_gopher_heuristics(doc_valid) is True
    assert refinery.passes_gopher_heuristics(doc_short) is False
    assert refinery.passes_gopher_heuristics(doc_symbols) is False

def test_minhash_duplicate_detection():
    refinery = DocumentRefinery(num_perm=64)
    text1 = "Transformers use self attention to process sequence data efficiently across GPU clusters."
    text2 = "Transformers use self attention to process sequence data efficiently across GPU clusters and nodes."
    text3 = "Quantum computing uses qubits and superposition to solve complex cryptographic algorithms."

    sig1 = refinery.compute_minhash_signature(text1)
    sig2 = refinery.compute_minhash_signature(text2)
    sig3 = refinery.compute_minhash_signature(text3)

    jaccard_near = refinery.estimate_jaccard(sig1, sig2)
    jaccard_diff = refinery.estimate_jaccard(sig1, sig3)

    assert jaccard_near > 0.60
    assert jaccard_diff < 0.20
metadata.yml (450 bytes)
lesson_id: D239
day: 239
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/how_large_language_models_are_trained_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/how_large_language_models_are_trained_lib.py (525 bytes)
import re
import hashlib
from typing import List, Set, Dict

class DocumentRefinery:
    def __init__(self, num_perm: int = 32):
        self.num_perm = num_perm
        self.stopwords = {"the", "and", "is", "of", "in", "to", "a", "with", "for", "on"}

    def passes_gopher_heuristics(self, text: str) -> bool:
        # TODO: Filter text by length, stopwords, and symbols
        pass

    def compute_minhash_signature(self, text: str, k: int = 3) -> List[int]:
        # TODO: Compute MinHash signature list
        pass
starter/test_how_large_language_models_are_trained_lib.py (1295 bytes)
import pytest
from examples.how_large_language_models_are_trained_lib import DocumentRefinery

def test_gopher_heuristics():
    refinery = DocumentRefinery()
    doc_valid = "Artificial intelligence and language models are transforming modern computer science with deep learning."
    doc_short = "Hi there"
    doc_symbols = "#$$%^^&&! @@@!!! 1234"

    assert refinery.passes_gopher_heuristics(doc_valid) is True
    assert refinery.passes_gopher_heuristics(doc_short) is False
    assert refinery.passes_gopher_heuristics(doc_symbols) is False

def test_minhash_duplicate_detection():
    refinery = DocumentRefinery(num_perm=64)
    text1 = "Transformers use self attention to process sequence data efficiently across GPU clusters."
    text2 = "Transformers use self attention to process sequence data efficiently across GPU clusters and nodes."
    text3 = "Quantum computing uses qubits and superposition to solve complex cryptographic algorithms."

    sig1 = refinery.compute_minhash_signature(text1)
    sig2 = refinery.compute_minhash_signature(text2)
    sig3 = refinery.compute_minhash_signature(text3)

    jaccard_near = refinery.estimate_jaccard(sig1, sig2)
    jaccard_diff = refinery.estimate_jaccard(sig1, sig3)

    assert jaccard_near > 0.60
    assert jaccard_diff < 0.20
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 239 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_how_large_language_models_are_trained_lib.py (1295 bytes)
import pytest
from examples.how_large_language_models_are_trained_lib import DocumentRefinery

def test_gopher_heuristics():
    refinery = DocumentRefinery()
    doc_valid = "Artificial intelligence and language models are transforming modern computer science with deep learning."
    doc_short = "Hi there"
    doc_symbols = "#$$%^^&&! @@@!!! 1234"

    assert refinery.passes_gopher_heuristics(doc_valid) is True
    assert refinery.passes_gopher_heuristics(doc_short) is False
    assert refinery.passes_gopher_heuristics(doc_symbols) is False

def test_minhash_duplicate_detection():
    refinery = DocumentRefinery(num_perm=64)
    text1 = "Transformers use self attention to process sequence data efficiently across GPU clusters."
    text2 = "Transformers use self attention to process sequence data efficiently across GPU clusters and nodes."
    text3 = "Quantum computing uses qubits and superposition to solve complex cryptographic algorithms."

    sig1 = refinery.compute_minhash_signature(text1)
    sig2 = refinery.compute_minhash_signature(text2)
    sig3 = refinery.compute_minhash_signature(text3)

    jaccard_near = refinery.estimate_jaccard(sig1, sig2)
    jaccard_diff = refinery.estimate_jaccard(sig1, sig3)

    assert jaccard_near > 0.60
    assert jaccard_diff < 0.20

Troubleshooting

Troubleshooting: Day 239 - How Large Language Models Are Trained

Common Issues

  1. Shingle size k too large for short texts:
    • Cause: Choosing $k=5$ on text with only 4 words.
    • Fix: Fall back to individual words if word count $< k$.

Security notes

Security & Privacy: Day 239 - How Large Language Models Are Trained

Security Guidance

  • All data cleaning pipelines execute locally in memory.