LLMs and Generative AI › The LLM Landscape › Day 239
Hands-on lab — Day 239: How Large Language Models Are Trained
- ← Back to the Day 239 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-239-how-large-language-models-are-trained/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/how_large_language_models_are_trained_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/how_large_language_models_are_trained_lib.py examples/test_how_large_language_models_are_trained_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/how_large_language_models_are_trained_lib.py starter/test_how_large_language_models_are_trained_lib.py tests/run_tests.sh tests/test_how_large_language_models_are_trained_lib.py troubleshooting.md
Lab README
Lab: Day 239 -- How Large Language Models Are Trained
Lesson
Day number: 239 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Industrial LLM Pre-Training, Web Data Filtering, and MinHash Deduplication.
Purpose
Build and test an industrial LLM data preprocessing pipeline in Python. Implement Gopher rule-based quality heuristics, build a MinHash LSH near-duplicate detector, and verify that clean text passes while spam and duplicate documents are rejected.
Learning objectives
- Implement Gopher rule-based document quality filtering heuristics.
- Build a MinHash LSH signature generator to estimate Jaccard similarity.
- Filter out SEO spam, symbol-dense noise, and short text documents.
- Understand binary token memory mapping principles for cluster DataLoaders.
Prerequisites
- Day 238 (Section Project: Reproducing a Paper).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard math libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/how_large_language_models_are_trained_lib.py: Student scaffold file.examples/how_large_language_models_are_trained_lib.py: Complete reference implementation.tests/test_how_large_language_models_are_trained_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/how_large_language_models_are_trained_lib.py
What the commands do
- Evaluates Gopher heuristics on clean and spam text samples.
- Generates MinHash signatures and computes Jaccard similarity estimates.
- Runs unit test assertions.
Expected output
Data Pipeline Demo: Clean Passed = True, Spam Passed = False, Estimated Jaccard = 0.906
Validation steps
- Verify
passes_gopher_heuristicsrejects text under 8 words or with excessive symbols. - Confirm
compute_minhash_signaturegeneratesnum_perminteger hash values. - Confirm
estimate_jaccarddetects near-duplicate text ($> 0.60$) and differentiates unique text ($< 0.20$). - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Noisy Jaccard estimates: Increase
num_permfrom 32 to 64.
Security notes
All data filtering executes locally in process memory.
Extension exercises
- Implement a regex-based PII scrubber removing emails and credit cards.
- Build an educational quality heuristic scoring function.
Navigation
- Lesson title: How Large Language Models Are Trained
- Day number: 239 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-239-how-large-language-models-are-trained
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-239-how-large-language-models-are-trainedwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 239
- `Clean Passed`: Boolean status confirming clean text passed quality heuristics.
- `Spam Passed`: Boolean status confirming spam was rejected.
- `Estimated Jaccard`: Jaccard similarity estimate from MinHash signatures.
examples-run.txt
Data Pipeline Demo: Clean Passed = True, Spam Passed = False, Estimated Jaccard = 0.906
measured-values.txt
Clean Passed: True
Spam Passed: False
Estimated Jaccard: 0.906
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 2 items
tests/test_how_large_language_models_are_trained_lib.py::test_gopher_heuristics PASSED [ 50%]
tests/test_how_large_language_models_are_trained_lib.py::test_minhash_duplicate_detection PASSED [100%]
============================== 2 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/how_large_language_models_are_trained_lib.py (2447 bytes)
import re
import hashlib
import numpy as np
from typing import List, Set, Dict
class DocumentRefinery:
def __init__(self, num_perm: int = 32):
self.num_perm = num_perm
self.stopwords = {"the", "and", "is", "of", "in", "to", "a", "with", "for", "on"}
def passes_gopher_heuristics(self, text: str) -> bool:
words = re.findall(r'\b\w+\b', text.lower())
if len(words) < 8:
return False
mean_len = sum(len(w) for w in words) / len(words)
if mean_len < 3.0 or mean_len > 10.0:
return False
stopword_count = sum(1 for w in words if w in self.stopwords)
if stopword_count < 2:
return False
symbols = sum(1 for c in text if not c.isalnum() and not c.isspace())
if (symbols / max(len(text), 1)) > 0.20:
return False
return True
def compute_minhash_signature(self, text: str, k: int = 3) -> List[int]:
words = text.lower().split()
if len(words) < k:
shingles = set(words)
else:
shingles = {" ".join(words[i:i+k]) for i in range(len(words) - k + 1)}
sig = []
for i in range(self.num_perm):
min_val = float('inf')
for s in shingles:
h = int(hashlib.md5(f"{i}_{s}".encode()).hexdigest(), 16)
if h < min_val:
min_val = h
sig.append(int(min_val % 1000000))
return sig
def estimate_jaccard(self, sig_a: List[int], sig_b: List[int]) -> float:
matches = sum(1 for a, b in zip(sig_a, sig_b) if a == b)
return float(matches / len(sig_a))
def run_pipeline_demo():
refinery = DocumentRefinery(num_perm=64)
doc_clean = "The deep learning model is trained on massive datasets to predict the next word in the sequence."
doc_spam = "BUY NOW $$$ CLICK HERE 12345 !@#$ !!!"
passed_clean = refinery.passes_gopher_heuristics(doc_clean)
passed_spam = refinery.passes_gopher_heuristics(doc_spam)
sig_a = refinery.compute_minhash_signature(doc_clean)
sig_b = refinery.compute_minhash_signature(doc_clean + " Extra minor addition.")
jaccard = refinery.estimate_jaccard(sig_a, sig_b)
print(f"Data Pipeline Demo: Clean Passed = {passed_clean}, Spam Passed = {passed_spam}, Estimated Jaccard = {jaccard:.3f}")
return passed_clean, passed_spam, jaccard
if __name__ == "__main__":
run_pipeline_demo()
examples/test_how_large_language_models_are_trained_lib.py (1295 bytes)
import pytest
from examples.how_large_language_models_are_trained_lib import DocumentRefinery
def test_gopher_heuristics():
refinery = DocumentRefinery()
doc_valid = "Artificial intelligence and language models are transforming modern computer science with deep learning."
doc_short = "Hi there"
doc_symbols = "#$$%^^&&! @@@!!! 1234"
assert refinery.passes_gopher_heuristics(doc_valid) is True
assert refinery.passes_gopher_heuristics(doc_short) is False
assert refinery.passes_gopher_heuristics(doc_symbols) is False
def test_minhash_duplicate_detection():
refinery = DocumentRefinery(num_perm=64)
text1 = "Transformers use self attention to process sequence data efficiently across GPU clusters."
text2 = "Transformers use self attention to process sequence data efficiently across GPU clusters and nodes."
text3 = "Quantum computing uses qubits and superposition to solve complex cryptographic algorithms."
sig1 = refinery.compute_minhash_signature(text1)
sig2 = refinery.compute_minhash_signature(text2)
sig3 = refinery.compute_minhash_signature(text3)
jaccard_near = refinery.estimate_jaccard(sig1, sig2)
jaccard_diff = refinery.estimate_jaccard(sig1, sig3)
assert jaccard_near > 0.60
assert jaccard_diff < 0.20
metadata.yml (450 bytes)
lesson_id: D239
day: 239
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/how_large_language_models_are_trained_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/how_large_language_models_are_trained_lib.py (525 bytes)
import re
import hashlib
from typing import List, Set, Dict
class DocumentRefinery:
def __init__(self, num_perm: int = 32):
self.num_perm = num_perm
self.stopwords = {"the", "and", "is", "of", "in", "to", "a", "with", "for", "on"}
def passes_gopher_heuristics(self, text: str) -> bool:
# TODO: Filter text by length, stopwords, and symbols
pass
def compute_minhash_signature(self, text: str, k: int = 3) -> List[int]:
# TODO: Compute MinHash signature list
pass
starter/test_how_large_language_models_are_trained_lib.py (1295 bytes)
import pytest
from examples.how_large_language_models_are_trained_lib import DocumentRefinery
def test_gopher_heuristics():
refinery = DocumentRefinery()
doc_valid = "Artificial intelligence and language models are transforming modern computer science with deep learning."
doc_short = "Hi there"
doc_symbols = "#$$%^^&&! @@@!!! 1234"
assert refinery.passes_gopher_heuristics(doc_valid) is True
assert refinery.passes_gopher_heuristics(doc_short) is False
assert refinery.passes_gopher_heuristics(doc_symbols) is False
def test_minhash_duplicate_detection():
refinery = DocumentRefinery(num_perm=64)
text1 = "Transformers use self attention to process sequence data efficiently across GPU clusters."
text2 = "Transformers use self attention to process sequence data efficiently across GPU clusters and nodes."
text3 = "Quantum computing uses qubits and superposition to solve complex cryptographic algorithms."
sig1 = refinery.compute_minhash_signature(text1)
sig2 = refinery.compute_minhash_signature(text2)
sig3 = refinery.compute_minhash_signature(text3)
jaccard_near = refinery.estimate_jaccard(sig1, sig2)
jaccard_diff = refinery.estimate_jaccard(sig1, sig3)
assert jaccard_near > 0.60
assert jaccard_diff < 0.20
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 239 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_how_large_language_models_are_trained_lib.py (1295 bytes)
import pytest
from examples.how_large_language_models_are_trained_lib import DocumentRefinery
def test_gopher_heuristics():
refinery = DocumentRefinery()
doc_valid = "Artificial intelligence and language models are transforming modern computer science with deep learning."
doc_short = "Hi there"
doc_symbols = "#$$%^^&&! @@@!!! 1234"
assert refinery.passes_gopher_heuristics(doc_valid) is True
assert refinery.passes_gopher_heuristics(doc_short) is False
assert refinery.passes_gopher_heuristics(doc_symbols) is False
def test_minhash_duplicate_detection():
refinery = DocumentRefinery(num_perm=64)
text1 = "Transformers use self attention to process sequence data efficiently across GPU clusters."
text2 = "Transformers use self attention to process sequence data efficiently across GPU clusters and nodes."
text3 = "Quantum computing uses qubits and superposition to solve complex cryptographic algorithms."
sig1 = refinery.compute_minhash_signature(text1)
sig2 = refinery.compute_minhash_signature(text2)
sig3 = refinery.compute_minhash_signature(text3)
jaccard_near = refinery.estimate_jaccard(sig1, sig2)
jaccard_diff = refinery.estimate_jaccard(sig1, sig3)
assert jaccard_near > 0.60
assert jaccard_diff < 0.20
Troubleshooting
Troubleshooting: Day 239 - How Large Language Models Are Trained
Common Issues
- Shingle size k too large for short texts:
- Cause: Choosing $k=5$ on text with only 4 words.
- Fix: Fall back to individual words if word count $< k$.
Security notes
Security & Privacy: Day 239 - How Large Language Models Are Trained
Security Guidance
- All data cleaning pipelines execute locally in memory.