LLMs and Generative AI › The LLM Landscape › Day 240
Hands-on lab — Day 240: Pretraining, Fine-Tuning, and RLHF
- ← Back to the Day 240 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-240-pretraining-fine-tuning-and-rlhf/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/pretraining_fine_tuning_and_rlhf_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/pretraining_fine_tuning_and_rlhf_lib.py examples/test_pretraining_fine_tuning_and_rlhf_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/pretraining_fine_tuning_and_rlhf_lib.py starter/test_pretraining_fine_tuning_and_rlhf_lib.py tests/run_tests.sh tests/test_pretraining_fine_tuning_and_rlhf_lib.py troubleshooting.md
Lab README
Lab: Day 240 -- Pretraining, Fine-Tuning, and RLHF
Lesson
Day number: 240 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Pretraining, Supervised Fine-Tuning, and Direct Preference Optimization (DPO).
Purpose
Build and test a complete preference alignment module in PyTorch. Implement the Bradley-Terry preference probability function, compute Direct Preference Optimization (DPO) loss across winning and losing response tensors, and verify implicit reward margin separation.
Learning objectives
- Implement the Bradley-Terry pairwise preference probability formulation.
- Build the Direct Preference Optimization (DPO) implicit reward loss function.
- Compute implicit reward margins between chosen and rejected responses.
- Understand the role of the KL-divergence parameter $\beta$ in policy stability.
Prerequisites
- Day 239 (How Large Language Models Are Trained).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/pretraining_fine_tuning_and_rlhf_lib.py: Student scaffold file.examples/pretraining_fine_tuning_and_rlhf_lib.py: Complete reference implementation.tests/test_pretraining_fine_tuning_and_rlhf_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/pretraining_fine_tuning_and_rlhf_lib.py
What the commands do
- Evaluates Bradley-Terry preference probabilities.
- Computes DPO loss and implicit reward margins on test tensors.
- Runs unit test assertions.
Expected output
Alignment Demo: BT Prob = 0.973, DPO Loss = 0.4741, Mean Margin = 0.4000
Validation steps
- Verify
compute_bradley_terry_probabilityreturns $0.5$ when rewards are equal. - Confirm
compute_dpo_lossyields $\approx \ln(2) \approx 0.6931$ when policy equals reference. - Confirm implicit reward margins increase as policy likelihood on chosen tokens improves.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Underflow in Bradley-Terry sigmoid: Use
math.exp(-diff)safely clamped within $[-20, 20]$.
Security notes
All alignment calculations execute locally in process memory.
Extension exercises
- Implement the Kahneman-Tversky Optimization (KTO) loss function.
- Build an automated prompt-masking dataset collator for SFT.
Navigation
- Lesson title: Pretraining, Fine-Tuning, and RLHF
- Day number: 240 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-240-pretraining-fine-tuning-and-rlhf
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-240-pretraining-fine-tuning-and-rlhfwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 240
- `BT Prob`: Bradley-Terry pairwise preference probability.
- `DPO Loss`: Direct Preference Optimization loss value.
- `Mean Margin`: Implicit reward margin between chosen and rejected responses.
examples-run.txt
Alignment Demo: BT Prob = 0.973, DPO Loss = 0.4741, Mean Margin = 0.4000
measured-values.txt
BT Prob: 0.973
DPO Loss: 0.4741
Mean Margin: 0.4000
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_pretraining_fine_tuning_and_rlhf_lib.py::test_bradley_terry_probabilities PASSED [ 33%]
tests/test_pretraining_fine_tuning_and_rlhf_lib.py::test_dpo_loss_mechanics PASSED [ 66%]
tests/test_pretraining_fine_tuning_and_rlhf_lib.py::test_dpo_loss_symmetry_when_identical PASSED [100%]
============================== 3 passed in 0.42s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/pretraining_fine_tuning_and_rlhf_lib.py (1726 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from typing import Tuple
def compute_bradley_terry_probability(reward_chosen: float, reward_rejected: float) -> float:
# P(y_w > y_l) = 1 / (1 + exp(-(r_w - r_l))) = sigmoid(r_w - r_l)
diff = reward_chosen - reward_rejected
return float(1.0 / (1.0 + math.exp(-diff)))
def compute_dpo_loss(
policy_chosen_logp: torch.Tensor,
policy_rejected_logp: torch.Tensor,
ref_chosen_logp: torch.Tensor,
ref_rejected_logp: torch.Tensor,
beta: float = 0.1
) -> Tuple[torch.Tensor, torch.Tensor]:
pi_logratios = policy_chosen_logp - policy_rejected_logp
ref_logratios = ref_chosen_logp - ref_rejected_logp
logits = beta * (pi_logratios - ref_logratios)
loss = -F.logsigmoid(logits).mean()
# Implicit reward margin: beta * (log(pi_w/ref_w) - log(pi_l/ref_l))
implicit_margin = (beta * (policy_chosen_logp - ref_chosen_logp) -
beta * (policy_rejected_logp - ref_rejected_logp))
return loss, implicit_margin
def run_alignment_demo():
bt_prob = compute_bradley_terry_probability(2.4, -1.2)
# Simulated sequence log-probabilities (batch of 2)
pol_chosen = torch.tensor([-12.4, -10.2])
pol_rejected = torch.tensor([-18.2, -16.5])
ref_chosen = torch.tensor([-12.3, -10.1])
ref_rejected = torch.tensor([-14.1, -13.0])
loss, margin = compute_dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1)
print(f"Alignment Demo: BT Prob = {bt_prob:.3f}, DPO Loss = {loss.item():.4f}, Mean Margin = {margin.mean().item():.4f}")
return bt_prob, loss.item(), margin.mean().item()
if __name__ == "__main__":
run_alignment_demo()
examples/test_pretraining_fine_tuning_and_rlhf_lib.py (1260 bytes)
import pytest
import torch
from examples.pretraining_fine_tuning_and_rlhf_lib import (
compute_bradley_terry_probability,
compute_dpo_loss
)
def test_bradley_terry_probabilities():
p_equal = compute_bradley_terry_probability(1.0, 1.0)
p_higher = compute_bradley_terry_probability(2.4, -1.2)
p_lower = compute_bradley_terry_probability(-1.0, 2.0)
assert p_equal == pytest.approx(0.5, rel=1e-3)
assert p_higher > 0.95
assert p_lower < 0.10
def test_dpo_loss_mechanics():
# Setup batch where policy strongly prefers chosen over reference
pol_chosen = torch.tensor([-10.0])
pol_rejected = torch.tensor([-20.0])
ref_chosen = torch.tensor([-10.0])
ref_rejected = torch.tensor([-15.0])
loss, margin = compute_dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1)
assert loss.item() > 0.0
assert margin.item() > 0.0
def test_dpo_loss_symmetry_when_identical():
# If policy matches reference, logits = 0 -> loss = -log(sigmoid(0)) = log(2) ≈ 0.6931
logps = torch.tensor([-10.0])
loss, margin = compute_dpo_loss(logps, logps, logps, logps, beta=0.1)
assert loss.item() == pytest.approx(0.6931, rel=1e-3)
assert margin.item() == pytest.approx(0.0, abs=1e-5)
metadata.yml (445 bytes)
lesson_id: D240
day: 240
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/pretraining_fine_tuning_and_rlhf_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/pretraining_fine_tuning_and_rlhf_lib.py (530 bytes)
import torch
import torch.nn as nn
from typing import Tuple
def compute_bradley_terry_probability(reward_chosen: float, reward_rejected: float) -> float:
# TODO: Calculate P(chosen > rejected) using logistic sigmoid
pass
def compute_dpo_loss(
policy_chosen_logp: torch.Tensor,
policy_rejected_logp: torch.Tensor,
ref_chosen_logp: torch.Tensor,
ref_rejected_logp: torch.Tensor,
beta: float = 0.1
) -> Tuple[torch.Tensor, torch.Tensor]:
# TODO: Compute DPO loss and implicit reward margin
pass
starter/test_pretraining_fine_tuning_and_rlhf_lib.py (1260 bytes)
import pytest
import torch
from examples.pretraining_fine_tuning_and_rlhf_lib import (
compute_bradley_terry_probability,
compute_dpo_loss
)
def test_bradley_terry_probabilities():
p_equal = compute_bradley_terry_probability(1.0, 1.0)
p_higher = compute_bradley_terry_probability(2.4, -1.2)
p_lower = compute_bradley_terry_probability(-1.0, 2.0)
assert p_equal == pytest.approx(0.5, rel=1e-3)
assert p_higher > 0.95
assert p_lower < 0.10
def test_dpo_loss_mechanics():
# Setup batch where policy strongly prefers chosen over reference
pol_chosen = torch.tensor([-10.0])
pol_rejected = torch.tensor([-20.0])
ref_chosen = torch.tensor([-10.0])
ref_rejected = torch.tensor([-15.0])
loss, margin = compute_dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1)
assert loss.item() > 0.0
assert margin.item() > 0.0
def test_dpo_loss_symmetry_when_identical():
# If policy matches reference, logits = 0 -> loss = -log(sigmoid(0)) = log(2) ≈ 0.6931
logps = torch.tensor([-10.0])
loss, margin = compute_dpo_loss(logps, logps, logps, logps, beta=0.1)
assert loss.item() == pytest.approx(0.6931, rel=1e-3)
assert margin.item() == pytest.approx(0.0, abs=1e-5)
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 240 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_pretraining_fine_tuning_and_rlhf_lib.py (1260 bytes)
import pytest
import torch
from examples.pretraining_fine_tuning_and_rlhf_lib import (
compute_bradley_terry_probability,
compute_dpo_loss
)
def test_bradley_terry_probabilities():
p_equal = compute_bradley_terry_probability(1.0, 1.0)
p_higher = compute_bradley_terry_probability(2.4, -1.2)
p_lower = compute_bradley_terry_probability(-1.0, 2.0)
assert p_equal == pytest.approx(0.5, rel=1e-3)
assert p_higher > 0.95
assert p_lower < 0.10
def test_dpo_loss_mechanics():
# Setup batch where policy strongly prefers chosen over reference
pol_chosen = torch.tensor([-10.0])
pol_rejected = torch.tensor([-20.0])
ref_chosen = torch.tensor([-10.0])
ref_rejected = torch.tensor([-15.0])
loss, margin = compute_dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1)
assert loss.item() > 0.0
assert margin.item() > 0.0
def test_dpo_loss_symmetry_when_identical():
# If policy matches reference, logits = 0 -> loss = -log(sigmoid(0)) = log(2) ≈ 0.6931
logps = torch.tensor([-10.0])
loss, margin = compute_dpo_loss(logps, logps, logps, logps, beta=0.1)
assert loss.item() == pytest.approx(0.6931, rel=1e-3)
assert margin.item() == pytest.approx(0.0, abs=1e-5)
Troubleshooting
Troubleshooting: Day 240 - Pretraining, Fine-Tuning, and RLHF
Common Issues
- DPO Loss divergence:
- Cause: $\beta$ parameter set too high ($> 1.0$).
- Fix: Keep $\beta$ in range $[0.05, 0.20]$.
Security notes
Security & Privacy: Day 240 - Pretraining, Fine-Tuning, and RLHF
Security Guidance
- All alignment optimization loops execute locally in memory.