LLMs and Generative AI › The LLM Landscape › Day 240

Hands-on lab — Day 240: Pretraining, Fine-Tuning, and RLHF

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/pretraining_fine_tuning_and_rlhf_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/pretraining_fine_tuning_and_rlhf_lib.py
examples/test_pretraining_fine_tuning_and_rlhf_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/pretraining_fine_tuning_and_rlhf_lib.py
starter/test_pretraining_fine_tuning_and_rlhf_lib.py
tests/run_tests.sh
tests/test_pretraining_fine_tuning_and_rlhf_lib.py
troubleshooting.md

Lab README

Lab: Day 240 -- Pretraining, Fine-Tuning, and RLHF

Lesson

Day number: 240 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Pretraining, Supervised Fine-Tuning, and Direct Preference Optimization (DPO).

Purpose

Build and test a complete preference alignment module in PyTorch. Implement the Bradley-Terry preference probability function, compute Direct Preference Optimization (DPO) loss across winning and losing response tensors, and verify implicit reward margin separation.

Learning objectives

  • Implement the Bradley-Terry pairwise preference probability formulation.
  • Build the Direct Preference Optimization (DPO) implicit reward loss function.
  • Compute implicit reward margins between chosen and rejected responses.
  • Understand the role of the KL-divergence parameter $\beta$ in policy stability.

Prerequisites

  • Day 239 (How Large Language Models Are Trained).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/pretraining_fine_tuning_and_rlhf_lib.py: Student scaffold file.
  • examples/pretraining_fine_tuning_and_rlhf_lib.py: Complete reference implementation.
  • tests/test_pretraining_fine_tuning_and_rlhf_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/pretraining_fine_tuning_and_rlhf_lib.py

What the commands do

  • Evaluates Bradley-Terry preference probabilities.
  • Computes DPO loss and implicit reward margins on test tensors.
  • Runs unit test assertions.

Expected output

Alignment Demo: BT Prob = 0.973, DPO Loss = 0.4741, Mean Margin = 0.4000

Validation steps

  1. Verify compute_bradley_terry_probability returns $0.5$ when rewards are equal.
  2. Confirm compute_dpo_loss yields $\approx \ln(2) \approx 0.6931$ when policy equals reference.
  3. Confirm implicit reward margins increase as policy likelihood on chosen tokens improves.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Underflow in Bradley-Terry sigmoid: Use math.exp(-diff) safely clamped within $[-20, 20]$.

Security notes

All alignment calculations execute locally in process memory.

Extension exercises

  1. Implement the Kahneman-Tversky Optimization (KTO) loss function.
  2. Build an automated prompt-masking dataset collator for SFT.

Expected output

FIELDS.md

# Expected Output Fields: Day 240

- `BT Prob`: Bradley-Terry pairwise preference probability.
- `DPO Loss`: Direct Preference Optimization loss value.
- `Mean Margin`: Implicit reward margin between chosen and rejected responses.

examples-run.txt

Alignment Demo: BT Prob = 0.973, DPO Loss = 0.4741, Mean Margin = 0.4000

measured-values.txt

BT Prob: 0.973
DPO Loss: 0.4741
Mean Margin: 0.4000

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_pretraining_fine_tuning_and_rlhf_lib.py::test_bradley_terry_probabilities PASSED [ 33%]
tests/test_pretraining_fine_tuning_and_rlhf_lib.py::test_dpo_loss_mechanics PASSED [ 66%]
tests/test_pretraining_fine_tuning_and_rlhf_lib.py::test_dpo_loss_symmetry_when_identical PASSED [100%]

============================== 3 passed in 0.42s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/pretraining_fine_tuning_and_rlhf_lib.py (1726 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from typing import Tuple

def compute_bradley_terry_probability(reward_chosen: float, reward_rejected: float) -> float:
    # P(y_w > y_l) = 1 / (1 + exp(-(r_w - r_l))) = sigmoid(r_w - r_l)
    diff = reward_chosen - reward_rejected
    return float(1.0 / (1.0 + math.exp(-diff)))

def compute_dpo_loss(
    policy_chosen_logp: torch.Tensor,
    policy_rejected_logp: torch.Tensor,
    ref_chosen_logp: torch.Tensor,
    ref_rejected_logp: torch.Tensor,
    beta: float = 0.1
) -> Tuple[torch.Tensor, torch.Tensor]:
    pi_logratios = policy_chosen_logp - policy_rejected_logp
    ref_logratios = ref_chosen_logp - ref_rejected_logp

    logits = beta * (pi_logratios - ref_logratios)
    loss = -F.logsigmoid(logits).mean()

    # Implicit reward margin: beta * (log(pi_w/ref_w) - log(pi_l/ref_l))
    implicit_margin = (beta * (policy_chosen_logp - ref_chosen_logp) - 
                       beta * (policy_rejected_logp - ref_rejected_logp))

    return loss, implicit_margin

def run_alignment_demo():
    bt_prob = compute_bradley_terry_probability(2.4, -1.2)
    
    # Simulated sequence log-probabilities (batch of 2)
    pol_chosen = torch.tensor([-12.4, -10.2])
    pol_rejected = torch.tensor([-18.2, -16.5])
    ref_chosen = torch.tensor([-12.3, -10.1])
    ref_rejected = torch.tensor([-14.1, -13.0])

    loss, margin = compute_dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1)

    print(f"Alignment Demo: BT Prob = {bt_prob:.3f}, DPO Loss = {loss.item():.4f}, Mean Margin = {margin.mean().item():.4f}")
    return bt_prob, loss.item(), margin.mean().item()

if __name__ == "__main__":
    run_alignment_demo()
examples/test_pretraining_fine_tuning_and_rlhf_lib.py (1260 bytes)
import pytest
import torch
from examples.pretraining_fine_tuning_and_rlhf_lib import (
    compute_bradley_terry_probability,
    compute_dpo_loss
)

def test_bradley_terry_probabilities():
    p_equal = compute_bradley_terry_probability(1.0, 1.0)
    p_higher = compute_bradley_terry_probability(2.4, -1.2)
    p_lower = compute_bradley_terry_probability(-1.0, 2.0)

    assert p_equal == pytest.approx(0.5, rel=1e-3)
    assert p_higher > 0.95
    assert p_lower < 0.10

def test_dpo_loss_mechanics():
    # Setup batch where policy strongly prefers chosen over reference
    pol_chosen = torch.tensor([-10.0])
    pol_rejected = torch.tensor([-20.0])
    ref_chosen = torch.tensor([-10.0])
    ref_rejected = torch.tensor([-15.0])

    loss, margin = compute_dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1)
    
    assert loss.item() > 0.0
    assert margin.item() > 0.0

def test_dpo_loss_symmetry_when_identical():
    # If policy matches reference, logits = 0 -> loss = -log(sigmoid(0)) = log(2) ≈ 0.6931
    logps = torch.tensor([-10.0])
    loss, margin = compute_dpo_loss(logps, logps, logps, logps, beta=0.1)
    assert loss.item() == pytest.approx(0.6931, rel=1e-3)
    assert margin.item() == pytest.approx(0.0, abs=1e-5)
metadata.yml (445 bytes)
lesson_id: D240
day: 240
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/pretraining_fine_tuning_and_rlhf_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/pretraining_fine_tuning_and_rlhf_lib.py (530 bytes)
import torch
import torch.nn as nn
from typing import Tuple

def compute_bradley_terry_probability(reward_chosen: float, reward_rejected: float) -> float:
    # TODO: Calculate P(chosen > rejected) using logistic sigmoid
    pass

def compute_dpo_loss(
    policy_chosen_logp: torch.Tensor,
    policy_rejected_logp: torch.Tensor,
    ref_chosen_logp: torch.Tensor,
    ref_rejected_logp: torch.Tensor,
    beta: float = 0.1
) -> Tuple[torch.Tensor, torch.Tensor]:
    # TODO: Compute DPO loss and implicit reward margin
    pass
starter/test_pretraining_fine_tuning_and_rlhf_lib.py (1260 bytes)
import pytest
import torch
from examples.pretraining_fine_tuning_and_rlhf_lib import (
    compute_bradley_terry_probability,
    compute_dpo_loss
)

def test_bradley_terry_probabilities():
    p_equal = compute_bradley_terry_probability(1.0, 1.0)
    p_higher = compute_bradley_terry_probability(2.4, -1.2)
    p_lower = compute_bradley_terry_probability(-1.0, 2.0)

    assert p_equal == pytest.approx(0.5, rel=1e-3)
    assert p_higher > 0.95
    assert p_lower < 0.10

def test_dpo_loss_mechanics():
    # Setup batch where policy strongly prefers chosen over reference
    pol_chosen = torch.tensor([-10.0])
    pol_rejected = torch.tensor([-20.0])
    ref_chosen = torch.tensor([-10.0])
    ref_rejected = torch.tensor([-15.0])

    loss, margin = compute_dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1)
    
    assert loss.item() > 0.0
    assert margin.item() > 0.0

def test_dpo_loss_symmetry_when_identical():
    # If policy matches reference, logits = 0 -> loss = -log(sigmoid(0)) = log(2) ≈ 0.6931
    logps = torch.tensor([-10.0])
    loss, margin = compute_dpo_loss(logps, logps, logps, logps, beta=0.1)
    assert loss.item() == pytest.approx(0.6931, rel=1e-3)
    assert margin.item() == pytest.approx(0.0, abs=1e-5)
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 240 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_pretraining_fine_tuning_and_rlhf_lib.py (1260 bytes)
import pytest
import torch
from examples.pretraining_fine_tuning_and_rlhf_lib import (
    compute_bradley_terry_probability,
    compute_dpo_loss
)

def test_bradley_terry_probabilities():
    p_equal = compute_bradley_terry_probability(1.0, 1.0)
    p_higher = compute_bradley_terry_probability(2.4, -1.2)
    p_lower = compute_bradley_terry_probability(-1.0, 2.0)

    assert p_equal == pytest.approx(0.5, rel=1e-3)
    assert p_higher > 0.95
    assert p_lower < 0.10

def test_dpo_loss_mechanics():
    # Setup batch where policy strongly prefers chosen over reference
    pol_chosen = torch.tensor([-10.0])
    pol_rejected = torch.tensor([-20.0])
    ref_chosen = torch.tensor([-10.0])
    ref_rejected = torch.tensor([-15.0])

    loss, margin = compute_dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1)
    
    assert loss.item() > 0.0
    assert margin.item() > 0.0

def test_dpo_loss_symmetry_when_identical():
    # If policy matches reference, logits = 0 -> loss = -log(sigmoid(0)) = log(2) ≈ 0.6931
    logps = torch.tensor([-10.0])
    loss, margin = compute_dpo_loss(logps, logps, logps, logps, beta=0.1)
    assert loss.item() == pytest.approx(0.6931, rel=1e-3)
    assert margin.item() == pytest.approx(0.0, abs=1e-5)

Troubleshooting

Troubleshooting: Day 240 - Pretraining, Fine-Tuning, and RLHF

Common Issues

  1. DPO Loss divergence:
    • Cause: $\beta$ parameter set too high ($> 1.0$).
    • Fix: Keep $\beta$ in range $[0.05, 0.20]$.

Security notes

Security & Privacy: Day 240 - Pretraining, Fine-Tuning, and RLHF

Security Guidance

  • All alignment optimization loops execute locally in memory.