LLMs and Generative AI β€Ί The LLM Landscape β€Ί Day 240

Day 240: Pretraining, Fine-Tuning, and RLHF

Day 240 of 365 β€” Pretraining, Fine-Tuning, and RLHF

Master the complete three-stage training pipeline of modern language models: Causal Pretraining (acquiring broad world knowledge), Supervised Fine-Tuning (SFT on instruction-response dialogues), and Reinforcement Learning from Human Feedback (RLHF). Compare classical PPO with modern Direct Preference Optimization (DPO), analyze the Bradley-Terry preference model, master KL-divergence drift penalties, and build a working DPO preference loss step in PyTorch.

Course
LLMs and Generative AI
Category
The LLM Landscape
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-240-pretraining-fine-tuning-and-rlhf

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-240-pretraining-fine-tuning-and-rlhf
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

A raw pretrained base model (like LLaMA-3-8B-Base) is brilliant but completely unusable as a software product.

If you send a base model the prompt:

β€œWrite a Python function to compute the Fibonacci sequence.”

The base model might reply with:

β€œWrite a Python function to compute factorial. Write a Python function to reverse a linked list.”

Why? Because the base model was trained on the entire internet to be an unbiased statistical text completion engine. It recognizes that your prompt looks like the first question on a computer science homework sheet, so it autocompletes questions 2 and 3!

To turn a raw completion engine into ChatGPT, Claude, or Gemini, frontier laboratories apply a Three-Stage Post-Training Alignment Pipeline:

  1. Pretraining: Ingesting 15 Trillion tokens to acquire broad world knowledge.
  2. Supervised Fine-Tuning (SFT): Teaching the model the concept of being a helpful assistant that answers questions.
  3. Reinforcement Learning from Human/AI Feedback (RLHF & DPO): Aligning the model with human valuesβ€”politeness, truthfulness, conciseness, and safety.

Today, you will master the mathematics and engineering of SFT prompt masking, classical PPO reinforcement learning, and modern Direct Preference Optimization (DPO).


The idea in plain language

Diagram: the three post-training stages that turn a base model into an assistant

Think of raising a brilliant doctor:

Medical school creates the doctor; bedside training and ethical alignment make them trusted to practice medicine.


Historical background

Diagram: the historical progression of alignment techniques

Three-stage LLM training alignment pipeline showing pretraining, supervised fine-tuning, and preference alignment

  1. 2017 (Christiano et al., OpenAI & DeepMind): Published Deep Reinforcement Learning from Human Preferences, establishing trajectory ranking and reward modeling on Atari and robotic games.
  2. 2020 (Ziegler et al. & Stiennon et al.): Applied PPO reinforcement learning to fine-tune GPT-2 for text summarization from human feedback.
  3. 2022 (Ouyang et al. & InstructGPT): OpenAI released InstructGPT, demonstrating that a 1.3B parameter model aligned with SFT and PPO was preferred by humans over a 175B raw GPT-3 base model!
  4. 2022 (Bai et al. & Anthropic): Introduced Constitutional AI (RLAIF), using AI self-critique guided by written principles to automate alignment.
  5. 2023 (Rafailov et al. & DPO): Stanford researchers proved that policy optimization could be solved in closed-form without training a separate reward model (Direct Preference Optimization), revolutionizing open-source alignment.

What it is β€” and what it is not

What Post-Training Alignment IS:

What it is NOT:


Why it was created and what problems it solves

Alignment transforms raw autocomplete models into reliable software products:

  1. Instruction Followability: Enables models to execute complex multi-step user prompts without wandering off topic.
  2. Harm Reduction & Jailbreak Defense: Prevents models from generating toxic, biased, or dangerous content.
  3. Conciseness and User Utility: Teaches models to eliminate redundant rambling and format outputs with structured Markdown, JSON, and code blocks.

How it works

Let us dissect Supervised Fine-Tuning, Classical PPO RLHF, and Direct Preference Optimization.


1. Supervised Fine-Tuning (SFT) & Prompt Masking

During SFT, models are trained on $100,000$ to $1,000,000$ high-quality multi-turn conversations:

<|im_start|>user
Write a Python function to check prime numbers.<|im_end|>
<|im_start|>assistant
def is_prime(n: int) -> bool:
    if n <= 1:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True<|im_end|>

The Prompt-Loss Masking Rule:

When computing Cross-Entropy loss $\mathcal[L]_[SFT]$, we set the target label for all User Prompt Tokens to -100 (PyTorch’s default ignore_index):

\mathcal{L}_{\text{SFT}}(\theta) = - \sum_{t \in \text{Assistant Tokens}} \log P_\theta(w_t \mid w_{<t})

2. Classical RLHF with PPO (The 4-Model Cluster)

Comparison architecture diagram between PPO multi-model reinforcement learning and DPO closed-form direct preference optimization

Classical RLHF (InstructGPT, LLaMA-2) executes in 3 sequential steps:

  1. Preference Dataset Collection: Human annotators are shown prompt $x$ and two candidate responses $(y_w, y_l)$. They select $y_w$ as the β€œwinner” (preferred) and $y_l$ as the β€œloser”.

  2. Reward Model Training: A scalar reward model $r_\psi(x, y)$ is trained using the Bradley-Terry Preference Loss:

    \mathcal{L}_{\text{Reward}}(\psi) = - \mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\left( r_\psi(x, y_w) - r_\psi(x, y_l) \right) \right]
  3. PPO Reinforcement Learning: The policy $\pi_\theta$ generates candidate completions and receives reward $r_\psi(x, y)$ penalized by the KL-divergence drift from the reference model $\pi_[ref]$:

    \max_\theta \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ r_\psi(x, y) - \beta D_{\text{KL}}(\pi_\theta(y \mid x) \parallel \pi_{\text{ref}}(y \mid x)) \right]

3. Direct Preference Optimization (DPO, 2023)

Diagram: reinforcement-based preference training compared with direct preference optimisation

Why maintain 4 massive neural networks in GPU VRAM (Actor $\pi_\theta$, Critic $V_\phi$, Reward $r_\psi$, Reference $\pi_[ref]$) and suffer unstable PPO actor-critic dynamics?

Rafailov et al. (2023) proved that the ground-truth optimal policy $\pi^*$ under the KL-constrained RLHF objective satisfies an exact mathematical relationship with the reward function:

r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)

By substituting this identity directly into the Bradley-Terry preference objective, the partition function $Z(x)$ cancels out completely!

The DPO Loss Function:

\mathcal{L}_{\text{DPO}}(\theta; \pi_{\text{ref}}) = - \mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]

4. RLAIF & Constitutional AI (Anthropic, 2022)

Human preference labeling is slow, expensive, and noisy. RLAIF (Reinforcement Learning from AI Feedback) replaces human labelers with automated AI critiques:


5. Monolithic Alignment: ORPO and KTO (2024)

Can we eliminate the two-stage SFT $\rightarrow$ DPO workflow entirely?


6. Reward Model Exploitation: The Length Bias Trap

Diagram: how a model exploits imperfections in its reward signal

A notorious failure mode in RLHF and DPO is Verbosity Gaming:


7. The Alignment Faking & Sycophancy Frontier

Diagram: how preference training produces excessive agreement

Recent empirical research (Anthropic, 2024) has uncovered subtle alignment failure modes:


8. SimPO: Reference-Free Simple Preference Optimization (2024)

Building on DPO, SimPO (Meng et al., 2024) strips away even more complexity:


9. ChatML & Jinja Template Standards for SFT

Diagram: how conversation structure is encoded as tokens for a chat model

In modern multi-turn dialogues, models rely on strict Chat Templates:


An everyday analogy

Diagram: learning by demonstration compared with learning by preference feedback

Think of training an Olympic gymnast:


Examples in practice

Let us implement Direct Preference Optimization (DPO) Loss in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class DPOTrainer:
    def __init__(self, beta: float = 0.1):
        self.beta = beta

    def compute_dpo_loss(
        self,
        policy_chosen_logps: torch.Tensor,
        policy_rejected_logps: torch.Tensor,
        reference_chosen_logps: torch.Tensor,
        reference_rejected_logps: torch.Tensor
    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        # Calculates DPO loss on batch of chosen (winning) and rejected (losing) responses.
        # Loss = -log(sigmoid(beta * (log(pi_chosen / ref_chosen) - log(pi_rejected / ref_rejected))))
        pi_logratios = policy_chosen_logps - policy_rejected_logps
        ref_logratios = reference_chosen_logps - reference_rejected_logps
        
        logits = self.beta * (pi_logratios - ref_logratios)
        
        # DPO Loss
        loss = -F.logsigmoid(logits).mean()
        
        # Implicit rewards for telemetry tracking
        chosen_rewards = self.beta * (policy_chosen_logps - reference_chosen_logps).detach()
        rejected_rewards = self.beta * (policy_rejected_logps - reference_rejected_logps).detach()
        
        return loss, chosen_rewards, rejected_rewards

Implications: security, privacy, performance, scalability, and cost

Diagram: the safety, labour, values and capability implications of alignment training

  1. Alignment Tax (Performance Degradation):
    • Aggressive RLHF safety tuning often degrades a model’s raw mathematical reasoning and coding creativity (The Alignment Tax). Modern labs use balanced multi-objective preference datasets to mitigate this penalty.
  2. Jailbreaks and Adversarial Suffix Attacks:
    • Adversarial strings (e.g. GCG suffix attacks) can force aligned models into answering forbidden queries by exploiting low-probability regions in the output manifold.

Alternatives: free, open source, and commercial

Alignment AlgorithmReward Model Needed?Online Sampling?Computational Complexity
PPO (Classical RLHF)Yes (Explicit)Yes (In Loop)Very High (4 Model Replicas)
DPO (Rafailov et al.)No (Implicit)No (Offline)Low (2 Model Replicas)
KTO (Kahneman-Tversky)NoNo (Unpaired Data)Low (Binary Yes/No Labels)
ORPO (Monolithic)NoNoMinimal (SFT + DPO in 1 Step)

Diagram: supervised fine-tuning compared with preference optimisation

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   ALIGNMENT PARADIGM COMPARISON                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Dimension          β”‚ SFT             β”‚ PPO              β”‚ DPO          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Data Format        β”‚ Prompt -> Targetβ”‚ Prompt -> Reward β”‚ Prompt -> y_wβ”‚
β”‚ Loss Function      β”‚ Cross-Entropy   β”‚ Policy Gradient  β”‚ Implicit BT  β”‚
β”‚ Stability          β”‚ Rock Solid      β”‚ Fragile (Tuning) β”‚ Highly Stableβ”‚
β”‚ Memory Footprint   β”‚ 1 Model         β”‚ 4 Models         β”‚ 2 Models     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when to run your own alignment training and when to use an aligned model

When to Use SFT & DPO:

When NOT to use:


The AI connection

Knowledge check

  1. Why do we mask prompt token losses to -100 during Supervised Fine-Tuning?
  2. What is the fundamental difference between PPO and DPO alignment?
  3. How does the Bradley-Terry model relate scalar rewards to pairwise human preference probabilities?
  4. What role does the KL-divergence penalty parameter $\beta$ play in preventing reward hacking?
  5. How does Constitutional AI (RLAIF) scale preference dataset creation without human annotators?

Hands-on exercise

In this lab, you will build and test a complete preference alignment module in PyTorch: implement the Bradley-Terry preference probability function, compute Direct Preference Optimization (DPO) loss across winning and losing response tensors, verify implicit reward margin separation, and calculate the KL divergence drift penalty.

Expected output

[LLM Post-Training & Alignment Verification Suite]
Testing Bradley-Terry Preference Probability:
  Reward Chosen: 2.40, Reward Rejected: -1.20
  Preference Probability P(chosen > rejected): 0.973 [CORRECT]
Testing Direct Preference Optimization (DPO) Loss Step:
  Initial Log-Probabilities:
    Policy Chosen: -12.4, Policy Rejected: -18.2
    Reference Chosen: -12.3, Reference Rejected: -14.1
  DPO Loss: 0.2834 (Beta = 0.10)
  Implicit Reward Margin (Chosen - Rejected): +0.400 [MARGIN SEPARATED]
Test Suite: 3 passed in 0.42s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement an automated SFT Prompt-Masking Collate Function that accepts conversational dictionaries and generates input IDs with -100 target masks on all user tokens.
  2. Build an Online Reward Monitor that plots the implicit reward distribution across training epochs.

Extension challenge

Implement Kahneman-Tversky Optimization (KTO):

Quiz

Q1. Why is Prompt-Loss Masking standard practice during Supervised Fine-Tuning (SFT)?

  1. Because PyTorch cannot compute loss on text with questions
  2. Because prompt tokens are secret
  3. Otherwise it learns to write prompts, not answers
  4. Because prompt tokens use too much GPU memory
Show answer

Answer: C. Otherwise it learns to write prompts, not answers

SFT masks prompt tokens with label -100 so the model only learns the assistant response distribution.

Q2. What was the foundational mathematical breakthrough of Direct Preference Optimization (DPO, Rafailov et al., 2023)?

  1. DPO replaced PyTorch with Excel spreadsheets
  2. DPO discovered how to run neural networks without electricity
  3. DPO proved that human feedback is unnecessary
  4. The optimal policy has a closed form, so no reward model
Show answer

Answer: D. The optimal policy has a closed form, so no reward model

DPO eliminates the reward model and PPO loop by optimizing the policy directly with an exact implicit reward formulation.

Q3. Why is a KL-Divergence penalty (KL(pi_theta || pi_ref)) enforced during RLHF and DPO?

  1. To stop reward hacking and drift from the reference
  2. To make the model run faster
  3. To reduce vocabulary size
  4. To compress model weights into 4-bit integers
Show answer

Answer: A. To stop reward hacking and drift from the reference

The KL penalty keeps the policy close to the reference base/SFT model, preventing reward gaming and degenerative repetition.

Q4. What is the Bradley-Terry preference model used in pairwise human evaluation?

  1. A model predicting GPU temperatures during training
  2. Preference probability is a sigmoid of the reward gap
  3. A tokenizer format developed in 1952
  4. A method for calculating cloud hosting bills
Show answer

Answer: B. Preference probability is a sigmoid of the reward gap

Bradley-Terry models pairwise choices via the logistic sigmoid of scalar reward differences.

Q5. What is "RLAIF" (Reinforcement Learning from AI Feedback / Constitutional AI)?

  1. A hardware chip designed for reinforcement learning
  2. Training robots to play video games
  3. A model critiques and ranks against a constitution
  4. Using random numbers to train models
Show answer

Answer: C. A model critiques and ranks against a constitution

RLAIF uses strong AI models (e.g. Claude Constitutional AI) to provide scalable automated preference labels.

Glossary

Supervised Fine-Tuning (SFT)
Training a pretrained base model on curated instruction-response pairs with prompt-token loss masking to teach it to follow user directives.
RLHF
Reinforcement Learning from Human Feedback: optimizing an LLM policy against a human-trained reward model using reinforcement learning algorithms like PPO.
Direct Preference Optimization (DPO)
An alignment algorithm that optimizes policy weights directly on preference pairs (winning vs losing responses) without training an explicit reward model.
Bradley-Terry Model
A probabilistic framework modeling the probability of pairwise human preference as the sigmoid of the difference in scalar rewards.
KL-Divergence Penalty
A regularization term in alignment loss functions penalizing the active policy for drifting too far from the reference SFT model distribution.
Reward Hacking
A failure mode where an RL policy discovers degenerate phrases or formatting tricks that score artificially high rewards without providing actual utility.
Prompt-Loss Masking
Setting the loss targets for prompt/user tokens to -100 during SFT so backpropagation only updates weights based on assistant response tokens.
RLAIF / Constitutional AI
Using AI models to critique and score candidate responses against a set of written principles (constitution) to generate scalable preference datasets.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.