LLMs and Generative AI › Multimodal and Generative Media › Day 281

Day 281: How Diffusion Models Generate Images

Day 281 of 365 — How Diffusion Models Generate Images

Master the foundational mathematics and algorithmic mechanics of Denoising Diffusion Probabilistic Models (DDPM): derive the closed-form forward noise schedule q(xt|x0), understand the reverse denoising Markov transitions p_theta(xt-1|xt), analyze U-Net and Diffusion Transformer (DiT) architectures, and implement a complete noise scheduler in Python.

Course
LLMs and Generative AI
Category
Multimodal and Generative Media
Reading time
≈ 35 min
Practical time
≈ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-281-how-diffusion-models-generate-images

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-281-how-diffusion-models-generate-images
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

For decades, generative image modeling was dominated by Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). While GANs could produce sharp images in a single forward pass, training them was notoriously unstable—minimax game dynamics frequently collapsed into mode collapse, where the generator learned only a handful of repetitive patterns.

In 2020, Jonathan Ho et al. published Denoising Diffusion Probabilistic Models (DDPM), completely overturning computer vision. Diffusion models reframed image generation not as a one-shot guess, but as a gradual thermodynamic process:

  1. Take a clean image and systematically destroy its structure by adding tiny amounts of Gaussian noise over 1,000 discrete timesteps until it becomes pure, unrecognizable static (Forward Diffusion).
  2. Train a deep neural network (a U-Net or Diffusion Transformer) to look at a noisy image at timestep $t$ and predict the exact noise vector that was added.
  3. Start from pure random white noise and iteratively subtract predicted noise step-by-step to reveal a photorealistic image from the void (Reverse Diffusion).

Today, diffusion models power the entire state of the art in generative media—including Stable Diffusion, Midjourney, DALL-E 3, Sora, and Flux.

Forward Diffusion Noise Schedule Pipeline showing closed-form Gaussian noise addition from clean image x0 to pure noise xT

The idea in plain language

Diagram: the four steps of diffusion-based generation

Imagine you are a sculptor carving an intricate marble statue of a lion:

Historical background

Diagram: the historical progression of generative image models

The theoretical foundation of diffusion models originated in non-equilibrium thermodynamics. In 2015, Jascha Sohl-Dickstein et al. introduced the concept of using Markov chains to slowly destroy structure in a data distribution and learning the reverse transitions. However, early thermodynamic models were computationally heavy and produced low-resolution samples.

In 2020, Jonathan Ho, Ajay Jain, and Pieter Abbeel revolutionized the field with DDPM. By establishing an exact mathematical equivalence between diffusion models and denoising score matching (Langevin dynamics), they proved that training the neural network to predict the injected noise $\epsilon$ using simple Mean Squared Error (MSE) loss produced breathtaking image fidelity that rivaled GANs without any training instability.

In 2021, Robin Rombach et al. introduced Latent Diffusion Models (LDMs), the architecture behind Stable Diffusion. By training a Variational Autoencoder (VAE) to compress images into a low-dimensional latent space, diffusion was shifted from expensive pixel space ($512 \times 512 \times 3$) to compressed latent space ($64 \times 64 \times 4$), slashing compute requirements by $64\times$ and allowing state-of-the-art generation on consumer GPUs.

In 2023–2024, Diffusion Transformers (DiT) replaced convolutional U-Nets with scalable Vision Transformer backbones, powering next-generation models like Stable Diffusion 3 and Flux.

What it is — and what it is not

Let us formalize the technical boundaries of diffusion models:

What Diffusion Models Are:

What Diffusion Models Are Not:

+-------------------------------------------------------------------------------+
|                    GENERATIVE IMAGE PARADIGM COMPARISON                       |
+---------------------+--------------------+--------------------+---------------+
| Dimension           | GANs               | VAEs               | Diffusion     |
+---------------------+--------------------+--------------------+---------------+
| Generation Speed    | 1 Step (Instant)   | 1 Step (Instant)   | 20 - 50 Steps |
| Training Stability  | Very Fragile (Minimax)| Stable (ELBO)   | Rock Solid    |
| Sample Diversity    | Poor (Mode Collapse)| High               | State-of-Art  |
| Visual Sharpness    | Very High          | Low (Blurry)       | Pristine      |
| Text Controllability| Difficult          | Moderate           | Flawless      |
+---------------------+--------------------+--------------------+---------------+

Why it was created and what problems it solves

Diffusion architectures solve the core mathematical limitations of prior generative vision paradigms:

1. Eliminating Adversarial Mode Collapse (Solved vs GANs)

GANs require training two competing networks (Generator vs Discriminator). If the Discriminator overpowers the Generator, gradients vanish; if the Generator finds a single image that fools the Discriminator, it produces that identical image forever (mode collapse). Diffusion models optimize a stationary, convex Mean Squared Error loss over noise estimation, completely eliminating adversarial instability.

Diagram: adversarial generative training compared with diffusion training

2. Eliminating Visual Blur (Solved vs VAEs)

Standard VAEs optimize pixel reconstruction via Mean Squared Error in image space, forcing the model to average over ambiguous high-frequency details, resulting in blurry edges. Diffusion models predict noise distributions iteratively, allowing high-frequency textures (hair, pores, fabric weave) to resolve crisply across timesteps.

3. Deep Text Conditioning via Cross-Attention

Because diffusion operates over multi-step refinement, textual conditioning embeddings (from CLIP or T5 encoders) can inject guidance into cross-attention layers at every single denoising step, enabling unprecedented text-to-image prompt fidelity.

Diagram: unconditional generation compared with text-conditioned generation

How it works

Let us derive the mathematics governing the forward and reverse diffusion processes:

[Clean Image x0]
       │
       ├───> Forward Noise Schedule: q(xt | x0) = N(xt; sqrt(alpha_bar_t)*x0, (1 - alpha_bar_t)*I)
       │
       ▼
[Noisy Image xt at Timestep t]
       │
       ├───> Input to U-Net / DiT + Timestep Embedding t + Text Conditioning c
       │
       ▼
[Predicted Noise epsilon_theta(xt, t, c)]
       │
       ▼
[Compute Loss]: L_simple = || epsilon - epsilon_theta(xt, t, c) ||^2

1. The Closed-Form Forward Process $q(x_t | x_0)$

In the forward process, small Gaussian noise with variance $\beta_t \in (0, 1)$ is added at each step: q(x_t | x_(t-1)) = mathcal(N)(x_t; sqrt(1 - \beta_t) x_(t-1), \beta_t I)

Diagram: how noise is added to an image in the forward diffusion process

Let $\alpha_t = 1 - \beta_t$ and \bar(\alpha)_t = prod_(i=1)^t \alpha_i. Using the property that the sum of independent Gaussians is Gaussian, we can express $x_t$ directly as a function of the original image $x_0$: x_t = sqrt(\bar(\alpha)_t) x_0 + sqrt(1 - \bar(\alpha)_t) epsilon, quad \text(where ) epsilon sim mathcal(N)(0, I)

This closed-form formulation is monumental for training: we can sample an arbitrary timestep $t \in [1, 1000]$, compute \bar(\alpha)_t, and noise the image in a single vector operation without stepping through $t-1$ intermediate states!

2. The Training Objective (L_(\text(simple)))

During training:

Diagram: the four steps of the diffusion training objective

  1. Sample a clean image $x_0 \sim q(x_0)$.
  2. Sample a random timestep t sim \text(Uniform)(1, T).
  3. Sample random Gaussian noise epsilon sim mathcal(N)(0, I).
  4. Construct noisy latent x_t = sqrt(\bar(\alpha)_t) x_0 + sqrt(1 - \bar(\alpha)_t) epsilon.
  5. Compute the model’s noise prediction: hat(epsilon) = epsilon_\theta(x_t, t).
  6. Optimize the simple Mean Squared Error: $$L_(\text(simple))(\theta) = \mathbb(E)(t, x_0, \epsilon) \left[ | \epsilon - \epsilon\theta(x_t, t) |^2 ight]$$

3. The Reverse Sampling Algorithm (DDPM Sampling)

During inference, we start from pure noise x_T sim mathcal(N)(0, I) and step backwards from $t = T$ down to $t = 1$: $$x_(t-1) = \frac(1)(\sqrt(\alpha_t)) \left( x_t - \frac(1 - \alpha_t)(\sqrt(1 - \bar(\alpha)t)) \epsilon\theta(x_t, t) ight) + \sigma_t z$$ where z sim mathcal(N)(0, I) for $t > 1$ and $z = 0$ for $t = 1$.

Diagram: the four steps of one reverse diffusion sampling iteration

Reverse Denoising UNet Architecture demonstrating neural noise prediction and iterative denoising steps

4. Noise Schedules: Linear vs Cosine

Diagram: how the noise schedule shapes image destruction during diffusion

An everyday analogy

Diagram: creating an image from scratch compared with restoring one from damage

Think of diffusion like cleaning an ancient mosaic tile floor covered in layers of mud:

Examples in practice

Let us inspect how developers implement noise scheduling and sampling in Python:

1. Vectorized Forward Noise Injection in Python

import numpy as np

class GaussianDiffusionScheduler:
    def __init__(self, timesteps: int = 1000, beta_start: float = 0.0001, beta_end: float = 0.02):
        self.timesteps = timesteps
        self.betas = np.linspace(beta_start, beta_end, timesteps, dtype=np.float32)
        self.alphas = 1.0 - self.betas
        self.alphas_cumprod = np.cumprod(self.alphas)

    def add_noise(self, original_samples: np.ndarray, noise: np.ndarray, timesteps: np.ndarray) -> np.ndarray:
        """Implements closed-form forward diffusion: x_t = sqrt(alpha_bar_t)*x0 + sqrt(1 - alpha_bar_t)*noise"""
        sqrt_alpha_prod = np.sqrt(self.alphas_cumprod[timesteps])
        sqrt_one_minus_alpha_prod = np.sqrt(1.0 - self.alphas_cumprod[timesteps])
        
        # Reshape for broadcast across image dimensions (Batch, Channels, Height, Width)
        sqrt_alpha_prod = sqrt_alpha_prod.reshape(-1, 1, 1, 1)
        sqrt_one_minus_alpha_prod = sqrt_one_minus_alpha_prod.reshape(-1, 1, 1, 1)
        
        return sqrt_alpha_prod * original_samples + sqrt_one_minus_alpha_prod * noise

2. Reverse Step Computation

    def step_denoise(self, model_output_noise: np.ndarray, timestep: int, sample: np.ndarray) -> np.ndarray:
        alpha = self.alphas[timestep]
        alpha_cumprod = self.alphas_cumprod[timestep]
        beta = self.betas[timestep]
        
        # 1. Compute predicted original sample x0 approximation
        pred_original_sample = (sample - np.sqrt(1.0 - alpha_cumprod) * model_output_noise) / np.sqrt(alpha_cumprod)
        
        # 2. Compute mean of reverse distribution mu_t
        pred_prev_sample = (1.0 / np.sqrt(alpha)) * (sample - (beta / np.sqrt(1.0 - alpha_cumprod)) * model_output_noise)
        
        if timestep > 0:
            noise = np.random.randn(*sample.shape).astype(np.float32)
            variance = np.sqrt(beta) * noise
            pred_prev_sample = pred_prev_sample + variance
            
        return pred_prev_sample

Implications: security, privacy, performance, scalability, and cost

Diagram: the quality, cost, misuse and provenance implications of diffusion models

1. Security & Deepfakes

Because diffusion models produce hyper-realistic human faces and documents, modern generative pipelines embed cryptographic watermarks (such as C2PA metadata or SynthID latent perturbations) to verify authenticity and track synthetic provenance.

2. Performance & Sampling Latency

Evaluating a deep U-Net 50 times in sequence introduces latency:

3. Compute Footprint

Pixel-space diffusion ($1024 \times 1024 \times 3$) requires clusters of high-end NVIDIA H100 GPUs. Latent diffusion ($128 \times 128 \times 4$) runs comfortably on consumer MacBooks and prosumer NVIDIA RTX 4070/4080 cards.

Alternatives: free, open source, and commercial

ArchitectureParadigmSample QualityGeneration SpeedCompute Cost
Latent Diffusion (SDXL / Flux)Latent DenoisingSOTA (Photorealistic)Fast (20-30 steps)Consumer GPU
Diffusion Transformers (DiT)Transformer DiffusionSOTA (Compositional)ModerateServer GPU
StyleGAN3Adversarial GANSharp (Narrow domain)Instant (1 step)Low Inference
VQ-VAE-2 / DALL-E 1Autoregressive TokensModerate (Pixel artifacts)Slow (Sequential)High
Consistency ModelsDistilled 1-Step DiffGoodInstant (1-2 steps)Very Low
+-----------------------+-----------------------+-----------------------+
| Attribute             | Pixel-Space Diffusion | Latent Diffusion (LDM)|
+-----------------------+-----------------------+-----------------------+
| Operating Resolution  | 512 x 512 x 3 (786k)  | 64 x 64 x 4 (16k)     |
| Compression Factor    | 1x (None)             | 48x Spatial Reduction |
| VRAM Requirement      | 24 GB - 80 GB         | 6 GB - 12 GB          |
| Fine Texture Detail   | Prone to noise drift  | Crystal Clear         |
| Training Throughput   | Slow (Heavy GEMM)     | 8x Faster Convergence |
+-----------------------+-----------------------+-----------------------+

When to use it — and when not to

Diagram: when diffusion models are appropriate

Choose Diffusion Models when:

Do NOT use Diffusion Models when:

The AI connection

Knowledge check

  1. Why is the forward diffusion process computable in closed form? Because Gaussian noise added over multiple Markov steps compounds linearly: x_t = sqrt(\bar(\alpha)_t) x_0 + sqrt(1 - \bar(\alpha)_t) epsilon.
  2. What does the neural network predict in DDPM? It predicts the injected noise vector $\epsilon$ from the noisy input $x_t$ and timestep embedding $t$.
  3. What is the spatial reduction factor of an 8x VAE in Latent Diffusion? A $512 \times 512 \times 3$ image (786,432 values) compresses to $64 \times 64 \times 4$ (16,384 values), achieving a $48\times$ numerical compression factor.

Hands-on exercise

In this lab, you will implement a Complete Gaussian Diffusion Noise Scheduler and Reverse Denoising Simulator in Python and NumPy. Your implementation will:

  1. Compute linear and cosine variance schedules (\beta_t, \alpha_t, \bar(\alpha)_t).
  2. Implement vectorized closed-form forward noise injection ($q(x_t | x_0)$).
  3. Implement a single-step reverse denoiser (p_\theta(x_(t-1) | x_t)).
  4. Run a multi-step reverse sampling loop starting from pure Gaussian noise.

Expected output

============================= test session starts ==============================
collected 6 items

tests/test_diffusion_scheduler.py ......                                 [100%]

============================== 6 passed in 0.04s ===============================
[DIFFUSION SCHEDULER] Timesteps: 1000 | Schedule: Linear (beta: 0.0001 -> 0.02)
[FORWARD DIFFUSION] Clean Tensor Shape: (4, 3, 32, 32)
  -> Injected Noise at Timestep t=500: alpha_bar = 0.5012
  -> SNR (Signal-to-Noise Ratio): 1.005 (Balanced Signal/Noise)
[REVERSE SAMPLING] Denoised 50 steps from Pure Noise x_T to Clean Sample x_0:
  -> Reconstruction MSE against synthetic target: 0.00031 (High Precision)

Validate your work

Run the pytest test suite to verify forward closed-form math, reverse sampling, and schedule transitions:

pytest tests/run_tests.sh

Troubleshooting

  1. Broadcast shape mismatch: Ensure sqrt_alpha_prod reshapes to (-1, 1, 1, 1) when broadcasting over 4D image tensors (B, C, H, W).
  2. Division by zero in reverse step: Add an epsilon guard when calculating 1 / sqrt(\alpha_t).

Common mistakes

Practice assignment

Implement a Cosine Variance Scheduler following Nichol and Dhariwal (2021) and compare its Signal-to-Noise Ratio (SNR) curve against the standard Linear schedule.

Extension challenge

Implement a DDIM (Denoising Diffusion Implicit Models) Deterministic Sampler in Python that allows skipping timesteps (e.g. taking 20 evenly-spaced steps from $T=1000$ to $t=0$) with zero stochastic noise injection.

Quiz

Q1. What mathematical property allows forward diffusion q(xt|x0) to sample directly at arbitrary timestep t without calculating intermediate steps?

  1. The cumulative product alpha_bar_t
  2. The Fourier Transform of discrete pixels
  3. Matrix inversion of the weight tensors
  4. Gradient descent backpropagation
Show answer

Answer: A. The cumulative product alpha_bar_t

Because the sum of two independent Gaussian variables is itself Gaussian, the composition of t Markov steps simplifies to a single closed-form distribution parameterized by alpha_bar_t.

Q2. What is the primary training objective of a DDPM U-Net / DiT network?

  1. Directly predicting the clean RGB image x0 from noise
  2. The exact noise added at timestep t
  3. Classifying the image into one of 1000 categories
  4. Generating text captions for input images
Show answer

Answer: B. The exact noise added at timestep t

DDPM trains the network epsilon_theta(xt, t) to predict the standardized noise vector epsilon using simple Mean Squared Error (MSE) loss: L = ||epsilon - epsilon_theta(xt, t)||^2.

Q3. Why do Latent Diffusion Models (like Stable Diffusion) perform diffusion in latent space rather than pixel space?

  1. Because latent diffusion models do not use GPUs
  2. Because RGB pixels cannot be noised
  3. A VAE compresses the image first
  4. Because latent spaces prevent copyright infringement
Show answer

Answer: C. A VAE compresses the image first

Operating in a compressed 4-channel latent space eliminates high-frequency pixel redundancy and enables fast consumer GPU generation.

Q4. What role do skip connections play in diffusion U-Net architectures?

  1. They skip training steps to save energy
  2. They delete corrupted noise tokens
  3. They bypass the self-attention mechanism
  4. They carry fine detail to the decoder
Show answer

Answer: D. They carry fine detail to the decoder

Skip connections concatenate high-resolution encoder features with upsampled decoder features, preventing blurriness and structural drift.

Q5. What is the Cosine Noise Schedule designed to fix compared to standard Linear Schedules?

  1. Cosine adds noise more gradually
  2. It speeds up file downloads
  3. It forces generated images to be circular
  4. It replaces U-Nets with CNNs
Show answer

Answer: A. Cosine adds noise more gradually

Nichol and Dhariwal (2021) demonstrated that linear schedules destroy image contrast in the first 20% of steps. Cosine schedules provide a smooth, near-linear decline in signal-to-noise ratio.

Glossary

Forward Diffusion
A Markov chain process that systematically adds Gaussian noise to an image over T steps until it becomes pure random noise.
Reverse Diffusion
A parameterized iterative process that learns to predict and remove noise at each timestep to reconstruct clean data.
Noise Schedule (Beta)
The sequence of variance parameters beta_1 to beta_T determining the magnitude of noise added at each diffusion step.
U-Net
A convolutional neural network architecture with encoder downsampling, bottleneck, and decoder upsampling connected by skip connections.
Diffusion Transformer (DiT)
A transformer architecture that replaces convolutional U-Nets for diffusion noise estimation on spatial patches.
Latent Diffusion Model (LDM)
A diffusion model that operates in the compressed latent space of a pretrained Variational Autoencoder (VAE) rather than pixel space.
Variational Lower Bound (ELBO)
The mathematical optimization objective used to train diffusion models by minimizing Kullback-Leibler divergence between forward and reverse transitions.
Classifier-Free Guidance (CFG)
A sampling technique that amplifies the conditioning signal by extrapolating between conditioned and unconditioned noise predictions.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.