Deep Learning β€Ί Training at Scale β€Ί Day 236

Day 236: Quantization and Distillation

Day 236 of 365 β€” Quantization and Distillation

Master deep learning model compression and edge deployment: understand Quantization principles (Post-Training Quantization vs Quantization-Aware Training), implement linear integer quantization (scale S and zero-point Z), analyze modern LLM 4-bit algorithms (GPTQ, AWQ, GGUF, bitsandbytes NF4), derive Knowledge Distillation with temperature scaling and KL-divergence, and build a complete student distillation pipeline in PyTorch.

Course
Deep Learning
Category
Training at Scale
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-236-quantization-and-distillation

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-236-quantization-and-distillation
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

The largest frontier AI models contain hundreds of billions of parameters.

Serving a 70-Billion parameter LLM in FP16 requires over 140 GB of VRAMβ€”requiring two $30,000 NVIDIA H100 GPUs just to answer a single user question.

To deploy AI to edge devices (iPhones, MacBooks, self-driving cars, embedded robotics) and slash datacenter inference costs by 80\%, we must compress models without destroying their intelligence:

Today, you will master linear quantization mathematics, modern 4-bit LLM algorithms (GPTQ, AWQ, GGUF), and Hinton’s formulation of Knowledge Distillation.


The idea in plain language

Diagram: the three families of model compression

Think of compressing audio files:

The apprentice becomes a master chef in 1/10\text{th} the time!


Historical background

Diagram: the historical progression of neural network compression techniques

Quantization mapping diagram showing continuous FP32 range mapped to discrete INT8 scale and zero point

  1. 2015 (Hinton, Vinyals, Dean & Distillation): Geoffrey Hinton published Distilling the Knowledge in a Neural Network, proving that soft probability targets contain β€œdark knowledge” richer than hard ground-truth labels.
  2. 2019 (Sanh et al. & DistilBERT): Hugging Face released DistilBERT, reducing BERT-base by 40\% in size while retaining 97\% of its language understanding capabilities and running 60\% faster.
  3. 2022 (Frantar et al. & GPTQ): Introduced second-order inverse Hessian quantization, enabling 4-bit post-training quantization for 175B LLMs in under 4 GPU hours.
  4. 2023 (Lin et al. & AWQ / Dettmers et al. & QLoRA): Proved that protecting the top 1\% of salient activation channels enables near-lossless 4-bit quantization on consumer hardware.

What it is β€” and what it is not

What Model Compression IS:

What it is NOT:


Why it was created and what problems it solves

Model compression solves three prohibitive barriers to AI deployment:

  1. Memory Bandwidth Bottleneck: Autoregressive LLM generation is memory-bound; 4-bit quantization reduces memory traffic by 75\%, increasing token generation speed by up to 3\times.
  2. VRAM Hardware Gatekeeping: Makes 70B models runnable on a single 48 GB RTX 6000 Ada or 32 GB Apple Silicon Mac.
  3. Inference Latency & Power: Reduces edge device battery drain by replacing power-hungry 32-bit floating-point ALUs with integer operations.

How it works

Let us dissect the mathematical mechanics of linear quantization, advanced LLM algorithms, and knowledge distillation loss functions.


1. Linear Integer Quantization Mathematics

Diagram: how floating point weights are mapped to integers and back

Linear quantization maps a continuous floating-point value x \in [\alpha, \beta] to a discrete b-bit integer q \in [q_{\min}, q_{\max}]:

A. Quantization Equation:

q = \text{clamp}\left(\left\lfloor \frac{x}{S} \right\rceil + Z, q_{\min}, q_{\max}\right)

where \lfloor \cdot \rceil denotes rounding to the nearest integer.

B. Dequantization Equation:

\hat{x} = S \times (q - Z)

C. Parameter Definitions:


2. Post-Training Quantization (PTQ) vs. Quantization-Aware Training (QAT)

Diagram: post-training quantisation compared with quantisation-aware training

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   PTQ VS. QAT COMPARISON SUMMARY                       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Feature            β”‚ Post-Training (PTQ)    β”‚ Quant-Aware Training (QATβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Requires Training? β”‚ No (Calibration only)  β”‚ Yes (Fine-tuning loop)   β”‚
β”‚ Compute Time       β”‚ Minutes                β”‚ Hours to Days            β”‚
β”‚ Target Bit-Width   β”‚ INT8 / INT4 (GPTQ/AWQ) β”‚ INT4 / INT2              β”‚
β”‚ Accuracy Retention β”‚ Near-lossless at 8-bit β”‚ Highest accuracy at <4-b β”‚
β”‚ Gradient Mechanics β”‚ None                   β”‚ Straight-Through Est (STEβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Straight-Through Estimator (STE):

Because the rounding function \lfloor x \rceil has zero derivative almost everywhere (\frac{d\lfloor x \rceil}{dx} = 0), standard backpropagation would fail. The Straight-Through Estimator passes the incoming gradient directly through rounding during the backward pass:

\frac{\partial \mathcal{L}}{\partial x} \approx \frac{\partial \mathcal{L}}{\partial q} \quad \text{for } x \in [\alpha, \beta]

3. Modern Sub-8-bit LLM Quantization Algorithms

  1. GPTQ (Frantar et al., 2022):
    • Employs second-order Taylor expansion of the layer-wise reconstruction loss \mathcal{L} = \|W x - \hat{W} x\|_2^2.
    • Uses the Inverse Hessian Matrix (H^{-1} = (X X^\top)^{-1}) to analytically adjust remaining unquantized weights whenever a column of weights is quantized to 4-bit, compensating for quantization error in real time!
  2. AWQ (Activation-aware Weight Quantization / Lin et al., 2023):
    • Discovered that not all weights are equally important: the top 1\% of weights associated with high-magnitude activation channels dictate 90\% of model accuracy.
    • Applies per-channel scaling to protect these salient channels before applying standard INT4/INT3 quantization.
  3. GGUF / llama.cpp (Gerganov et al.):
    • High-performance CPU/GPU unified quantization format supporting k-quants (block-wise 2-bit, 4-bit, 5-bit, and 6-bit quantization with mixed scales per tensor block).

4. Knowledge Distillation Formulation (Hinton et al., 2015)

Knowledge Distillation teacher student architecture showing soft probability logit matching with temperature tau and ground truth cross entropy

How does a small student learn from a massive teacher?

1. Temperature-Softened Probabilities:

Raw logits z_i are scaled by temperature \tau \ge 1:

p_i(\tau) = \frac{\exp(z_i / \tau)}{\sum_j \exp(z_j / \tau)}

2. The Combined Distillation Loss:

\mathcal{L}_{\text{total}} = \alpha \cdot \tau^2 \cdot D_{\text{KL}}\left(p_{\text{teacher}}(\tau) \;\parallel\; q_{\text{student}}(\tau)\right) + (1 - \alpha) \cdot \mathcal{L}_{\text{CE}}\left(y_{\text{true}}, q_{\text{student}}(1)\right)

5. Speculative Decoding (Leviathan et al. & Chen et al., 2023)

In autoregressive generation, large models are severely memory-bandwidth bound when producing tokens one-by-one:


6. SmoothQuant: Taming Activation Outliers (Xiao et al., 2022)

Diagram: how activation outliers degrade quantisation and how scaling mitigates them

In LLMs exceeding 6.7B parameters, activation tensors exhibit extreme systemic outlier channels (magnitudes up to 100\times larger than normal values), destroying standard INT8 activation quantization:


7. QLoRA: 4-bit NormalFloat & Double Quantization (Dettmers et al., 2023)

Diagram: how adapters are trained on top of a four-bit frozen base model

How do you fine-tune a 65B model on a single 48 GB GPU?


8. Structured Sparsity: NVIDIA 2:4 Sparse Tensor Cores

Beyond quantization, Pruning sets redundant weight parameters to zero:


An everyday analogy

Diagram: lossless storage compared with perceptually lossy compression

Think of compression and education:

The student learns deep positional intuition in a fraction of the time!


Examples in practice

Let us implement Symmetric INT8 Quantization and a Knowledge Distillation Loss Module in PyTorch:

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

def quantize_symmetric_int8(tensor: torch.Tensor) -> Tuple[torch.Tensor, float]:
    # Quantizes a floating point tensor to symmetric INT8 [-127, 127].
    alpha = tensor.abs().max().item()
    scale = alpha / 127.0 if alpha > 0 else 1.0
    
    # Quantize: q = clamp(round(x / scale), -127, 127)
    q_tensor = torch.clamp(torch.round(tensor / scale), -127, 127).to(torch.int8)
    return q_tensor, scale

def dequantize_symmetric_int8(q_tensor: torch.Tensor, scale: float) -> torch.Tensor:
    # Dequantizes an INT8 tensor back to float32: x_hat = q * scale
    return q_tensor.to(torch.float32) * scale

class KnowledgeDistillationLoss(nn.Module):
    # Computes Hinton's combined Knowledge Distillation Loss.
    def __init__(self, temperature: float = 4.0, alpha: float = 0.7):
        super().__init__()
        self.temperature = temperature
        self.alpha = alpha
        self.kl_div = nn.KLDivLoss(reduction="batchmean")
        self.ce_loss = nn.CrossEntropyLoss()

    def forward(self, student_logits: torch.Tensor, teacher_logits: torch.Tensor,
                targets: torch.Tensor) -> torch.Tensor:
        tau = self.temperature
        
        # Softened probabilities
        soft_student = F.log_softmax(student_logits / tau, dim=-1)
        soft_teacher = F.softmax(teacher_logits / tau, dim=-1)

        # Soft loss scaled by tau^2
        soft_loss = self.kl_div(soft_student, soft_teacher) * (tau ** 2)
        # Hard loss with normal logits (tau=1)
        hard_loss = self.ce_loss(student_logits, targets)

        return self.alpha * soft_loss + (1.0 - self.alpha) * hard_loss

Implications: security, privacy, performance, scalability, and cost

Diagram: the cost, reach, accuracy and safety implications of model compression

  1. Quantization Drift & Corner-Case Vulnerability:
    • Aggressive 4-bit or 3-bit quantization can create blind spots where adversarial perturbations or rare edge cases produce catastrophic errors not seen in full-precision models.
  2. Side-Channel Timing Attacks on Edge Devices:
    • Quantized integer inference on mobile NPUs can leak timing variations across zero-valued activations. Ensure constant-time kernel implementations in cryptographic applications.

Alternatives: free, open source, and commercial

Compression MethodTarget PrecisionTooling / FrameworkPrimary Benefit
GPTQ4-bit / 3-bit WeightsAutoGPTQ / ExLlamaV2Fast GPU Batch Inference
AWQ4-bit WeightsAutoAWQ / vLLMExceptional Context Recall
GGUF2-bit to 8-bit Mixllama.cpp / OllamaNative CPU & Apple Silicon
Knowledge DistillSmaller ArchitectureHugging Face Transformers2\times - 5\times Speedup on Any Hardware

Diagram: quantisation compared with distillation as compression strategies

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   MODEL COMPRESSION COMPARISON                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Technique          β”‚ Changes Arch?   β”‚ Memory Reduction β”‚ Compute Speedβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ INT8 Quantization  β”‚ No              β”‚ 50% (2x)         β”‚ 1.5x - 2.0x  β”‚
β”‚ INT4 (GPTQ / AWQ)  β”‚ No              β”‚ 75% (4x)         β”‚ 2.0x - 3.5x  β”‚
β”‚ Distillation       β”‚ Yes (Smaller)   β”‚ 40% - 80%        β”‚ 2.0x - 5.0x  β”‚
β”‚ Pruning (Sparse)   β”‚ No (Zero-masks) β”‚ 30% - 50%        β”‚ 1.2x - 1.5x  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when model compression is worthwhile and when it is premature

When to Use Quantization & Distillation:

When NOT to use:


The AI connection

Knowledge check

  1. What is the difference between symmetric and asymmetric linear quantization?
  2. Why does Hinton’s distillation loss multiply the KL-divergence term by \tau^2?
  3. How does AWQ prevent accuracy degradation in 4-bit LLMs?
  4. What is the role of the Straight-Through Estimator (STE) in Quantization-Aware Training?
  5. How does GPTQ use the inverse Hessian matrix during post-training compression?

Hands-on exercise

In this lab, you will build and test a modular model compression and knowledge distillation engine in PyTorch: implement symmetric linear INT8 quantization and dequantization, compute signal-to-quantization-noise ratio (SQNR), construct Hinton’s temperature-softened Knowledge Distillation loss module, and verify student training convergence.

Expected output

[Model Compression and Distillation Verification Suite]
Testing Symmetric INT8 Quantizer:
  Input Tensor: Shape (4, 4), Max Abs: 3.810
  Calculated Scale S: 0.0300
  Quantized INT8: Min=-127, Max=127 [EXACT BOUNDS]
  Reconstruction MSE: 0.00012 (SQNR > 40 dB) [LOSSLESS INT8 VERIFIED]
Testing Knowledge Distillation Loss (Tau=4.0, Alpha=0.7):
  Teacher Logits (355M): Logit Gap = 4.2
  Student Logits (66M): Forward Soft Loss Computed
  Combined Distillation Loss: 1.842 [CONVERGENCE VERIFIED]
Test Suite: 3 passed in 0.43s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement an Asymmetric UINT8 Quantizer that calculates dynamic Scale S and ZeroPoint Z.
  2. Train a small 2-layer student MLP using Knowledge Distillation from a 6-layer teacher model on MNIST/CIFAR.

Extension challenge

Implement Simulated 4-bit Weight Quantization with Straight-Through Estimator:

Quiz

Q1. What is the mathematical definition of the scale factor S in symmetric INT8 quantization across range [-alpha, +alpha]?

  1. S = alpha / 127
  2. S = alpha * 127
  3. S = 255 / alpha
  4. S = alpha^2
Show answer

Answer: A. S = alpha / 127

In symmetric quantization, scale S = max(|x|) / 127 maps the dynamic range centered at zero.

Q2. What is "Dark Knowledge" in Hinton et al. Knowledge Distillation?

  1. Encrypted passwords in model weights
  2. The ratios among the wrong classes
  3. Data that was deleted from the training set
  4. Unlabeled text scraped from the dark web
Show answer

Answer: B. The ratios among the wrong classes

Softened non-target logits reveal semantic class similarities and structural geometry learned by the teacher.

Q3. Why is the KL-divergence distillation loss multiplied by tau^2 (temperature squared) during student training?

  1. Because temperatures must always be positive
  2. Because tau^2 makes the model run 2x faster
  3. Dividing by tau scales gradients by 1/tau^2
  4. Because Hinton preferred square numbers
Show answer

Answer: C. Dividing by tau scales gradients by 1/tau^2

Multiplying by tau^2 compensates for the 1/tau^2 gradient attenuation introduced by temperature softening.

Q4. How does Activation-aware Weight Quantization (AWQ / Lin et al., 2023) protect LLM accuracy at 4-bit precision?

  1. It trains a separate model for every token
  2. It deletes all activation layers
  3. It rounds all numbers to the nearest power of 2
  4. It protects the 1% of salient weight channels
Show answer

Answer: D. It protects the 1% of salient weight channels

AWQ selectively protects the 1% most salient weight channels corresponding to high-magnitude activation outliers.

Q5. What is the difference between Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT)?

  1. PTQ needs no retraining
  2. PTQ is for text; QAT is for images
  3. QAT deletes the optimizer
  4. PTQ requires 1,000 GPUs
Show answer

Answer: A. PTQ needs no retraining

QAT models quantization noise during backpropagation, yielding superior accuracy at ultra-low bit widths.

Glossary

Quantization
Mapping continuous 32-bit or 16-bit floating-point weights and activations to low-bitwidth discrete integer representations (INT8, INT4).
Knowledge Distillation
A compression technique where a compact student neural network is trained to reproduce the softened probability distributions of a large teacher model.
Temperature Scaling (tau)
Dividing raw logits by a scalar temperature tau before Softmax to soften probability distributions and reveal relative class similarities.
Post-Training Quantization (PTQ)
Quantizing model parameters after training without fine-tuning weights.
Quantization-Aware Training (QAT)
Simulating discrete integer rounding errors during training using the Straight-Through Estimator (STE).
GPTQ
A second-order post-training quantization algorithm utilizing inverse Hessian matrix updates to compress LLM weights to 4 bits with near-zero perplexity loss.
AWQ (Activation-aware Weight Quantization)
A 4-bit quantization method that protects salient weight channels associated with high-magnitude activation outliers.
NormalFloat4 (NF4)
An information-theoretically optimal 4-bit quantile data type for zero-mean, unit-variance normally distributed neural network weights.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.