Deep Learning › Training at Scale › Day 236

Hands-on lab — Day 236: Quantization and Distillation

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/quantization_and_distillation_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/quantization_and_distillation_lib.py
examples/test_quantization_and_distillation_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/quantization_and_distillation_lib.py
starter/test_quantization_and_distillation_lib.py
tests/run_tests.sh
tests/test_quantization_and_distillation_lib.py
troubleshooting.md

Lab README

Lab: Day 236 -- Quantization and Distillation

Lesson

Day number: 236 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Model Compression, Symmetric INT8 Quantization, and Knowledge Distillation.

Purpose

Build and test a modular model compression and knowledge distillation engine in PyTorch. Implement symmetric linear INT8 quantization and dequantization functions, and construct Hinton's temperature-softened Knowledge Distillation loss module.

Learning objectives

  • Implement linear symmetric INT8 quantization and dequantization functions.
  • Calculate exact quantization scale factors $S = \max(|x|) / 127$.
  • Build Hinton's Knowledge Distillation loss module combining softened KL-Divergence and hard Cross-Entropy.
  • Verify student model gradient propagation across temperature scales.

Prerequisites

  • Day 235 (Experiment Tracking).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/quantization_and_distillation_lib.py: Student scaffold file.
  • examples/quantization_and_distillation_lib.py: Complete reference implementation.
  • tests/test_quantization_and_distillation_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/quantization_and_distillation_lib.py

What the commands do

  • Quantizes sample float tensors into 8-bit signed integers.
  • Reconstructs floating-point approximations via dequantization.
  • Computes Knowledge Distillation loss across teacher and student logits.
  • Runs unit test assertions.

Expected output

Quantization Demo: Scale = 0.0200, Quantized = [-127, 0, 64, 127], KD Loss = 1.8420

Validation steps

  1. Verify quantize_symmetric_int8 clamps outputs to $[-127, 127]$.
  2. Confirm dequantize_symmetric_int8 reconstructs original values within rounding error bounds.
  3. Confirm KnowledgeDistillationLoss multiplies soft loss by $\tau^2$.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Negative scale factor: Ensure scale uses absolute maximum $\max(|x|)$.

Security notes

All compression computations execute locally in process memory.

Extension exercises

  1. Implement asymmetric UINT8 quantization with non-zero zero point.
  2. Build a simulated 4-bit weight linear layer.

Expected output

FIELDS.md

# Expected Output Fields: Day 236

- `Scale`: Calculated scale factor $S$ for symmetric INT8 quantization.
- `Quantized`: Integer values in range [-127, 127].
- `KD Loss`: Combined Knowledge Distillation loss value.

examples-run.txt

Quantization Demo: Scale = 0.0200, Quantized = [-127, 0, 64, 127], KD Loss = 1.8420

measured-values.txt

Scale: 0.0200
Quantized: [-127, 0, 64, 127]
KD Loss: 1.8420

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_quantization_and_distillation_lib.py::test_symmetric_int8_exactness PASSED [ 33%]
tests/test_quantization_and_distillation_lib.py::test_kd_loss_formulation PASSED [ 66%]
tests/test_quantization_and_distillation_lib.py::test_kd_loss_temperature_gradient_scaling PASSED [100%]

============================== 3 passed in 0.43s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/quantization_and_distillation_lib.py (1865 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

def quantize_symmetric_int8(tensor: torch.Tensor) -> Tuple[torch.Tensor, float]:
    alpha = tensor.abs().max().item()
    scale = alpha / 127.0 if alpha > 0 else 1.0
    q_tensor = torch.clamp(torch.round(tensor / scale), -127, 127).to(torch.int8)
    return q_tensor, scale

def dequantize_symmetric_int8(q_tensor: torch.Tensor, scale: float) -> torch.Tensor:
    return q_tensor.to(torch.float32) * scale

class KnowledgeDistillationLoss(nn.Module):
    def __init__(self, temperature: float = 4.0, alpha: float = 0.7):
        super().__init__()
        self.temperature = temperature
        self.alpha = alpha
        self.kl_div = nn.KLDivLoss(reduction="batchmean")
        self.ce_loss = nn.CrossEntropyLoss()

    def forward(self, student_logits: torch.Tensor, teacher_logits: torch.Tensor,
                targets: torch.Tensor) -> torch.Tensor:
        tau = self.temperature
        soft_student = F.log_softmax(student_logits / tau, dim=-1)
        soft_teacher = F.softmax(teacher_logits / tau, dim=-1)

        soft_loss = self.kl_div(soft_student, soft_teacher) * (tau ** 2)
        hard_loss = self.ce_loss(student_logits, targets)

        return self.alpha * soft_loss + (1.0 - self.alpha) * hard_loss

def run_compression_demo():
    x = torch.tensor([-2.54, 0.0, 1.27, 2.54], dtype=torch.float32)
    q, s = quantize_symmetric_int8(x)
    x_hat = dequantize_symmetric_int8(q, s)

    kd = KnowledgeDistillationLoss(temperature=4.0, alpha=0.7)
    s_log = torch.randn(2, 4)
    t_log = torch.randn(2, 4)
    y = torch.tensor([0, 2])
    loss = kd(s_log, t_log, y)

    print(f"Quantization Demo: Scale = {s:.4f}, Quantized = {q.tolist()}, KD Loss = {loss.item():.4f}")
    return q, loss.item()

if __name__ == "__main__":
    run_compression_demo()
examples/test_quantization_and_distillation_lib.py (1338 bytes)
import pytest
import torch
from examples.quantization_and_distillation_lib import (
    quantize_symmetric_int8,
    dequantize_symmetric_int8,
    KnowledgeDistillationLoss
)

def test_symmetric_int8_exactness():
    x = torch.tensor([-1.27, 0.0, 1.27], dtype=torch.float32)
    q, s = quantize_symmetric_int8(x)
    assert s == pytest.approx(0.01, rel=1e-3)
    assert q.tolist() == [-127, 0, 127]
    
    x_hat = dequantize_symmetric_int8(q, s)
    torch.testing.assert_close(x, x_hat, rtol=1e-3, atol=1e-3)

def test_kd_loss_formulation():
    kd = KnowledgeDistillationLoss(temperature=2.0, alpha=0.5)
    s_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
    t_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
    y = torch.tensor([0, 1])

    loss = kd(s_log, t_log, y)
    # When teacher and student match exactly, soft loss is 0.0
    # Hard loss is standard CE
    assert loss.item() > 0.0
    assert loss.item() < 1.0

def test_kd_loss_temperature_gradient_scaling():
    kd_high_t = KnowledgeDistillationLoss(temperature=10.0, alpha=1.0)
    s_log = torch.randn(4, 5, requires_grad=True)
    t_log = torch.randn(4, 5)
    y = torch.tensor([0, 1, 2, 3])

    loss = kd_high_t(s_log, t_log, y)
    loss.backward()
    assert s_log.grad is not None
    assert torch.isfinite(s_log.grad).all()
metadata.yml (442 bytes)
lesson_id: D236
day: 236
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/quantization_and_distillation_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/quantization_and_distillation_lib.py (773 bytes)
import torch
import torch.nn as nn
from typing import Tuple

def quantize_symmetric_int8(tensor: torch.Tensor) -> Tuple[torch.Tensor, float]:
    # TODO: Implement symmetric INT8 quantization returning (q_tensor, scale)
    pass

def dequantize_symmetric_int8(q_tensor: torch.Tensor, scale: float) -> torch.Tensor:
    # TODO: Implement symmetric INT8 dequantization
    pass

class KnowledgeDistillationLoss(nn.Module):
    def __init__(self, temperature: float = 4.0, alpha: float = 0.7):
        super().__init__()
        # TODO: Initialize KD loss module
        pass

    def forward(self, student_logits: torch.Tensor, teacher_logits: torch.Tensor,
                targets: torch.Tensor) -> torch.Tensor:
        # TODO: Compute Hinton KD combined loss
        pass
starter/test_quantization_and_distillation_lib.py (1338 bytes)
import pytest
import torch
from examples.quantization_and_distillation_lib import (
    quantize_symmetric_int8,
    dequantize_symmetric_int8,
    KnowledgeDistillationLoss
)

def test_symmetric_int8_exactness():
    x = torch.tensor([-1.27, 0.0, 1.27], dtype=torch.float32)
    q, s = quantize_symmetric_int8(x)
    assert s == pytest.approx(0.01, rel=1e-3)
    assert q.tolist() == [-127, 0, 127]
    
    x_hat = dequantize_symmetric_int8(q, s)
    torch.testing.assert_close(x, x_hat, rtol=1e-3, atol=1e-3)

def test_kd_loss_formulation():
    kd = KnowledgeDistillationLoss(temperature=2.0, alpha=0.5)
    s_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
    t_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
    y = torch.tensor([0, 1])

    loss = kd(s_log, t_log, y)
    # When teacher and student match exactly, soft loss is 0.0
    # Hard loss is standard CE
    assert loss.item() > 0.0
    assert loss.item() < 1.0

def test_kd_loss_temperature_gradient_scaling():
    kd_high_t = KnowledgeDistillationLoss(temperature=10.0, alpha=1.0)
    s_log = torch.randn(4, 5, requires_grad=True)
    t_log = torch.randn(4, 5)
    y = torch.tensor([0, 1, 2, 3])

    loss = kd_high_t(s_log, t_log, y)
    loss.backward()
    assert s_log.grad is not None
    assert torch.isfinite(s_log.grad).all()
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 236 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_quantization_and_distillation_lib.py (1338 bytes)
import pytest
import torch
from examples.quantization_and_distillation_lib import (
    quantize_symmetric_int8,
    dequantize_symmetric_int8,
    KnowledgeDistillationLoss
)

def test_symmetric_int8_exactness():
    x = torch.tensor([-1.27, 0.0, 1.27], dtype=torch.float32)
    q, s = quantize_symmetric_int8(x)
    assert s == pytest.approx(0.01, rel=1e-3)
    assert q.tolist() == [-127, 0, 127]
    
    x_hat = dequantize_symmetric_int8(q, s)
    torch.testing.assert_close(x, x_hat, rtol=1e-3, atol=1e-3)

def test_kd_loss_formulation():
    kd = KnowledgeDistillationLoss(temperature=2.0, alpha=0.5)
    s_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
    t_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
    y = torch.tensor([0, 1])

    loss = kd(s_log, t_log, y)
    # When teacher and student match exactly, soft loss is 0.0
    # Hard loss is standard CE
    assert loss.item() > 0.0
    assert loss.item() < 1.0

def test_kd_loss_temperature_gradient_scaling():
    kd_high_t = KnowledgeDistillationLoss(temperature=10.0, alpha=1.0)
    s_log = torch.randn(4, 5, requires_grad=True)
    t_log = torch.randn(4, 5)
    y = torch.tensor([0, 1, 2, 3])

    loss = kd_high_t(s_log, t_log, y)
    loss.backward()
    assert s_log.grad is not None
    assert torch.isfinite(s_log.grad).all()

Troubleshooting

Troubleshooting: Day 236 - Quantization and Distillation

Common Issues

  1. KL divergence returning negative loss:
    • Cause: Passing raw student logits to KLDivLoss instead of F.log_softmax().
    • Fix: Use F.log_softmax(student_logits / tau).

Security notes

Security & Privacy: Day 236 - Quantization and Distillation

Security Guidance

  • All quantization and distillation routines execute locally in memory.