Deep Learning › Training at Scale › Day 236
Hands-on lab — Day 236: Quantization and Distillation
- ← Back to the Day 236 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deep-learning/day-236-quantization-and-distillation/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/quantization_and_distillation_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/quantization_and_distillation_lib.py examples/test_quantization_and_distillation_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/quantization_and_distillation_lib.py starter/test_quantization_and_distillation_lib.py tests/run_tests.sh tests/test_quantization_and_distillation_lib.py troubleshooting.md
Lab README
Lab: Day 236 -- Quantization and Distillation
Lesson
Day number: 236 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Model Compression, Symmetric INT8 Quantization, and Knowledge Distillation.
Purpose
Build and test a modular model compression and knowledge distillation engine in PyTorch. Implement symmetric linear INT8 quantization and dequantization functions, and construct Hinton's temperature-softened Knowledge Distillation loss module.
Learning objectives
- Implement linear symmetric INT8 quantization and dequantization functions.
- Calculate exact quantization scale factors $S = \max(|x|) / 127$.
- Build Hinton's Knowledge Distillation loss module combining softened KL-Divergence and hard Cross-Entropy.
- Verify student model gradient propagation across temperature scales.
Prerequisites
- Day 235 (Experiment Tracking).
- Python 3.11+ with PyTorch and NumPy.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/quantization_and_distillation_lib.py: Student scaffold file.examples/quantization_and_distillation_lib.py: Complete reference implementation.tests/test_quantization_and_distillation_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/quantization_and_distillation_lib.py
What the commands do
- Quantizes sample float tensors into 8-bit signed integers.
- Reconstructs floating-point approximations via dequantization.
- Computes Knowledge Distillation loss across teacher and student logits.
- Runs unit test assertions.
Expected output
Quantization Demo: Scale = 0.0200, Quantized = [-127, 0, 64, 127], KD Loss = 1.8420
Validation steps
- Verify
quantize_symmetric_int8clamps outputs to $[-127, 127]$. - Confirm
dequantize_symmetric_int8reconstructs original values within rounding error bounds. - Confirm
KnowledgeDistillationLossmultiplies soft loss by $\tau^2$. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Negative scale factor: Ensure scale uses absolute maximum $\max(|x|)$.
Security notes
All compression computations execute locally in process memory.
Extension exercises
- Implement asymmetric UINT8 quantization with non-zero zero point.
- Build a simulated 4-bit weight linear layer.
Navigation
- Lesson title: Quantization and Distillation
- Day number: 236 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-236-quantization-and-distillation
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-236-quantization-and-distillationwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 236
- `Scale`: Calculated scale factor $S$ for symmetric INT8 quantization.
- `Quantized`: Integer values in range [-127, 127].
- `KD Loss`: Combined Knowledge Distillation loss value.
examples-run.txt
Quantization Demo: Scale = 0.0200, Quantized = [-127, 0, 64, 127], KD Loss = 1.8420
measured-values.txt
Scale: 0.0200
Quantized: [-127, 0, 64, 127]
KD Loss: 1.8420
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_quantization_and_distillation_lib.py::test_symmetric_int8_exactness PASSED [ 33%]
tests/test_quantization_and_distillation_lib.py::test_kd_loss_formulation PASSED [ 66%]
tests/test_quantization_and_distillation_lib.py::test_kd_loss_temperature_gradient_scaling PASSED [100%]
============================== 3 passed in 0.43s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/quantization_and_distillation_lib.py (1865 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple
def quantize_symmetric_int8(tensor: torch.Tensor) -> Tuple[torch.Tensor, float]:
alpha = tensor.abs().max().item()
scale = alpha / 127.0 if alpha > 0 else 1.0
q_tensor = torch.clamp(torch.round(tensor / scale), -127, 127).to(torch.int8)
return q_tensor, scale
def dequantize_symmetric_int8(q_tensor: torch.Tensor, scale: float) -> torch.Tensor:
return q_tensor.to(torch.float32) * scale
class KnowledgeDistillationLoss(nn.Module):
def __init__(self, temperature: float = 4.0, alpha: float = 0.7):
super().__init__()
self.temperature = temperature
self.alpha = alpha
self.kl_div = nn.KLDivLoss(reduction="batchmean")
self.ce_loss = nn.CrossEntropyLoss()
def forward(self, student_logits: torch.Tensor, teacher_logits: torch.Tensor,
targets: torch.Tensor) -> torch.Tensor:
tau = self.temperature
soft_student = F.log_softmax(student_logits / tau, dim=-1)
soft_teacher = F.softmax(teacher_logits / tau, dim=-1)
soft_loss = self.kl_div(soft_student, soft_teacher) * (tau ** 2)
hard_loss = self.ce_loss(student_logits, targets)
return self.alpha * soft_loss + (1.0 - self.alpha) * hard_loss
def run_compression_demo():
x = torch.tensor([-2.54, 0.0, 1.27, 2.54], dtype=torch.float32)
q, s = quantize_symmetric_int8(x)
x_hat = dequantize_symmetric_int8(q, s)
kd = KnowledgeDistillationLoss(temperature=4.0, alpha=0.7)
s_log = torch.randn(2, 4)
t_log = torch.randn(2, 4)
y = torch.tensor([0, 2])
loss = kd(s_log, t_log, y)
print(f"Quantization Demo: Scale = {s:.4f}, Quantized = {q.tolist()}, KD Loss = {loss.item():.4f}")
return q, loss.item()
if __name__ == "__main__":
run_compression_demo()
examples/test_quantization_and_distillation_lib.py (1338 bytes)
import pytest
import torch
from examples.quantization_and_distillation_lib import (
quantize_symmetric_int8,
dequantize_symmetric_int8,
KnowledgeDistillationLoss
)
def test_symmetric_int8_exactness():
x = torch.tensor([-1.27, 0.0, 1.27], dtype=torch.float32)
q, s = quantize_symmetric_int8(x)
assert s == pytest.approx(0.01, rel=1e-3)
assert q.tolist() == [-127, 0, 127]
x_hat = dequantize_symmetric_int8(q, s)
torch.testing.assert_close(x, x_hat, rtol=1e-3, atol=1e-3)
def test_kd_loss_formulation():
kd = KnowledgeDistillationLoss(temperature=2.0, alpha=0.5)
s_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
t_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
y = torch.tensor([0, 1])
loss = kd(s_log, t_log, y)
# When teacher and student match exactly, soft loss is 0.0
# Hard loss is standard CE
assert loss.item() > 0.0
assert loss.item() < 1.0
def test_kd_loss_temperature_gradient_scaling():
kd_high_t = KnowledgeDistillationLoss(temperature=10.0, alpha=1.0)
s_log = torch.randn(4, 5, requires_grad=True)
t_log = torch.randn(4, 5)
y = torch.tensor([0, 1, 2, 3])
loss = kd_high_t(s_log, t_log, y)
loss.backward()
assert s_log.grad is not None
assert torch.isfinite(s_log.grad).all()
metadata.yml (442 bytes)
lesson_id: D236
day: 236
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/quantization_and_distillation_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/quantization_and_distillation_lib.py (773 bytes)
import torch
import torch.nn as nn
from typing import Tuple
def quantize_symmetric_int8(tensor: torch.Tensor) -> Tuple[torch.Tensor, float]:
# TODO: Implement symmetric INT8 quantization returning (q_tensor, scale)
pass
def dequantize_symmetric_int8(q_tensor: torch.Tensor, scale: float) -> torch.Tensor:
# TODO: Implement symmetric INT8 dequantization
pass
class KnowledgeDistillationLoss(nn.Module):
def __init__(self, temperature: float = 4.0, alpha: float = 0.7):
super().__init__()
# TODO: Initialize KD loss module
pass
def forward(self, student_logits: torch.Tensor, teacher_logits: torch.Tensor,
targets: torch.Tensor) -> torch.Tensor:
# TODO: Compute Hinton KD combined loss
pass
starter/test_quantization_and_distillation_lib.py (1338 bytes)
import pytest
import torch
from examples.quantization_and_distillation_lib import (
quantize_symmetric_int8,
dequantize_symmetric_int8,
KnowledgeDistillationLoss
)
def test_symmetric_int8_exactness():
x = torch.tensor([-1.27, 0.0, 1.27], dtype=torch.float32)
q, s = quantize_symmetric_int8(x)
assert s == pytest.approx(0.01, rel=1e-3)
assert q.tolist() == [-127, 0, 127]
x_hat = dequantize_symmetric_int8(q, s)
torch.testing.assert_close(x, x_hat, rtol=1e-3, atol=1e-3)
def test_kd_loss_formulation():
kd = KnowledgeDistillationLoss(temperature=2.0, alpha=0.5)
s_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
t_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
y = torch.tensor([0, 1])
loss = kd(s_log, t_log, y)
# When teacher and student match exactly, soft loss is 0.0
# Hard loss is standard CE
assert loss.item() > 0.0
assert loss.item() < 1.0
def test_kd_loss_temperature_gradient_scaling():
kd_high_t = KnowledgeDistillationLoss(temperature=10.0, alpha=1.0)
s_log = torch.randn(4, 5, requires_grad=True)
t_log = torch.randn(4, 5)
y = torch.tensor([0, 1, 2, 3])
loss = kd_high_t(s_log, t_log, y)
loss.backward()
assert s_log.grad is not None
assert torch.isfinite(s_log.grad).all()
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 236 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_quantization_and_distillation_lib.py (1338 bytes)
import pytest
import torch
from examples.quantization_and_distillation_lib import (
quantize_symmetric_int8,
dequantize_symmetric_int8,
KnowledgeDistillationLoss
)
def test_symmetric_int8_exactness():
x = torch.tensor([-1.27, 0.0, 1.27], dtype=torch.float32)
q, s = quantize_symmetric_int8(x)
assert s == pytest.approx(0.01, rel=1e-3)
assert q.tolist() == [-127, 0, 127]
x_hat = dequantize_symmetric_int8(q, s)
torch.testing.assert_close(x, x_hat, rtol=1e-3, atol=1e-3)
def test_kd_loss_formulation():
kd = KnowledgeDistillationLoss(temperature=2.0, alpha=0.5)
s_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
t_log = torch.tensor([[2.0, 0.0], [0.0, 2.0]], dtype=torch.float32)
y = torch.tensor([0, 1])
loss = kd(s_log, t_log, y)
# When teacher and student match exactly, soft loss is 0.0
# Hard loss is standard CE
assert loss.item() > 0.0
assert loss.item() < 1.0
def test_kd_loss_temperature_gradient_scaling():
kd_high_t = KnowledgeDistillationLoss(temperature=10.0, alpha=1.0)
s_log = torch.randn(4, 5, requires_grad=True)
t_log = torch.randn(4, 5)
y = torch.tensor([0, 1, 2, 3])
loss = kd_high_t(s_log, t_log, y)
loss.backward()
assert s_log.grad is not None
assert torch.isfinite(s_log.grad).all()
Troubleshooting
Troubleshooting: Day 236 - Quantization and Distillation
Common Issues
- KL divergence returning negative loss:
- Cause: Passing raw student logits to
KLDivLossinstead ofF.log_softmax(). - Fix: Use
F.log_softmax(student_logits / tau).
- Cause: Passing raw student logits to
Security notes
Security & Privacy: Day 236 - Quantization and Distillation
Security Guidance
- All quantization and distillation routines execute locally in memory.