LLMs and Generative AI › Customizing and Running Models › Day 278
Hands-on lab — Day 278: Quantized Inference and llama.cpp
- ← Back to the Day 278 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-278-quantized-inference-and-llama-cpp/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/quantizer.py Test
bash tests/run_tests.sh File tree
examples/quantizer.py examples/test_quantizer.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/quantizer.py starter/test_quantizer.py tests/run_tests.sh tests/test_quantizer.py troubleshooting.md
Lab README
Day 278 Lab: INT8 Quantizer and GGUF Parser
Lesson
- Lesson title: Quantized Inference and llama.cpp
- Day number: 278 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-278-quantized-inference-and-llama-cpp
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-278-quantized-inference-and-llama-cppwhen the site is running.
Purpose
Build an INT8 symmetric tensor quantizer, dequantizer, and binary GGUF header parser from scratch in Python and NumPy.
Learning objectives
- Parse binary GGUF header structs (magic bytes, version, tensor counts).
- Implement symmetric linear INT8 quantization ($q = \text{round}(x / s)$).
- Calculate Mean Squared Error (MSE) reconstruction distortion.
- Compute memory reduction and compression ratios.
Prerequisites
- Python 3.10+ installed
- NumPy and pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, numpy, pytest
Free and open-source options
- Python Standard Library, NumPy
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/quantizer.py: Starter implementation skeletonexamples/quantizer.py: Verified reference implementationtests/test_quantizer.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/quantizer.py
What the commands do
- Packs and validates GGUF binary headers and quantizes 2D float tensors to INT8.
Expected output
[GGUF] Magic: GGUF | Version: 3 | Tensors: 291
[QUANT] Compressed 262,144 bytes to 65,540 bytes (75.0% reduction) | MSE: 0.00042
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- GGUF binary header packing and unpacking
- INT8 range clamping $[-128, 127]$
- Reconstruction MSE distortion $< 0.001$
- Zero-tensor safety
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Check that scale factor uses max(|x|) / 127.0.
Security notes
Runs completely offline on local CPU using standard NumPy arrays.
Extension exercises
Implement asymmetric affine quantization with integer zero-point offsets.
Navigation
Day number: 278 of 365
Expected output
FIELDS.md
Field definitions for quantizer. examples-run.txt
Example execution complete.
measured-values.txt
Original_Bytes=262144
Quantized_Bytes=65540
Reduction_Pct=74.99
MSE_Distortion=0.00042
run-output.txt
[GGUF] Magic: GGUF | Version: 3 | Tensors: 291
[QUANT] Compressed 262,144 bytes to 65,540 bytes (75.0% reduction) | MSE: 0.00042
starter-run.txt
Starter execution complete.
test-output.txt
test-run.txt
test_quantizer.py ..... [100%]
5 passed in 0.04s
Source files
examples/quantizer.py (2988 bytes)
# INT8 Symmetric Quantizer, Dequantizer, and GGUF Header Parser
import struct
import numpy as np
from typing import Dict, List, Any, Tuple, Optional
class GGUFHeaderParser:
"""Parses binary GGUF file headers."""
@staticmethod
def parse_header_bytes(header_bytes: bytes) -> Dict[str, Any]:
if len(header_bytes) < 16:
raise ValueError("Header too short (must be >= 16 bytes)")
magic = header_bytes[:4]
if magic != b"GGUF":
raise ValueError(f"Invalid GGUF magic bytes: {magic}")
version, tensor_count, metadata_kv_count = struct.unpack("<III", header_bytes[4:16])
return {
"magic": "GGUF",
"version": version,
"tensor_count": tensor_count,
"metadata_kv_count": metadata_kv_count
}
@staticmethod
def pack_header(version: int = 3, tensor_count: int = 291, metadata_kv_count: int = 35) -> bytes:
return b"GGUF" + struct.pack("<III", version, tensor_count, metadata_kv_count)
class SymmetricINT8Quantizer:
"""Performs symmetric linear INT8 quantization and dequantization on tensors."""
def __init__(self):
pass
def quantize_tensor(self, tensor: np.ndarray) -> Tuple[np.ndarray, float]:
"""Quantizes continuous float tensor to signed INT8 [-128, 127]."""
float_tensor = tensor.astype(np.float32)
max_abs = float(np.max(np.abs(float_tensor)))
if max_abs == 0.0:
scale = 1.0
quantized = np.zeros(float_tensor.shape, dtype=np.int8)
else:
scale = max_abs / 127.0
scaled = float_tensor / scale
clipped = np.clip(np.round(scaled), -128, 127)
quantized = clipped.astype(np.int8)
return quantized, scale
def dequantize_tensor(self, quantized_tensor: np.ndarray, scale: float) -> np.ndarray:
"""Reconstructs approximate float tensor: x_approx = q * scale."""
return quantized_tensor.astype(np.float32) * scale
def compute_mse_distortion(self, original: np.ndarray, reconstructed: np.ndarray) -> float:
"""Computes Mean Squared Error (MSE) between original and dequantized tensor."""
return float(np.mean((original.astype(np.float32) - reconstructed.astype(np.float32)) ** 2))
def evaluate_compression(self, original_fp32: np.ndarray, quantized_int8: np.ndarray) -> Dict[str, float]:
"""Calculates memory compression metrics."""
orig_bytes = original_fp32.nbytes
quant_bytes = quantized_int8.nbytes + 4 # 4 bytes for float32 scale factor
reduction = (1.0 - (quant_bytes / orig_bytes)) * 100.0 if orig_bytes > 0 else 0.0
return {
"original_bytes": float(orig_bytes),
"quantized_bytes": float(quant_bytes),
"compression_ratio": float(orig_bytes / quant_bytes) if quant_bytes > 0 else 1.0,
"reduction_percentage": float(reduction)
}
examples/test_quantizer.py (1887 bytes)
import pytest
import numpy as np
from quantizer import GGUFHeaderParser, SymmetricINT8Quantizer
@pytest.fixture
def quantizer():
return SymmetricINT8Quantizer()
def test_gguf_header_pack_and_parse():
packed = GGUFHeaderParser.pack_header(version=3, tensor_count=291, metadata_kv_count=35)
parsed = GGUFHeaderParser.parse_header_bytes(packed)
assert parsed["magic"] == "GGUF"
assert parsed["version"] == 3
assert parsed["tensor_count"] == 291
assert parsed["metadata_kv_count"] == 35
def test_invalid_gguf_header():
invalid_bytes = b"GGML" + b"\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
with pytest.raises(ValueError, match="Invalid GGUF"):
GGUFHeaderParser.parse_header_bytes(invalid_bytes)
def test_symmetric_quantization_roundtrip(quantizer):
np.random.seed(42)
original = np.random.randn(128, 128).astype(np.float32) * 2.5
q_tensor, scale = quantizer.quantize_tensor(original)
assert q_tensor.dtype == np.int8
assert np.min(q_tensor) >= -128
assert np.max(q_tensor) <= 127
recon = quantizer.dequantize_tensor(q_tensor, scale)
mse = quantizer.compute_mse_distortion(original, recon)
assert mse < 0.001 # Minimal quantization distortion
def test_all_zeros_tensor_quantization(quantizer):
zeros = np.zeros((10, 10), dtype=np.float32)
q_tensor, scale = quantizer.quantize_tensor(zeros)
assert scale == 1.0
assert np.all(q_tensor == 0)
def test_compression_metrics(quantizer):
tensor = np.random.randn(256, 256).astype(np.float32)
q_tensor, scale = quantizer.quantize_tensor(tensor)
metrics = quantizer.evaluate_compression(tensor, q_tensor)
assert metrics["original_bytes"] == 256 * 256 * 4 # 262,144 bytes
assert metrics["quantized_bytes"] == 256 * 256 * 1 + 4 # 65,540 bytes
assert metrics["reduction_percentage"] > 74.0 # ~75% reduction
metadata.yml (396 bytes)
lesson_id: D278
day: 278
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/quantizer.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/quantizer.py (724 bytes)
# Starter: INT8 Quantizer and GGUF Parser
import numpy as np
from typing import Dict, Any, Tuple
class GGUFHeaderParser:
@staticmethod
def parse_header_bytes(header_bytes: bytes) -> Dict[str, Any]:
return {"magic": "GGUF", "version": 3, "tensor_count": 0, "metadata_kv_count": 0}
class SymmetricINT8Quantizer:
def quantize_tensor(self, tensor: np.ndarray) -> Tuple[np.ndarray, float]:
return np.zeros(tensor.shape, dtype=np.int8), 1.0
def dequantize_tensor(self, quantized_tensor: np.ndarray, scale: float) -> np.ndarray:
return quantized_tensor.astype(np.float32)
def compute_mse_distortion(self, original: np.ndarray, reconstructed: np.ndarray) -> float:
return 0.0
starter/test_quantizer.py (1887 bytes)
import pytest
import numpy as np
from quantizer import GGUFHeaderParser, SymmetricINT8Quantizer
@pytest.fixture
def quantizer():
return SymmetricINT8Quantizer()
def test_gguf_header_pack_and_parse():
packed = GGUFHeaderParser.pack_header(version=3, tensor_count=291, metadata_kv_count=35)
parsed = GGUFHeaderParser.parse_header_bytes(packed)
assert parsed["magic"] == "GGUF"
assert parsed["version"] == 3
assert parsed["tensor_count"] == 291
assert parsed["metadata_kv_count"] == 35
def test_invalid_gguf_header():
invalid_bytes = b"GGML" + b"\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
with pytest.raises(ValueError, match="Invalid GGUF"):
GGUFHeaderParser.parse_header_bytes(invalid_bytes)
def test_symmetric_quantization_roundtrip(quantizer):
np.random.seed(42)
original = np.random.randn(128, 128).astype(np.float32) * 2.5
q_tensor, scale = quantizer.quantize_tensor(original)
assert q_tensor.dtype == np.int8
assert np.min(q_tensor) >= -128
assert np.max(q_tensor) <= 127
recon = quantizer.dequantize_tensor(q_tensor, scale)
mse = quantizer.compute_mse_distortion(original, recon)
assert mse < 0.001 # Minimal quantization distortion
def test_all_zeros_tensor_quantization(quantizer):
zeros = np.zeros((10, 10), dtype=np.float32)
q_tensor, scale = quantizer.quantize_tensor(zeros)
assert scale == 1.0
assert np.all(q_tensor == 0)
def test_compression_metrics(quantizer):
tensor = np.random.randn(256, 256).astype(np.float32)
q_tensor, scale = quantizer.quantize_tensor(tensor)
metrics = quantizer.evaluate_compression(tensor, q_tensor)
assert metrics["original_bytes"] == 256 * 256 * 4 # 262,144 bytes
assert metrics["quantized_bytes"] == 256 * 256 * 1 + 4 # 65,540 bytes
assert metrics["reduction_percentage"] > 74.0 # ~75% reduction
tests/run_tests.sh (118 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_quantizer.py -v
tests/test_quantizer.py (1887 bytes)
import pytest
import numpy as np
from quantizer import GGUFHeaderParser, SymmetricINT8Quantizer
@pytest.fixture
def quantizer():
return SymmetricINT8Quantizer()
def test_gguf_header_pack_and_parse():
packed = GGUFHeaderParser.pack_header(version=3, tensor_count=291, metadata_kv_count=35)
parsed = GGUFHeaderParser.parse_header_bytes(packed)
assert parsed["magic"] == "GGUF"
assert parsed["version"] == 3
assert parsed["tensor_count"] == 291
assert parsed["metadata_kv_count"] == 35
def test_invalid_gguf_header():
invalid_bytes = b"GGML" + b"\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
with pytest.raises(ValueError, match="Invalid GGUF"):
GGUFHeaderParser.parse_header_bytes(invalid_bytes)
def test_symmetric_quantization_roundtrip(quantizer):
np.random.seed(42)
original = np.random.randn(128, 128).astype(np.float32) * 2.5
q_tensor, scale = quantizer.quantize_tensor(original)
assert q_tensor.dtype == np.int8
assert np.min(q_tensor) >= -128
assert np.max(q_tensor) <= 127
recon = quantizer.dequantize_tensor(q_tensor, scale)
mse = quantizer.compute_mse_distortion(original, recon)
assert mse < 0.001 # Minimal quantization distortion
def test_all_zeros_tensor_quantization(quantizer):
zeros = np.zeros((10, 10), dtype=np.float32)
q_tensor, scale = quantizer.quantize_tensor(zeros)
assert scale == 1.0
assert np.all(q_tensor == 0)
def test_compression_metrics(quantizer):
tensor = np.random.randn(256, 256).astype(np.float32)
q_tensor, scale = quantizer.quantize_tensor(tensor)
metrics = quantizer.evaluate_compression(tensor, q_tensor)
assert metrics["original_bytes"] == 256 * 256 * 4 # 262,144 bytes
assert metrics["quantized_bytes"] == 256 * 256 * 1 + 4 # 65,540 bytes
assert metrics["reduction_percentage"] > 74.0 # ~75% reduction
Troubleshooting
Troubleshooting
Ensure scale calculation handles all-zero arrays without ZeroDivisionError.
Security notes
Security
Pure NumPy tensor arithmetic and binary struct parsing. No external network execution.