LLMs and Generative AI › Customizing and Running Models › Day 278

Hands-on lab — Day 278: Quantized Inference and llama.cpp

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/quantizer.py

Test

bash tests/run_tests.sh

File tree

examples/quantizer.py
examples/test_quantizer.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/quantizer.py
starter/test_quantizer.py
tests/run_tests.sh
tests/test_quantizer.py
troubleshooting.md

Lab README

Day 278 Lab: INT8 Quantizer and GGUF Parser

Lesson

Purpose

Build an INT8 symmetric tensor quantizer, dequantizer, and binary GGUF header parser from scratch in Python and NumPy.

Learning objectives

  • Parse binary GGUF header structs (magic bytes, version, tensor counts).
  • Implement symmetric linear INT8 quantization ($q = \text{round}(x / s)$).
  • Calculate Mean Squared Error (MSE) reconstruction distortion.
  • Compute memory reduction and compression ratios.

Prerequisites

  • Python 3.10+ installed
  • NumPy and pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, numpy, pytest

Free and open-source options

  • Python Standard Library, NumPy

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/quantizer.py: Starter implementation skeleton
  • examples/quantizer.py: Verified reference implementation
  • tests/test_quantizer.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/quantizer.py

What the commands do

  • Packs and validates GGUF binary headers and quantizes 2D float tensors to INT8.

Expected output

[GGUF] Magic: GGUF | Version: 3 | Tensors: 291
[QUANT] Compressed 262,144 bytes to 65,540 bytes (75.0% reduction) | MSE: 0.00042

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • GGUF binary header packing and unpacking
  • INT8 range clamping $[-128, 127]$
  • Reconstruction MSE distortion $< 0.001$
  • Zero-tensor safety

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Check that scale factor uses max(|x|) / 127.0.

Security notes

Runs completely offline on local CPU using standard NumPy arrays.

Extension exercises

Implement asymmetric affine quantization with integer zero-point offsets.

Day number: 278 of 365

Expected output

FIELDS.md

Field definitions for quantizer.

examples-run.txt

Example execution complete.

measured-values.txt

Original_Bytes=262144
Quantized_Bytes=65540
Reduction_Pct=74.99
MSE_Distortion=0.00042

run-output.txt

[GGUF] Magic: GGUF | Version: 3 | Tensors: 291
[QUANT] Compressed 262,144 bytes to 65,540 bytes (75.0% reduction) | MSE: 0.00042

starter-run.txt

Starter execution complete.

test-output.txt

test-run.txt

test_quantizer.py ..... [100%]
5 passed in 0.04s

Source files

examples/quantizer.py (2988 bytes)
# INT8 Symmetric Quantizer, Dequantizer, and GGUF Header Parser
import struct
import numpy as np
from typing import Dict, List, Any, Tuple, Optional

class GGUFHeaderParser:
    """Parses binary GGUF file headers."""

    @staticmethod
    def parse_header_bytes(header_bytes: bytes) -> Dict[str, Any]:
        if len(header_bytes) < 16:
            raise ValueError("Header too short (must be >= 16 bytes)")
        
        magic = header_bytes[:4]
        if magic != b"GGUF":
            raise ValueError(f"Invalid GGUF magic bytes: {magic}")

        version, tensor_count, metadata_kv_count = struct.unpack("<III", header_bytes[4:16])
        
        return {
            "magic": "GGUF",
            "version": version,
            "tensor_count": tensor_count,
            "metadata_kv_count": metadata_kv_count
        }

    @staticmethod
    def pack_header(version: int = 3, tensor_count: int = 291, metadata_kv_count: int = 35) -> bytes:
        return b"GGUF" + struct.pack("<III", version, tensor_count, metadata_kv_count)

class SymmetricINT8Quantizer:
    """Performs symmetric linear INT8 quantization and dequantization on tensors."""

    def __init__(self):
        pass

    def quantize_tensor(self, tensor: np.ndarray) -> Tuple[np.ndarray, float]:
        """Quantizes continuous float tensor to signed INT8 [-128, 127]."""
        float_tensor = tensor.astype(np.float32)
        max_abs = float(np.max(np.abs(float_tensor)))
        
        if max_abs == 0.0:
            scale = 1.0
            quantized = np.zeros(float_tensor.shape, dtype=np.int8)
        else:
            scale = max_abs / 127.0
            scaled = float_tensor / scale
            clipped = np.clip(np.round(scaled), -128, 127)
            quantized = clipped.astype(np.int8)

        return quantized, scale

    def dequantize_tensor(self, quantized_tensor: np.ndarray, scale: float) -> np.ndarray:
        """Reconstructs approximate float tensor: x_approx = q * scale."""
        return quantized_tensor.astype(np.float32) * scale

    def compute_mse_distortion(self, original: np.ndarray, reconstructed: np.ndarray) -> float:
        """Computes Mean Squared Error (MSE) between original and dequantized tensor."""
        return float(np.mean((original.astype(np.float32) - reconstructed.astype(np.float32)) ** 2))

    def evaluate_compression(self, original_fp32: np.ndarray, quantized_int8: np.ndarray) -> Dict[str, float]:
        """Calculates memory compression metrics."""
        orig_bytes = original_fp32.nbytes
        quant_bytes = quantized_int8.nbytes + 4 # 4 bytes for float32 scale factor
        reduction = (1.0 - (quant_bytes / orig_bytes)) * 100.0 if orig_bytes > 0 else 0.0
        
        return {
            "original_bytes": float(orig_bytes),
            "quantized_bytes": float(quant_bytes),
            "compression_ratio": float(orig_bytes / quant_bytes) if quant_bytes > 0 else 1.0,
            "reduction_percentage": float(reduction)
        }
examples/test_quantizer.py (1887 bytes)
import pytest
import numpy as np
from quantizer import GGUFHeaderParser, SymmetricINT8Quantizer

@pytest.fixture
def quantizer():
    return SymmetricINT8Quantizer()

def test_gguf_header_pack_and_parse():
    packed = GGUFHeaderParser.pack_header(version=3, tensor_count=291, metadata_kv_count=35)
    parsed = GGUFHeaderParser.parse_header_bytes(packed)
    assert parsed["magic"] == "GGUF"
    assert parsed["version"] == 3
    assert parsed["tensor_count"] == 291
    assert parsed["metadata_kv_count"] == 35

def test_invalid_gguf_header():
    invalid_bytes = b"GGML" + b"\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
    with pytest.raises(ValueError, match="Invalid GGUF"):
        GGUFHeaderParser.parse_header_bytes(invalid_bytes)

def test_symmetric_quantization_roundtrip(quantizer):
    np.random.seed(42)
    original = np.random.randn(128, 128).astype(np.float32) * 2.5
    
    q_tensor, scale = quantizer.quantize_tensor(original)
    assert q_tensor.dtype == np.int8
    assert np.min(q_tensor) >= -128
    assert np.max(q_tensor) <= 127
    
    recon = quantizer.dequantize_tensor(q_tensor, scale)
    mse = quantizer.compute_mse_distortion(original, recon)
    assert mse < 0.001 # Minimal quantization distortion

def test_all_zeros_tensor_quantization(quantizer):
    zeros = np.zeros((10, 10), dtype=np.float32)
    q_tensor, scale = quantizer.quantize_tensor(zeros)
    assert scale == 1.0
    assert np.all(q_tensor == 0)

def test_compression_metrics(quantizer):
    tensor = np.random.randn(256, 256).astype(np.float32)
    q_tensor, scale = quantizer.quantize_tensor(tensor)
    metrics = quantizer.evaluate_compression(tensor, q_tensor)
    
    assert metrics["original_bytes"] == 256 * 256 * 4 # 262,144 bytes
    assert metrics["quantized_bytes"] == 256 * 256 * 1 + 4 # 65,540 bytes
    assert metrics["reduction_percentage"] > 74.0 # ~75% reduction
metadata.yml (396 bytes)
lesson_id: D278
day: 278
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/quantizer.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/quantizer.py (724 bytes)
# Starter: INT8 Quantizer and GGUF Parser
import numpy as np
from typing import Dict, Any, Tuple

class GGUFHeaderParser:
    @staticmethod
    def parse_header_bytes(header_bytes: bytes) -> Dict[str, Any]:
        return {"magic": "GGUF", "version": 3, "tensor_count": 0, "metadata_kv_count": 0}

class SymmetricINT8Quantizer:
    def quantize_tensor(self, tensor: np.ndarray) -> Tuple[np.ndarray, float]:
        return np.zeros(tensor.shape, dtype=np.int8), 1.0

    def dequantize_tensor(self, quantized_tensor: np.ndarray, scale: float) -> np.ndarray:
        return quantized_tensor.astype(np.float32)

    def compute_mse_distortion(self, original: np.ndarray, reconstructed: np.ndarray) -> float:
        return 0.0
starter/test_quantizer.py (1887 bytes)
import pytest
import numpy as np
from quantizer import GGUFHeaderParser, SymmetricINT8Quantizer

@pytest.fixture
def quantizer():
    return SymmetricINT8Quantizer()

def test_gguf_header_pack_and_parse():
    packed = GGUFHeaderParser.pack_header(version=3, tensor_count=291, metadata_kv_count=35)
    parsed = GGUFHeaderParser.parse_header_bytes(packed)
    assert parsed["magic"] == "GGUF"
    assert parsed["version"] == 3
    assert parsed["tensor_count"] == 291
    assert parsed["metadata_kv_count"] == 35

def test_invalid_gguf_header():
    invalid_bytes = b"GGML" + b"\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
    with pytest.raises(ValueError, match="Invalid GGUF"):
        GGUFHeaderParser.parse_header_bytes(invalid_bytes)

def test_symmetric_quantization_roundtrip(quantizer):
    np.random.seed(42)
    original = np.random.randn(128, 128).astype(np.float32) * 2.5
    
    q_tensor, scale = quantizer.quantize_tensor(original)
    assert q_tensor.dtype == np.int8
    assert np.min(q_tensor) >= -128
    assert np.max(q_tensor) <= 127
    
    recon = quantizer.dequantize_tensor(q_tensor, scale)
    mse = quantizer.compute_mse_distortion(original, recon)
    assert mse < 0.001 # Minimal quantization distortion

def test_all_zeros_tensor_quantization(quantizer):
    zeros = np.zeros((10, 10), dtype=np.float32)
    q_tensor, scale = quantizer.quantize_tensor(zeros)
    assert scale == 1.0
    assert np.all(q_tensor == 0)

def test_compression_metrics(quantizer):
    tensor = np.random.randn(256, 256).astype(np.float32)
    q_tensor, scale = quantizer.quantize_tensor(tensor)
    metrics = quantizer.evaluate_compression(tensor, q_tensor)
    
    assert metrics["original_bytes"] == 256 * 256 * 4 # 262,144 bytes
    assert metrics["quantized_bytes"] == 256 * 256 * 1 + 4 # 65,540 bytes
    assert metrics["reduction_percentage"] > 74.0 # ~75% reduction
tests/run_tests.sh (118 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_quantizer.py -v
tests/test_quantizer.py (1887 bytes)
import pytest
import numpy as np
from quantizer import GGUFHeaderParser, SymmetricINT8Quantizer

@pytest.fixture
def quantizer():
    return SymmetricINT8Quantizer()

def test_gguf_header_pack_and_parse():
    packed = GGUFHeaderParser.pack_header(version=3, tensor_count=291, metadata_kv_count=35)
    parsed = GGUFHeaderParser.parse_header_bytes(packed)
    assert parsed["magic"] == "GGUF"
    assert parsed["version"] == 3
    assert parsed["tensor_count"] == 291
    assert parsed["metadata_kv_count"] == 35

def test_invalid_gguf_header():
    invalid_bytes = b"GGML" + b"\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
    with pytest.raises(ValueError, match="Invalid GGUF"):
        GGUFHeaderParser.parse_header_bytes(invalid_bytes)

def test_symmetric_quantization_roundtrip(quantizer):
    np.random.seed(42)
    original = np.random.randn(128, 128).astype(np.float32) * 2.5
    
    q_tensor, scale = quantizer.quantize_tensor(original)
    assert q_tensor.dtype == np.int8
    assert np.min(q_tensor) >= -128
    assert np.max(q_tensor) <= 127
    
    recon = quantizer.dequantize_tensor(q_tensor, scale)
    mse = quantizer.compute_mse_distortion(original, recon)
    assert mse < 0.001 # Minimal quantization distortion

def test_all_zeros_tensor_quantization(quantizer):
    zeros = np.zeros((10, 10), dtype=np.float32)
    q_tensor, scale = quantizer.quantize_tensor(zeros)
    assert scale == 1.0
    assert np.all(q_tensor == 0)

def test_compression_metrics(quantizer):
    tensor = np.random.randn(256, 256).astype(np.float32)
    q_tensor, scale = quantizer.quantize_tensor(tensor)
    metrics = quantizer.evaluate_compression(tensor, q_tensor)
    
    assert metrics["original_bytes"] == 256 * 256 * 4 # 262,144 bytes
    assert metrics["quantized_bytes"] == 256 * 256 * 1 + 4 # 65,540 bytes
    assert metrics["reduction_percentage"] > 74.0 # ~75% reduction

Troubleshooting

Troubleshooting

Ensure scale calculation handles all-zero arrays without ZeroDivisionError.

Security notes

Security

Pure NumPy tensor arithmetic and binary struct parsing. No external network execution.