LLMs and Generative AI › Customizing and Running Models › Day 276

Hands-on lab — Day 276: Building Fine-Tuning Datasets

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/dataset_pipeline.py

Test

bash tests/run_tests.sh

File tree

examples/dataset_pipeline.py
examples/test_dataset_pipeline.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/dataset_pipeline.py
starter/test_dataset_pipeline.py
tests/run_tests.sh
tests/test_dataset_pipeline.py
troubleshooting.md

Lab README

Day 276 Lab: Dataset Processing & Validation Pipeline

Lesson

Purpose

Build a robust fine-tuning dataset curation and preprocessing pipeline in Python supporting Alpaca/ShareGPT conversion to ChatML, n-gram decontamination filtering, token length analytics, and response-only loss masking.

Learning objectives

  • Convert Alpaca and ShareGPT schemas to standardized ChatML format.
  • Execute automated n-gram decontamination against benchmark query sets.
  • Calculate token length statistics (mean, median, P95, max).
  • Construct PyTorch-compatible -100 response loss masks.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/dataset_pipeline.py: Starter implementation
  • examples/dataset_pipeline.py: Reference implementation
  • tests/test_dataset_pipeline.py: Unit test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/dataset_pipeline.py

What the commands do

  • Processes conversation datasets, applies decontamination filters, and formats ChatML sequences with response loss masks.

Expected output

[DATASET] Ingested 100 samples | Decontaminated: 2 matches
[MASKING] System/User masked with -100 | Assistant tokens preserved

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Correct schema translation from Alpaca and ShareGPT to ChatML
  • N-gram overlap detection and sample decontamination
  • Token summary statistics calculation
  • Exact prompt token masking (-100) and response token retention

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Verify regex word extraction re.findall(r'\b\w+\b', text.lower()) correctly captures clean tokens.

Security notes

Runs completely offline on local CPU. Zero network transmission.

Extension exercises

Implement dynamic multi-sample sequence packing up to 2,048 tokens.

Day number: 276 of 365

Expected output

FIELDS.md

Field definitions for dataset pipeline.

examples-run.txt

Example execution complete.

measured-values.txt

Samples_Ingested=100
Decontaminated_Count=2
Mean_Length=84

run-output.txt

[DATASET] Ingested 100 samples | Decontaminated: 2 matches
[MASKING] System/User masked with -100 | Assistant tokens preserved

starter-run.txt

Starter execution complete.

test-output.txt

test-run.txt

test_dataset_pipeline.py ..... [100%]
5 passed in 0.04s

Source files

examples/dataset_pipeline.py (4610 bytes)
# Fine-Tuning Dataset Processing & Validation Pipeline
import re
from typing import Dict, List, Any, Set, Tuple

class DatasetPipeline:
    """End-to-end dataset transformation, decontamination, and loss-masking engine."""

    def __init__(self, pad_token_id: int = 0, ignore_index: int = -100):
        self.pad_token_id = pad_token_id
        self.ignore_index = ignore_index

    def convert_alpaca_to_chatml(self, alpaca_item: Dict[str, str]) -> Dict[str, List[Dict[str, str]]]:
        """Converts Alpaca (instruction, input, output) to ChatML format."""
        user_content = alpaca_item["instruction"]
        if alpaca_item.get("input") and alpaca_item["input"].strip():
            user_content += "\n\n" + alpaca_item["input"].strip()
            
        return {
            "messages": [
                {"role": "system", "content": "You are a helpful AI assistant."},
                {"role": "user", "content": user_content},
                {"role": "assistant", "content": alpaca_item["output"]}
            ]
        }

    def convert_sharegpt_to_chatml(self, sharegpt_item: Dict[str, Any]) -> Dict[str, List[Dict[str, str]]]:
        """Converts ShareGPT conversations list to ChatML format."""
        role_map = {"human": "user", "gpt": "assistant", "system": "system"}
        messages = []
        for msg in sharegpt_item["conversations"]:
            role = role_map.get(msg["from"], "user")
            messages.append({"role": role, "content": msg["value"]})
        return {"messages": messages}

    def decontaminate_samples(
        self,
        train_samples: List[Dict[str, Any]],
        eval_queries: List[str],
        n_gram_size: int = 5
    ) -> Tuple[List[Dict[str, Any]], int]:
        """Filters out training samples that share n-gram overlaps with eval queries."""
        eval_ngrams: Set[str] = set()
        for q in eval_queries:
            words = self._tokenize_words(q)
            for i in range(len(words) - n_gram_size + 1):
                eval_ngrams.add(" ".join(words[i:i + n_gram_size]))

        clean_samples = []
        filtered_count = 0

        for sample in train_samples:
            text = " ".join([m["content"] for m in sample["messages"]])
            words = self._tokenize_words(text)
            sample_ngrams = {" ".join(words[i:i + n_gram_size]) for i in range(len(words) - n_gram_size + 1)}
            
            if sample_ngrams.intersection(eval_ngrams):
                filtered_count += 1
            else:
                clean_samples.append(sample)

        return clean_samples, filtered_count

    def compute_token_statistics(self, samples: List[Dict[str, Any]]) -> Dict[str, float]:
        """Computes summary statistics on sequence word lengths."""
        lengths = [sum(len(self._tokenize_words(m["content"])) for m in s["messages"]) for s in samples]
        if not lengths:
            return {"count": 0, "mean": 0.0, "median": 0.0, "p95": 0.0, "max": 0.0}

        sorted_lens = sorted(lengths)
        n = len(sorted_lens)
        mean_val = sum(sorted_lens) / n
        median_val = sorted_lens[n // 2]
        p95_val = sorted_lens[int(n * 0.95)] if n > 0 else 0
        max_val = sorted_lens[-1]

        return {
            "count": float(n),
            "mean": float(mean_val),
            "median": float(median_val),
            "p95": float(p95_val),
            "max": float(max_val)
        }

    def generate_loss_masked_labels(
        self,
        chatml_sample: Dict[str, List[Dict[str, str]]],
        mock_vocab: Dict[str, int]
    ) -> Dict[str, List[int]]:
        """Simulates ChatML tokenization and generates -100 loss masks on prompt tokens."""
        input_ids: List[int] = []
        labels: List[int] = []

        for msg in chatml_sample["messages"]:
            role = msg["role"]
            content = msg["content"]
            # Format: <|im_start|>role\\ncontent<|im_end|>
            words = [f"<|im_start|>{role}"] + self._tokenize_words(content) + ["<|im_end|>"]
            tokens = [mock_vocab.get(w, 100) for w in words]
            
            input_ids.extend(tokens)
            if role in ["system", "user"]:
                # Mask prompt tokens with ignore_index (-100)
                labels.extend([self.ignore_index] * len(tokens))
            else:
                # Assistant tokens are trained (loss computed)
                labels.extend(tokens)

        return {
            "input_ids": input_ids,
            "labels": labels
        }

    def _tokenize_words(self, text: str) -> List[str]:
        return re.findall(r'\b\w+\b', text.lower())
examples/test_dataset_pipeline.py (3156 bytes)
import pytest
from dataset_pipeline import DatasetPipeline

@pytest.fixture
def pipeline():
    return DatasetPipeline()

def test_alpaca_to_chatml_conversion(pipeline):
    alpaca = {
        "instruction": "Translate to SQL",
        "input": "Find users where age > 30",
        "output": "SELECT * FROM users WHERE age > 30;"
    }
    chatml = pipeline.convert_alpaca_to_chatml(alpaca)
    assert len(chatml["messages"]) == 3
    assert chatml["messages"][0]["role"] == "system"
    assert "Translate to SQL" in chatml["messages"][1]["content"]
    assert "Find users where age > 30" in chatml["messages"][1]["content"]
    assert chatml["messages"][2]["role"] == "assistant"
    assert "SELECT * FROM users" in chatml["messages"][2]["content"]

def test_sharegpt_to_chatml_conversion(pipeline):
    sharegpt = {
        "conversations": [
            {"from": "system", "value": "System init."},
            {"from": "human", "value": "Hello!"},
            {"from": "gpt", "value": "Hi there!"}
        ]
    }
    chatml = pipeline.convert_sharegpt_to_chatml(sharegpt)
    assert len(chatml["messages"]) == 3
    assert chatml["messages"][0]["role"] == "system"
    assert chatml["messages"][1]["role"] == "user"
    assert chatml["messages"][2]["role"] == "assistant"

def test_decontamination_filtering(pipeline):
    train = [
        {"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "Paris"}]},
        {"messages": [{"role": "user", "content": "Write a python sorting algorithm for integers"}, {"role": "assistant", "content": "def sort(): pass"}]}
    ]
    eval_queries = ["What is the capital of France and Germany?"]
    
    clean, filtered = pipeline.decontaminate_samples(train, eval_queries, n_gram_size=5)
    assert filtered == 1
    assert len(clean) == 1
    assert "sorting algorithm" in clean[0]["messages"][0]["content"]

def test_token_statistics(pipeline):
    samples = [
        {"messages": [{"role": "user", "content": "one two three"}, {"role": "assistant", "content": "four five"}]},
        {"messages": [{"role": "user", "content": "six seven"}, {"role": "assistant", "content": "eight nine ten eleven twelve"}]}
    ]
    stats = pipeline.compute_token_statistics(samples)
    assert stats["count"] == 2
    assert stats["mean"] == 6.0
    assert stats["max"] == 7.0

def test_loss_masking_labels(pipeline):
    sample = {
        "messages": [
            {"role": "system", "content": "You are a code assistant."},
            {"role": "user", "content": "Write hello world"},
            {"role": "assistant", "content": "print('hello world')"}
        ]
    }
    vocab = {"<|im_start|>system": 1, "<|im_start|>user": 2, "<|im_start|>assistant": 3, "<|im_end|>": 4}
    res = pipeline.generate_loss_masked_labels(sample, vocab)
    
    input_ids = res["input_ids"]
    labels = res["labels"]
    assert len(input_ids) == len(labels)
    
    # First part must be masked with -100
    assert labels[0] == -100
    # Assistant response tokens must NOT be -100
    assistant_tokens = [l for l in labels if l != -100]
    assert len(assistant_tokens) > 0
metadata.yml (403 bytes)
lesson_id: D276
day: 276
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/dataset_pipeline.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/dataset_pipeline.py (846 bytes)
# Starter: Dataset Processing Pipeline
from typing import Dict, List, Any, Tuple

class DatasetPipeline:
    def __init__(self, pad_token_id: int = 0, ignore_index: int = -100):
        self.pad_token_id = pad_token_id
        self.ignore_index = ignore_index

    def convert_alpaca_to_chatml(self, alpaca_item: Dict[str, str]) -> Dict[str, List[Dict[str, str]]]:
        return {"messages": []}

    def decontaminate_samples(
        self,
        train_samples: List[Dict[str, Any]],
        eval_queries: List[str],
        n_gram_size: int = 5
    ) -> Tuple[List[Dict[str, Any]], int]:
        return train_samples, 0

    def generate_loss_masked_labels(
        self,
        chatml_sample: Dict[str, List[Dict[str, str]]],
        mock_vocab: Dict[str, int]
    ) -> Dict[str, List[int]]:
        return {"input_ids": [], "labels": []}
starter/test_dataset_pipeline.py (3156 bytes)
import pytest
from dataset_pipeline import DatasetPipeline

@pytest.fixture
def pipeline():
    return DatasetPipeline()

def test_alpaca_to_chatml_conversion(pipeline):
    alpaca = {
        "instruction": "Translate to SQL",
        "input": "Find users where age > 30",
        "output": "SELECT * FROM users WHERE age > 30;"
    }
    chatml = pipeline.convert_alpaca_to_chatml(alpaca)
    assert len(chatml["messages"]) == 3
    assert chatml["messages"][0]["role"] == "system"
    assert "Translate to SQL" in chatml["messages"][1]["content"]
    assert "Find users where age > 30" in chatml["messages"][1]["content"]
    assert chatml["messages"][2]["role"] == "assistant"
    assert "SELECT * FROM users" in chatml["messages"][2]["content"]

def test_sharegpt_to_chatml_conversion(pipeline):
    sharegpt = {
        "conversations": [
            {"from": "system", "value": "System init."},
            {"from": "human", "value": "Hello!"},
            {"from": "gpt", "value": "Hi there!"}
        ]
    }
    chatml = pipeline.convert_sharegpt_to_chatml(sharegpt)
    assert len(chatml["messages"]) == 3
    assert chatml["messages"][0]["role"] == "system"
    assert chatml["messages"][1]["role"] == "user"
    assert chatml["messages"][2]["role"] == "assistant"

def test_decontamination_filtering(pipeline):
    train = [
        {"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "Paris"}]},
        {"messages": [{"role": "user", "content": "Write a python sorting algorithm for integers"}, {"role": "assistant", "content": "def sort(): pass"}]}
    ]
    eval_queries = ["What is the capital of France and Germany?"]
    
    clean, filtered = pipeline.decontaminate_samples(train, eval_queries, n_gram_size=5)
    assert filtered == 1
    assert len(clean) == 1
    assert "sorting algorithm" in clean[0]["messages"][0]["content"]

def test_token_statistics(pipeline):
    samples = [
        {"messages": [{"role": "user", "content": "one two three"}, {"role": "assistant", "content": "four five"}]},
        {"messages": [{"role": "user", "content": "six seven"}, {"role": "assistant", "content": "eight nine ten eleven twelve"}]}
    ]
    stats = pipeline.compute_token_statistics(samples)
    assert stats["count"] == 2
    assert stats["mean"] == 6.0
    assert stats["max"] == 7.0

def test_loss_masking_labels(pipeline):
    sample = {
        "messages": [
            {"role": "system", "content": "You are a code assistant."},
            {"role": "user", "content": "Write hello world"},
            {"role": "assistant", "content": "print('hello world')"}
        ]
    }
    vocab = {"<|im_start|>system": 1, "<|im_start|>user": 2, "<|im_start|>assistant": 3, "<|im_end|>": 4}
    res = pipeline.generate_loss_masked_labels(sample, vocab)
    
    input_ids = res["input_ids"]
    labels = res["labels"]
    assert len(input_ids) == len(labels)
    
    # First part must be masked with -100
    assert labels[0] == -100
    # Assistant response tokens must NOT be -100
    assistant_tokens = [l for l in labels if l != -100]
    assert len(assistant_tokens) > 0
tests/run_tests.sh (125 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_dataset_pipeline.py -v
tests/test_dataset_pipeline.py (3156 bytes)
import pytest
from dataset_pipeline import DatasetPipeline

@pytest.fixture
def pipeline():
    return DatasetPipeline()

def test_alpaca_to_chatml_conversion(pipeline):
    alpaca = {
        "instruction": "Translate to SQL",
        "input": "Find users where age > 30",
        "output": "SELECT * FROM users WHERE age > 30;"
    }
    chatml = pipeline.convert_alpaca_to_chatml(alpaca)
    assert len(chatml["messages"]) == 3
    assert chatml["messages"][0]["role"] == "system"
    assert "Translate to SQL" in chatml["messages"][1]["content"]
    assert "Find users where age > 30" in chatml["messages"][1]["content"]
    assert chatml["messages"][2]["role"] == "assistant"
    assert "SELECT * FROM users" in chatml["messages"][2]["content"]

def test_sharegpt_to_chatml_conversion(pipeline):
    sharegpt = {
        "conversations": [
            {"from": "system", "value": "System init."},
            {"from": "human", "value": "Hello!"},
            {"from": "gpt", "value": "Hi there!"}
        ]
    }
    chatml = pipeline.convert_sharegpt_to_chatml(sharegpt)
    assert len(chatml["messages"]) == 3
    assert chatml["messages"][0]["role"] == "system"
    assert chatml["messages"][1]["role"] == "user"
    assert chatml["messages"][2]["role"] == "assistant"

def test_decontamination_filtering(pipeline):
    train = [
        {"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "Paris"}]},
        {"messages": [{"role": "user", "content": "Write a python sorting algorithm for integers"}, {"role": "assistant", "content": "def sort(): pass"}]}
    ]
    eval_queries = ["What is the capital of France and Germany?"]
    
    clean, filtered = pipeline.decontaminate_samples(train, eval_queries, n_gram_size=5)
    assert filtered == 1
    assert len(clean) == 1
    assert "sorting algorithm" in clean[0]["messages"][0]["content"]

def test_token_statistics(pipeline):
    samples = [
        {"messages": [{"role": "user", "content": "one two three"}, {"role": "assistant", "content": "four five"}]},
        {"messages": [{"role": "user", "content": "six seven"}, {"role": "assistant", "content": "eight nine ten eleven twelve"}]}
    ]
    stats = pipeline.compute_token_statistics(samples)
    assert stats["count"] == 2
    assert stats["mean"] == 6.0
    assert stats["max"] == 7.0

def test_loss_masking_labels(pipeline):
    sample = {
        "messages": [
            {"role": "system", "content": "You are a code assistant."},
            {"role": "user", "content": "Write hello world"},
            {"role": "assistant", "content": "print('hello world')"}
        ]
    }
    vocab = {"<|im_start|>system": 1, "<|im_start|>user": 2, "<|im_start|>assistant": 3, "<|im_end|>": 4}
    res = pipeline.generate_loss_masked_labels(sample, vocab)
    
    input_ids = res["input_ids"]
    labels = res["labels"]
    assert len(input_ids) == len(labels)
    
    # First part must be masked with -100
    assert labels[0] == -100
    # Assistant response tokens must NOT be -100
    assistant_tokens = [l for l in labels if l != -100]
    assert len(assistant_tokens) > 0

Troubleshooting

Troubleshooting

If loss masking masks all tokens, ensure the role tag matches 'assistant'.

Security notes

Security

Pure Python data preprocessing pipeline. Operates strictly in-memory without external network calls.