LLMs and Generative AI › Customizing and Running Models › Day 276
Hands-on lab — Day 276: Building Fine-Tuning Datasets
- ← Back to the Day 276 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-276-building-fine-tuning-datasets/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/dataset_pipeline.py Test
bash tests/run_tests.sh File tree
examples/dataset_pipeline.py examples/test_dataset_pipeline.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/dataset_pipeline.py starter/test_dataset_pipeline.py tests/run_tests.sh tests/test_dataset_pipeline.py troubleshooting.md
Lab README
Day 276 Lab: Dataset Processing & Validation Pipeline
Lesson
- Lesson title: Building Fine-Tuning Datasets
- Day number: 276 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-276-building-fine-tuning-datasets
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-276-building-fine-tuning-datasetswhen the site is running.
Purpose
Build a robust fine-tuning dataset curation and preprocessing pipeline in Python supporting Alpaca/ShareGPT conversion to ChatML, n-gram decontamination filtering, token length analytics, and response-only loss masking.
Learning objectives
- Convert Alpaca and ShareGPT schemas to standardized ChatML format.
- Execute automated n-gram decontamination against benchmark query sets.
- Calculate token length statistics (mean, median, P95, max).
- Construct PyTorch-compatible
-100response loss masks.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/dataset_pipeline.py: Starter implementationexamples/dataset_pipeline.py: Reference implementationtests/test_dataset_pipeline.py: Unit test suiteexpected-output/: Captured execution logs
How to run
python3 examples/dataset_pipeline.py
What the commands do
- Processes conversation datasets, applies decontamination filters, and formats ChatML sequences with response loss masks.
Expected output
[DATASET] Ingested 100 samples | Decontaminated: 2 matches
[MASKING] System/User masked with -100 | Assistant tokens preserved
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Correct schema translation from Alpaca and ShareGPT to ChatML
- N-gram overlap detection and sample decontamination
- Token summary statistics calculation
- Exact prompt token masking (-100) and response token retention
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Verify regex word extraction re.findall(r'\b\w+\b', text.lower()) correctly captures clean tokens.
Security notes
Runs completely offline on local CPU. Zero network transmission.
Extension exercises
Implement dynamic multi-sample sequence packing up to 2,048 tokens.
Navigation
Day number: 276 of 365
Expected output
FIELDS.md
Field definitions for dataset pipeline. examples-run.txt
Example execution complete.
measured-values.txt
Samples_Ingested=100
Decontaminated_Count=2
Mean_Length=84
run-output.txt
[DATASET] Ingested 100 samples | Decontaminated: 2 matches
[MASKING] System/User masked with -100 | Assistant tokens preserved
starter-run.txt
Starter execution complete.
test-output.txt
test-run.txt
test_dataset_pipeline.py ..... [100%]
5 passed in 0.04s
Source files
examples/dataset_pipeline.py (4610 bytes)
# Fine-Tuning Dataset Processing & Validation Pipeline
import re
from typing import Dict, List, Any, Set, Tuple
class DatasetPipeline:
"""End-to-end dataset transformation, decontamination, and loss-masking engine."""
def __init__(self, pad_token_id: int = 0, ignore_index: int = -100):
self.pad_token_id = pad_token_id
self.ignore_index = ignore_index
def convert_alpaca_to_chatml(self, alpaca_item: Dict[str, str]) -> Dict[str, List[Dict[str, str]]]:
"""Converts Alpaca (instruction, input, output) to ChatML format."""
user_content = alpaca_item["instruction"]
if alpaca_item.get("input") and alpaca_item["input"].strip():
user_content += "\n\n" + alpaca_item["input"].strip()
return {
"messages": [
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": user_content},
{"role": "assistant", "content": alpaca_item["output"]}
]
}
def convert_sharegpt_to_chatml(self, sharegpt_item: Dict[str, Any]) -> Dict[str, List[Dict[str, str]]]:
"""Converts ShareGPT conversations list to ChatML format."""
role_map = {"human": "user", "gpt": "assistant", "system": "system"}
messages = []
for msg in sharegpt_item["conversations"]:
role = role_map.get(msg["from"], "user")
messages.append({"role": role, "content": msg["value"]})
return {"messages": messages}
def decontaminate_samples(
self,
train_samples: List[Dict[str, Any]],
eval_queries: List[str],
n_gram_size: int = 5
) -> Tuple[List[Dict[str, Any]], int]:
"""Filters out training samples that share n-gram overlaps with eval queries."""
eval_ngrams: Set[str] = set()
for q in eval_queries:
words = self._tokenize_words(q)
for i in range(len(words) - n_gram_size + 1):
eval_ngrams.add(" ".join(words[i:i + n_gram_size]))
clean_samples = []
filtered_count = 0
for sample in train_samples:
text = " ".join([m["content"] for m in sample["messages"]])
words = self._tokenize_words(text)
sample_ngrams = {" ".join(words[i:i + n_gram_size]) for i in range(len(words) - n_gram_size + 1)}
if sample_ngrams.intersection(eval_ngrams):
filtered_count += 1
else:
clean_samples.append(sample)
return clean_samples, filtered_count
def compute_token_statistics(self, samples: List[Dict[str, Any]]) -> Dict[str, float]:
"""Computes summary statistics on sequence word lengths."""
lengths = [sum(len(self._tokenize_words(m["content"])) for m in s["messages"]) for s in samples]
if not lengths:
return {"count": 0, "mean": 0.0, "median": 0.0, "p95": 0.0, "max": 0.0}
sorted_lens = sorted(lengths)
n = len(sorted_lens)
mean_val = sum(sorted_lens) / n
median_val = sorted_lens[n // 2]
p95_val = sorted_lens[int(n * 0.95)] if n > 0 else 0
max_val = sorted_lens[-1]
return {
"count": float(n),
"mean": float(mean_val),
"median": float(median_val),
"p95": float(p95_val),
"max": float(max_val)
}
def generate_loss_masked_labels(
self,
chatml_sample: Dict[str, List[Dict[str, str]]],
mock_vocab: Dict[str, int]
) -> Dict[str, List[int]]:
"""Simulates ChatML tokenization and generates -100 loss masks on prompt tokens."""
input_ids: List[int] = []
labels: List[int] = []
for msg in chatml_sample["messages"]:
role = msg["role"]
content = msg["content"]
# Format: <|im_start|>role\\ncontent<|im_end|>
words = [f"<|im_start|>{role}"] + self._tokenize_words(content) + ["<|im_end|>"]
tokens = [mock_vocab.get(w, 100) for w in words]
input_ids.extend(tokens)
if role in ["system", "user"]:
# Mask prompt tokens with ignore_index (-100)
labels.extend([self.ignore_index] * len(tokens))
else:
# Assistant tokens are trained (loss computed)
labels.extend(tokens)
return {
"input_ids": input_ids,
"labels": labels
}
def _tokenize_words(self, text: str) -> List[str]:
return re.findall(r'\b\w+\b', text.lower())
examples/test_dataset_pipeline.py (3156 bytes)
import pytest
from dataset_pipeline import DatasetPipeline
@pytest.fixture
def pipeline():
return DatasetPipeline()
def test_alpaca_to_chatml_conversion(pipeline):
alpaca = {
"instruction": "Translate to SQL",
"input": "Find users where age > 30",
"output": "SELECT * FROM users WHERE age > 30;"
}
chatml = pipeline.convert_alpaca_to_chatml(alpaca)
assert len(chatml["messages"]) == 3
assert chatml["messages"][0]["role"] == "system"
assert "Translate to SQL" in chatml["messages"][1]["content"]
assert "Find users where age > 30" in chatml["messages"][1]["content"]
assert chatml["messages"][2]["role"] == "assistant"
assert "SELECT * FROM users" in chatml["messages"][2]["content"]
def test_sharegpt_to_chatml_conversion(pipeline):
sharegpt = {
"conversations": [
{"from": "system", "value": "System init."},
{"from": "human", "value": "Hello!"},
{"from": "gpt", "value": "Hi there!"}
]
}
chatml = pipeline.convert_sharegpt_to_chatml(sharegpt)
assert len(chatml["messages"]) == 3
assert chatml["messages"][0]["role"] == "system"
assert chatml["messages"][1]["role"] == "user"
assert chatml["messages"][2]["role"] == "assistant"
def test_decontamination_filtering(pipeline):
train = [
{"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "Paris"}]},
{"messages": [{"role": "user", "content": "Write a python sorting algorithm for integers"}, {"role": "assistant", "content": "def sort(): pass"}]}
]
eval_queries = ["What is the capital of France and Germany?"]
clean, filtered = pipeline.decontaminate_samples(train, eval_queries, n_gram_size=5)
assert filtered == 1
assert len(clean) == 1
assert "sorting algorithm" in clean[0]["messages"][0]["content"]
def test_token_statistics(pipeline):
samples = [
{"messages": [{"role": "user", "content": "one two three"}, {"role": "assistant", "content": "four five"}]},
{"messages": [{"role": "user", "content": "six seven"}, {"role": "assistant", "content": "eight nine ten eleven twelve"}]}
]
stats = pipeline.compute_token_statistics(samples)
assert stats["count"] == 2
assert stats["mean"] == 6.0
assert stats["max"] == 7.0
def test_loss_masking_labels(pipeline):
sample = {
"messages": [
{"role": "system", "content": "You are a code assistant."},
{"role": "user", "content": "Write hello world"},
{"role": "assistant", "content": "print('hello world')"}
]
}
vocab = {"<|im_start|>system": 1, "<|im_start|>user": 2, "<|im_start|>assistant": 3, "<|im_end|>": 4}
res = pipeline.generate_loss_masked_labels(sample, vocab)
input_ids = res["input_ids"]
labels = res["labels"]
assert len(input_ids) == len(labels)
# First part must be masked with -100
assert labels[0] == -100
# Assistant response tokens must NOT be -100
assistant_tokens = [l for l in labels if l != -100]
assert len(assistant_tokens) > 0
metadata.yml (403 bytes)
lesson_id: D276
day: 276
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/dataset_pipeline.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/dataset_pipeline.py (846 bytes)
# Starter: Dataset Processing Pipeline
from typing import Dict, List, Any, Tuple
class DatasetPipeline:
def __init__(self, pad_token_id: int = 0, ignore_index: int = -100):
self.pad_token_id = pad_token_id
self.ignore_index = ignore_index
def convert_alpaca_to_chatml(self, alpaca_item: Dict[str, str]) -> Dict[str, List[Dict[str, str]]]:
return {"messages": []}
def decontaminate_samples(
self,
train_samples: List[Dict[str, Any]],
eval_queries: List[str],
n_gram_size: int = 5
) -> Tuple[List[Dict[str, Any]], int]:
return train_samples, 0
def generate_loss_masked_labels(
self,
chatml_sample: Dict[str, List[Dict[str, str]]],
mock_vocab: Dict[str, int]
) -> Dict[str, List[int]]:
return {"input_ids": [], "labels": []}
starter/test_dataset_pipeline.py (3156 bytes)
import pytest
from dataset_pipeline import DatasetPipeline
@pytest.fixture
def pipeline():
return DatasetPipeline()
def test_alpaca_to_chatml_conversion(pipeline):
alpaca = {
"instruction": "Translate to SQL",
"input": "Find users where age > 30",
"output": "SELECT * FROM users WHERE age > 30;"
}
chatml = pipeline.convert_alpaca_to_chatml(alpaca)
assert len(chatml["messages"]) == 3
assert chatml["messages"][0]["role"] == "system"
assert "Translate to SQL" in chatml["messages"][1]["content"]
assert "Find users where age > 30" in chatml["messages"][1]["content"]
assert chatml["messages"][2]["role"] == "assistant"
assert "SELECT * FROM users" in chatml["messages"][2]["content"]
def test_sharegpt_to_chatml_conversion(pipeline):
sharegpt = {
"conversations": [
{"from": "system", "value": "System init."},
{"from": "human", "value": "Hello!"},
{"from": "gpt", "value": "Hi there!"}
]
}
chatml = pipeline.convert_sharegpt_to_chatml(sharegpt)
assert len(chatml["messages"]) == 3
assert chatml["messages"][0]["role"] == "system"
assert chatml["messages"][1]["role"] == "user"
assert chatml["messages"][2]["role"] == "assistant"
def test_decontamination_filtering(pipeline):
train = [
{"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "Paris"}]},
{"messages": [{"role": "user", "content": "Write a python sorting algorithm for integers"}, {"role": "assistant", "content": "def sort(): pass"}]}
]
eval_queries = ["What is the capital of France and Germany?"]
clean, filtered = pipeline.decontaminate_samples(train, eval_queries, n_gram_size=5)
assert filtered == 1
assert len(clean) == 1
assert "sorting algorithm" in clean[0]["messages"][0]["content"]
def test_token_statistics(pipeline):
samples = [
{"messages": [{"role": "user", "content": "one two three"}, {"role": "assistant", "content": "four five"}]},
{"messages": [{"role": "user", "content": "six seven"}, {"role": "assistant", "content": "eight nine ten eleven twelve"}]}
]
stats = pipeline.compute_token_statistics(samples)
assert stats["count"] == 2
assert stats["mean"] == 6.0
assert stats["max"] == 7.0
def test_loss_masking_labels(pipeline):
sample = {
"messages": [
{"role": "system", "content": "You are a code assistant."},
{"role": "user", "content": "Write hello world"},
{"role": "assistant", "content": "print('hello world')"}
]
}
vocab = {"<|im_start|>system": 1, "<|im_start|>user": 2, "<|im_start|>assistant": 3, "<|im_end|>": 4}
res = pipeline.generate_loss_masked_labels(sample, vocab)
input_ids = res["input_ids"]
labels = res["labels"]
assert len(input_ids) == len(labels)
# First part must be masked with -100
assert labels[0] == -100
# Assistant response tokens must NOT be -100
assistant_tokens = [l for l in labels if l != -100]
assert len(assistant_tokens) > 0
tests/run_tests.sh (125 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_dataset_pipeline.py -v
tests/test_dataset_pipeline.py (3156 bytes)
import pytest
from dataset_pipeline import DatasetPipeline
@pytest.fixture
def pipeline():
return DatasetPipeline()
def test_alpaca_to_chatml_conversion(pipeline):
alpaca = {
"instruction": "Translate to SQL",
"input": "Find users where age > 30",
"output": "SELECT * FROM users WHERE age > 30;"
}
chatml = pipeline.convert_alpaca_to_chatml(alpaca)
assert len(chatml["messages"]) == 3
assert chatml["messages"][0]["role"] == "system"
assert "Translate to SQL" in chatml["messages"][1]["content"]
assert "Find users where age > 30" in chatml["messages"][1]["content"]
assert chatml["messages"][2]["role"] == "assistant"
assert "SELECT * FROM users" in chatml["messages"][2]["content"]
def test_sharegpt_to_chatml_conversion(pipeline):
sharegpt = {
"conversations": [
{"from": "system", "value": "System init."},
{"from": "human", "value": "Hello!"},
{"from": "gpt", "value": "Hi there!"}
]
}
chatml = pipeline.convert_sharegpt_to_chatml(sharegpt)
assert len(chatml["messages"]) == 3
assert chatml["messages"][0]["role"] == "system"
assert chatml["messages"][1]["role"] == "user"
assert chatml["messages"][2]["role"] == "assistant"
def test_decontamination_filtering(pipeline):
train = [
{"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "Paris"}]},
{"messages": [{"role": "user", "content": "Write a python sorting algorithm for integers"}, {"role": "assistant", "content": "def sort(): pass"}]}
]
eval_queries = ["What is the capital of France and Germany?"]
clean, filtered = pipeline.decontaminate_samples(train, eval_queries, n_gram_size=5)
assert filtered == 1
assert len(clean) == 1
assert "sorting algorithm" in clean[0]["messages"][0]["content"]
def test_token_statistics(pipeline):
samples = [
{"messages": [{"role": "user", "content": "one two three"}, {"role": "assistant", "content": "four five"}]},
{"messages": [{"role": "user", "content": "six seven"}, {"role": "assistant", "content": "eight nine ten eleven twelve"}]}
]
stats = pipeline.compute_token_statistics(samples)
assert stats["count"] == 2
assert stats["mean"] == 6.0
assert stats["max"] == 7.0
def test_loss_masking_labels(pipeline):
sample = {
"messages": [
{"role": "system", "content": "You are a code assistant."},
{"role": "user", "content": "Write hello world"},
{"role": "assistant", "content": "print('hello world')"}
]
}
vocab = {"<|im_start|>system": 1, "<|im_start|>user": 2, "<|im_start|>assistant": 3, "<|im_end|>": 4}
res = pipeline.generate_loss_masked_labels(sample, vocab)
input_ids = res["input_ids"]
labels = res["labels"]
assert len(input_ids) == len(labels)
# First part must be masked with -100
assert labels[0] == -100
# Assistant response tokens must NOT be -100
assistant_tokens = [l for l in labels if l != -100]
assert len(assistant_tokens) > 0
Troubleshooting
Troubleshooting
If loss masking masks all tokens, ensure the role tag matches 'assistant'.
Security notes
Security
Pure Python data preprocessing pipeline. Operates strictly in-memory without external network calls.