AI Engineering: Agents and Applications › Evaluation and Reliability › Day 310
Hands-on lab — Day 310: Building Evaluation Datasets
- ← Back to the Day 310 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-310-building-evaluation-datasets/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/dataset_curator.py Test
bash tests/run_tests.sh File tree
examples/dataset_curator.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/dataset_curator.py tests/run_tests.sh tests/test_dataset_curator.py troubleshooting.md
Lab README
Day 310 Lab: Building Evaluation Datasets
Lesson
- Lesson title: Building Evaluation Datasets
- Day number: 310 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-310-building-evaluation-datasets
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-310-building-evaluation-datasetswhen the site is running.
Purpose
Implement an Evaluation Dataset Curator in Python that loads, validates, de-duplicates, stratifies, and exports JSONL evaluation datasets.
Learning objectives
- Load and parse JSONL evaluation files.
- Validate record schemas and filter illegal categories.
- De-duplicate queries using normalized string matching.
- Compute stratified category distribution counts and export clean JSONL benchmarks.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/dataset_curator.py: Starter implementation skeletonexamples/dataset_curator.py: Verified reference implementationtests/test_dataset_curator.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/dataset_curator.py
What the commands do
- Curates, validates, and exports stratified evaluation datasets in JSONL format.
Expected output
Stratified counts: {'happy_path': 1, 'hard_negative': 0, 'schema_boundary': 0, 'adversarial': 0}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Record insertion and category validation
- Normalized query de-duplication
- Stratified category counting
- JSONL file loading and exporting
Cleanup
rm -rf .pytest_cache __pycache__ .venv *.jsonl
Troubleshooting
Ensure input records contain non-empty id, query, and expected_output fields.
Security notes
Runs completely offline on local CPU.
Extension exercises
Add JSON schema validation for complex structured outputs.
Navigation
Day number: 310 of 365
Expected output
FIELDS.md
# Measured Values
- `records_curated`: 25
- `stratified_categories`: 4
examples-run.txt
Stratified counts: {'happy_path': 1, 'hard_negative': 0, 'schema_boundary': 0, 'adversarial': 0}
measured-values.txt
records_curated=25
stratified_categories=4
run-output.txt
Stratified counts: {'happy_path': 1, 'hard_negative': 0, 'schema_boundary': 0, 'adversarial': 0}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.dataset_curator.EvalDatasetCurator object at 0x108cba780>
record_id = 'id_01', category = 'happy_path', query = 'Query 1'
expected_output = 'Ans 1', metadata = None
def add_record(self, record_id: str, category: str, query: str, expected_output: str, metadata: Optional[Dict[str, Any]]=None) -> bool:
> raise NotImplementedError('TASK 1: implement add_record.')
E NotImplementedError: TASK 1: implement add_record.
examples/dataset_curator.py:18: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_dataset_curator.py::test_add_record_valid - NotImplementedE...
FAILED tests/test_dataset_curator.py::test_add_record_invalid_category_or_empty
FAILED tests/test_dataset_curator.py::test_deduplication - NotImplementedErro...
FAILED tests/test_dataset_curator.py::test_stratified_counts - NotImplemented...
FAILED tests/test_dataset_curator.py::test_jsonl_export_and_import - NotImple...
============================== 5 failed in 0.04s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-310-building-evaluation-datasets
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_dataset_curator.py::test_add_record_valid PASSED [ 20%]
tests/test_dataset_curator.py::test_add_record_invalid_category_or_empty PASSED [ 40%]
tests/test_dataset_curator.py::test_deduplication PASSED [ 60%]
tests/test_dataset_curator.py::test_stratified_counts PASSED [ 80%]
tests/test_dataset_curator.py::test_jsonl_export_and_import PASSED [100%]
============================== 5 passed in 0.02s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-310-building-evaluation-datasets
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_dataset_curator.py::test_add_record_valid PASSED [ 20%]
tests/test_dataset_curator.py::test_add_record_invalid_category_or_empty PASSED [ 40%]
tests/test_dataset_curator.py::test_deduplication PASSED [ 60%]
tests/test_dataset_curator.py::test_stratified_counts PASSED [ 80%]
tests/test_dataset_curator.py::test_jsonl_export_and_import PASSED [100%]
============================== 5 passed in 0.02s ===============================
Source files
examples/dataset_curator.py (2443 bytes)
import json
import os
from typing import List, Dict, Any, Optional
class EvalDatasetCurator:
def __init__(self):
self.records: List[Dict[str, Any]] = []
def add_record(self, record_id: str, category: str, query: str, expected_output: str, metadata: Optional[Dict[str, Any]] = None) -> bool:
if not record_id or not query or not expected_output:
return False
valid_categories = {"happy_path", "hard_negative", "schema_boundary", "adversarial"}
if category not in valid_categories:
return False
for r in self.records:
if r["query"].strip().lower() == query.strip().lower():
return False
record = {
"id": str(record_id).strip(),
"category": category,
"query": query.strip(),
"expected_output": expected_output.strip(),
"metadata": metadata or {}
}
self.records.append(record)
return True
def get_stratified_counts(self) -> Dict[str, int]:
counts = {"happy_path": 0, "hard_negative": 0, "schema_boundary": 0, "adversarial": 0}
for r in self.records:
cat = r["category"]
counts[cat] = counts.get(cat, 0) + 1
return counts
def export_jsonl(self, filepath: str) -> int:
with open(filepath, "w", encoding="utf-8") as f:
for r in self.records:
f.write(json.dumps(r) + "\n")
return len(self.records)
def load_jsonl(self, filepath: str) -> int:
count = 0
if not os.path.exists(filepath):
return 0
with open(filepath, "r", encoding="utf-8") as f:
for line in f:
line_str = line.strip()
if not line_str:
continue
try:
data = json.loads(line_str)
if self.add_record(data.get("id"), data.get("category"), data.get("query"), data.get("expected_output"), data.get("metadata")):
count += 1
except Exception:
continue
return count
if __name__ == "__main__":
curator = EvalDatasetCurator()
curator.add_record("eval_01", "happy_path", "What is the return policy?", "30 days with receipt.")
print("Stratified counts:", curator.get_stratified_counts())
metadata.yml (410 bytes)
lesson_id: D310
day: 310
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/dataset_curator.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache *.jsonl
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/dataset_curator.py (1252 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import json
import os
from typing import List, Dict, Any, Optional
class EvalDatasetCurator:
def __init__(self):
self.records: List[Dict[str, Any]] = []
def add_record(self, record_id: str, category: str, query: str, expected_output: str, metadata: Optional[Dict[str, Any]]=None) -> bool:
raise NotImplementedError('TASK 1: implement add_record.')
def get_stratified_counts(self) -> Dict[str, int]:
raise NotImplementedError('TASK 2: implement get_stratified_counts.')
def export_jsonl(self, filepath: str) -> int:
raise NotImplementedError('TASK 3: implement export_jsonl.')
def load_jsonl(self, filepath: str) -> int:
raise NotImplementedError('TASK 4: implement load_jsonl.')
if __name__ == '__main__':
curator = EvalDatasetCurator()
curator.add_record('eval_01', 'happy_path', 'What is the return policy?', '30 days with receipt.')
print('Stratified counts:', curator.get_stratified_counts())
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_dataset_curator.py (1900 bytes)
import pytest
import os
from examples.dataset_curator import EvalDatasetCurator
def test_add_record_valid():
curator = EvalDatasetCurator()
assert curator.add_record("id_01", "happy_path", "Hello", "Hi there") is True
assert len(curator.records) == 1
def test_add_record_invalid_category_or_empty():
curator = EvalDatasetCurator()
assert curator.add_record("id_02", "unknown_category", "Hello", "Hi") is False
assert curator.add_record("id_03", "happy_path", "", "Hi") is False
assert curator.add_record("id_04", "happy_path", "Hello", "") is False
def test_deduplication():
curator = EvalDatasetCurator()
assert curator.add_record("id_01", "happy_path", "What is Python?", "A language") is True
assert curator.add_record("id_02", "happy_path", " what is python? ", "Duplicate") is False
assert len(curator.records) == 1
def test_stratified_counts():
curator = EvalDatasetCurator()
curator.add_record("id_01", "happy_path", "Query 1", "Ans 1")
curator.add_record("id_02", "hard_negative", "Query 2", "Ans 2")
curator.add_record("id_03", "adversarial", "Query 3", "Ans 3")
counts = curator.get_stratified_counts()
assert counts["happy_path"] == 1
assert counts["hard_negative"] == 1
assert counts["adversarial"] == 1
assert counts["schema_boundary"] == 0
def test_jsonl_export_and_import(tmp_path):
curator = EvalDatasetCurator()
curator.add_record("id_01", "happy_path", "Query 1", "Ans 1")
curator.add_record("id_02", "schema_boundary", "Query 2", "Ans 2")
file_path = str(tmp_path / "test_eval.jsonl")
exported_count = curator.export_jsonl(file_path)
assert exported_count == 2
assert os.path.exists(file_path)
new_curator = EvalDatasetCurator()
loaded_count = new_curator.load_jsonl(file_path)
assert loaded_count == 2
assert len(new_curator.records) == 2
Troubleshooting
Troubleshooting - Day 310
Ensure JSONL files have exactly one JSON object per line.
Security notes
Security - Day 310
Never store unencrypted production API keys or customer PII in evaluation datasets.