LLMs and Generative AI › Embeddings and Vector Search › Day 263
Hands-on lab — Day 263: Chunking Strategies
- ← Back to the Day 263 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-263-chunking-strategies/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/chunking_strategies_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/chunking_strategies_lib.py examples/test_chunking_strategies_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/chunking_strategies_lib.py starter/test_chunking_strategies_lib.py tests/run_tests.sh tests/test_chunking_strategies_lib.py troubleshooting.md
Lab README
Lab: Day 263 -- Chunking Strategies
Lesson
Day number: 263 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: Document Chunking Strategies, Sliding Windows, Markdown Parsing, and Hierarchies.
Purpose
Build and test a Multi-Strategy Document Chunker in Python. Implement fixed sliding windows with token overlaps, Markdown structure-aware splitting with breadcrumbs, and parent-child hierarchical chunk trees.
Learning objectives
- Implement sliding window token chunking with parameterized boundary overlaps.
- Extract logical document sections from Markdown files while maintaining heading context.
- Build parent-child hierarchical chunk mappings to decouple retrieval matching from generation context.
- Prevent semantic boundary severance in structured documents.
Prerequisites
- Day 262 (Vector Databases).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 512 MB RAM.
- 50 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Pure Python standard libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/chunking_strategies_lib.py: Student scaffold file.examples/chunking_strategies_lib.py: Complete reference implementation.tests/test_chunking_strategies_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/chunking_strategies_lib.py
What the commands do
- Executes sliding window token chunking.
- Parses Markdown documents into section chunks.
- Builds parent-child hierarchy structures.
- Runs unit test assertions.
Expected output
Chunker Demo Executed. Generated 2 overlapping chunks.
Validation steps
- Verify sliding windows maintain exact overlap counts.
- Confirm Markdown parser preserves heading text.
- Validate child chunks reference parent document IDs.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Infinite loop: Verify overlap is strictly less than chunk_size.
Security notes
All computations execute locally.
Extension exercises
- Implement a Python AST chunker splitting along function declarations.
- Implement cosine-distance semantic chunking.
Navigation
- Lesson title: Chunking Strategies
- Day number: 263 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-263-chunking-strategies
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-263-chunking-strategieswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 263
- `Chunker Demo`: Status of document chunking execution.
examples-run.txt
Chunker Demo Executed. Generated 2 overlapping chunks.
measured-values.txt
Chunker Demo: Success
Generated Chunks: 3
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 4 items
tests/test_chunking_strategies_lib.py::test_sliding_window_chunking PASSED [ 25%]
tests/test_chunking_strategies_lib.py::test_markdown_header_chunking PASSED [ 50%]
tests/test_chunking_strategies_lib.py::test_parent_child_hierarchy PASSED [ 75%]
tests/test_chunking_strategies_lib.py::test_empty_string_handling PASSED [100%]
============================== 4 passed in 0.28s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/chunking_strategies_lib.py (3526 bytes)
import re
from typing import List, Dict, Any
class DocumentChunker:
def __init__(self, chunk_size: int = 200, overlap: int = 40):
self.chunk_size = chunk_size
self.overlap = overlap
def sliding_window_chunk(self, text: str) -> List[Dict[str, Any]]:
words = text.split()
if not words:
return []
chunks = []
stride = max(1, self.chunk_size - self.overlap)
for i in range(0, len(words), stride):
chunk_words = words[i:i + self.chunk_size]
chunks.append({
"chunk_id": len(chunks),
"text": " ".join(chunk_words),
"word_count": len(chunk_words),
"start_idx": i,
"end_idx": i + len(chunk_words)
})
if i + self.chunk_size >= len(words):
break
return chunks
def markdown_header_chunk(self, markdown_text: str) -> List[Dict[str, Any]]:
lines = markdown_text.split("\n")
chunks = []
current_header = "Introduction"
current_lines = []
for line in lines:
if line.startswith("#"):
if current_lines:
text_content = "\n".join(current_lines).strip()
if text_content:
chunks.append({
"chunk_id": len(chunks),
"header": current_header,
"text": text_content
})
current_lines = []
current_header = line.strip("# ").strip()
else:
current_lines.append(line)
if current_lines:
text_content = "\n".join(current_lines).strip()
if text_content:
chunks.append({
"chunk_id": len(chunks),
"header": current_header,
"text": text_content
})
return chunks
def create_parent_child_hierarchy(self, text: str, parent_size: int = 400, child_size: int = 100) -> List[Dict[str, Any]]:
words = text.split()
if not words:
return []
parents = []
for p_idx in range(0, len(words), parent_size):
p_words = words[p_idx:p_idx + parent_size]
parent_id = f"parent_{len(parents)}"
parent_text = " ".join(p_words)
# Subdivide parent into children
children = []
for c_idx in range(0, len(p_words), child_size):
c_words = p_words[c_idx:c_idx + child_size]
children.append({
"child_id": f"{parent_id}_child_{len(children)}",
"parent_id": parent_id,
"text": " ".join(c_words)
})
parents.append({
"parent_id": parent_id,
"parent_text": parent_text,
"children": children
})
if p_idx + parent_size >= len(words):
break
return parents
def run_chunker_demo():
chunker = DocumentChunker(chunk_size=10, overlap=3)
sample_text = "The quick brown fox jumps over the lazy dog and runs across the wide green open meadow."
chunks = chunker.sliding_window_chunk(sample_text)
print(f"Chunker Demo Executed. Generated {len(chunks)} overlapping chunks.")
return chunks
if __name__ == "__main__":
run_chunker_demo()
examples/test_chunking_strategies_lib.py (1498 bytes)
import pytest
from examples.chunking_strategies_lib import (
DocumentChunker
)
def test_sliding_window_chunking():
chunker = DocumentChunker(chunk_size=5, overlap=2)
text = "one two three four five six seven eight nine ten"
chunks = chunker.sliding_window_chunk(text)
assert len(chunks) == 3
assert chunks[0]["text"] == "one two three four five"
assert chunks[1]["text"] == "four five six seven eight"
assert chunks[2]["text"] == "seven eight nine ten"
def test_markdown_header_chunking():
chunker = DocumentChunker()
md = """# Title
This is intro text.
## Section 1
Content of section 1.
## Section 2
Content of section 2."""
chunks = chunker.markdown_header_chunk(md)
assert len(chunks) == 3
assert chunks[0]["header"] == "Title"
assert chunks[0]["text"] == "This is intro text."
assert chunks[1]["header"] == "Section 1"
assert chunks[2]["header"] == "Section 2"
def test_parent_child_hierarchy():
chunker = DocumentChunker()
text = "word " * 300
hierarchy = chunker.create_parent_child_hierarchy(text, parent_size=100, child_size=25)
assert len(hierarchy) == 3
assert len(hierarchy[0]["children"]) == 4
assert hierarchy[0]["children"][0]["parent_id"] == "parent_0"
def test_empty_string_handling():
chunker = DocumentChunker()
assert chunker.sliding_window_chunk("") == []
assert chunker.markdown_header_chunk("") == []
assert chunker.create_parent_child_hierarchy("") == []
metadata.yml (432 bytes)
lesson_id: D263
day: 263
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/chunking_strategies_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/chunking_strategies_lib.py (735 bytes)
import re
from typing import List, Dict, Any
class DocumentChunker:
def __init__(self, chunk_size: int = 200, overlap: int = 40):
self.chunk_size = chunk_size
self.overlap = overlap
def sliding_window_chunk(self, text: str) -> List[Dict[str, Any]]:
# TODO: Implement sliding window chunking with overlap
pass
def markdown_header_chunk(self, markdown_text: str) -> List[Dict[str, Any]]:
# TODO: Implement Markdown header splitting with breadcrumb preservation
pass
def create_parent_child_hierarchy(self, text: str, parent_size: int = 400, child_size: int = 100) -> List[Dict[str, Any]]:
# TODO: Implement parent-child hierarchical chunk mappings
pass
starter/test_chunking_strategies_lib.py (1498 bytes)
import pytest
from examples.chunking_strategies_lib import (
DocumentChunker
)
def test_sliding_window_chunking():
chunker = DocumentChunker(chunk_size=5, overlap=2)
text = "one two three four five six seven eight nine ten"
chunks = chunker.sliding_window_chunk(text)
assert len(chunks) == 3
assert chunks[0]["text"] == "one two three four five"
assert chunks[1]["text"] == "four five six seven eight"
assert chunks[2]["text"] == "seven eight nine ten"
def test_markdown_header_chunking():
chunker = DocumentChunker()
md = """# Title
This is intro text.
## Section 1
Content of section 1.
## Section 2
Content of section 2."""
chunks = chunker.markdown_header_chunk(md)
assert len(chunks) == 3
assert chunks[0]["header"] == "Title"
assert chunks[0]["text"] == "This is intro text."
assert chunks[1]["header"] == "Section 1"
assert chunks[2]["header"] == "Section 2"
def test_parent_child_hierarchy():
chunker = DocumentChunker()
text = "word " * 300
hierarchy = chunker.create_parent_child_hierarchy(text, parent_size=100, child_size=25)
assert len(hierarchy) == 3
assert len(hierarchy[0]["children"]) == 4
assert hierarchy[0]["children"][0]["parent_id"] == "parent_0"
def test_empty_string_handling():
chunker = DocumentChunker()
assert chunker.sliding_window_chunk("") == []
assert chunker.markdown_header_chunk("") == []
assert chunker.create_parent_child_hierarchy("") == []
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 263 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_chunking_strategies_lib.py (1498 bytes)
import pytest
from examples.chunking_strategies_lib import (
DocumentChunker
)
def test_sliding_window_chunking():
chunker = DocumentChunker(chunk_size=5, overlap=2)
text = "one two three four five six seven eight nine ten"
chunks = chunker.sliding_window_chunk(text)
assert len(chunks) == 3
assert chunks[0]["text"] == "one two three four five"
assert chunks[1]["text"] == "four five six seven eight"
assert chunks[2]["text"] == "seven eight nine ten"
def test_markdown_header_chunking():
chunker = DocumentChunker()
md = """# Title
This is intro text.
## Section 1
Content of section 1.
## Section 2
Content of section 2."""
chunks = chunker.markdown_header_chunk(md)
assert len(chunks) == 3
assert chunks[0]["header"] == "Title"
assert chunks[0]["text"] == "This is intro text."
assert chunks[1]["header"] == "Section 1"
assert chunks[2]["header"] == "Section 2"
def test_parent_child_hierarchy():
chunker = DocumentChunker()
text = "word " * 300
hierarchy = chunker.create_parent_child_hierarchy(text, parent_size=100, child_size=25)
assert len(hierarchy) == 3
assert len(hierarchy[0]["children"]) == 4
assert hierarchy[0]["children"][0]["parent_id"] == "parent_0"
def test_empty_string_handling():
chunker = DocumentChunker()
assert chunker.sliding_window_chunk("") == []
assert chunker.markdown_header_chunk("") == []
assert chunker.create_parent_child_hierarchy("") == []
Troubleshooting
Troubleshooting: Day 263 - Chunking Strategies
Common Issues
- Empty chunks produced:
- Cause: Consecutive newlines or empty header sections.
- Fix: Check text_content.strip() before appending to chunks list.
Security notes
Security & Privacy: Day 263 - Chunking Strategies
Security Guidance
- Sanitize and redact PII/credentials prior to document chunking and indexing.