LLMs and Generative AI › Embeddings and Vector Search › Day 263

Hands-on lab — Day 263: Chunking Strategies

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/chunking_strategies_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/chunking_strategies_lib.py
examples/test_chunking_strategies_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/chunking_strategies_lib.py
starter/test_chunking_strategies_lib.py
tests/run_tests.sh
tests/test_chunking_strategies_lib.py
troubleshooting.md

Lab README

Lab: Day 263 -- Chunking Strategies

Lesson

Day number: 263 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: Document Chunking Strategies, Sliding Windows, Markdown Parsing, and Hierarchies.

Purpose

Build and test a Multi-Strategy Document Chunker in Python. Implement fixed sliding windows with token overlaps, Markdown structure-aware splitting with breadcrumbs, and parent-child hierarchical chunk trees.

Learning objectives

  • Implement sliding window token chunking with parameterized boundary overlaps.
  • Extract logical document sections from Markdown files while maintaining heading context.
  • Build parent-child hierarchical chunk mappings to decouple retrieval matching from generation context.
  • Prevent semantic boundary severance in structured documents.

Prerequisites

  • Day 262 (Vector Databases).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Pure Python standard libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/chunking_strategies_lib.py: Student scaffold file.
  • examples/chunking_strategies_lib.py: Complete reference implementation.
  • tests/test_chunking_strategies_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/chunking_strategies_lib.py

What the commands do

  • Executes sliding window token chunking.
  • Parses Markdown documents into section chunks.
  • Builds parent-child hierarchy structures.
  • Runs unit test assertions.

Expected output

Chunker Demo Executed. Generated 2 overlapping chunks.

Validation steps

  1. Verify sliding windows maintain exact overlap counts.
  2. Confirm Markdown parser preserves heading text.
  3. Validate child chunks reference parent document IDs.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Infinite loop: Verify overlap is strictly less than chunk_size.

Security notes

All computations execute locally.

Extension exercises

  1. Implement a Python AST chunker splitting along function declarations.
  2. Implement cosine-distance semantic chunking.
  • Lesson title: Chunking Strategies
  • Day number: 263 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-263-chunking-strategies
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-263-chunking-strategies when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 263

- `Chunker Demo`: Status of document chunking execution.

examples-run.txt

Chunker Demo Executed. Generated 2 overlapping chunks.

measured-values.txt

Chunker Demo: Success
Generated Chunks: 3

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_chunking_strategies_lib.py::test_sliding_window_chunking PASSED       [ 25%]
tests/test_chunking_strategies_lib.py::test_markdown_header_chunking PASSED      [ 50%]
tests/test_chunking_strategies_lib.py::test_parent_child_hierarchy PASSED        [ 75%]
tests/test_chunking_strategies_lib.py::test_empty_string_handling PASSED         [100%]

============================== 4 passed in 0.28s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/chunking_strategies_lib.py (3526 bytes)
import re
from typing import List, Dict, Any

class DocumentChunker:
    def __init__(self, chunk_size: int = 200, overlap: int = 40):
        self.chunk_size = chunk_size
        self.overlap = overlap

    def sliding_window_chunk(self, text: str) -> List[Dict[str, Any]]:
        words = text.split()
        if not words:
            return []
        
        chunks = []
        stride = max(1, self.chunk_size - self.overlap)
        
        for i in range(0, len(words), stride):
            chunk_words = words[i:i + self.chunk_size]
            chunks.append({
                "chunk_id": len(chunks),
                "text": " ".join(chunk_words),
                "word_count": len(chunk_words),
                "start_idx": i,
                "end_idx": i + len(chunk_words)
            })
            if i + self.chunk_size >= len(words):
                break
        return chunks

    def markdown_header_chunk(self, markdown_text: str) -> List[Dict[str, Any]]:
        lines = markdown_text.split("\n")
        chunks = []
        current_header = "Introduction"
        current_lines = []

        for line in lines:
            if line.startswith("#"):
                if current_lines:
                    text_content = "\n".join(current_lines).strip()
                    if text_content:
                        chunks.append({
                            "chunk_id": len(chunks),
                            "header": current_header,
                            "text": text_content
                        })
                    current_lines = []
                current_header = line.strip("# ").strip()
            else:
                current_lines.append(line)

        if current_lines:
            text_content = "\n".join(current_lines).strip()
            if text_content:
                chunks.append({
                    "chunk_id": len(chunks),
                    "header": current_header,
                    "text": text_content
                })
        return chunks

    def create_parent_child_hierarchy(self, text: str, parent_size: int = 400, child_size: int = 100) -> List[Dict[str, Any]]:
        words = text.split()
        if not words:
            return []

        parents = []
        for p_idx in range(0, len(words), parent_size):
            p_words = words[p_idx:p_idx + parent_size]
            parent_id = f"parent_{len(parents)}"
            parent_text = " ".join(p_words)
            
            # Subdivide parent into children
            children = []
            for c_idx in range(0, len(p_words), child_size):
                c_words = p_words[c_idx:c_idx + child_size]
                children.append({
                    "child_id": f"{parent_id}_child_{len(children)}",
                    "parent_id": parent_id,
                    "text": " ".join(c_words)
                })
            
            parents.append({
                "parent_id": parent_id,
                "parent_text": parent_text,
                "children": children
            })
            if p_idx + parent_size >= len(words):
                break
        return parents

def run_chunker_demo():
    chunker = DocumentChunker(chunk_size=10, overlap=3)
    sample_text = "The quick brown fox jumps over the lazy dog and runs across the wide green open meadow."
    chunks = chunker.sliding_window_chunk(sample_text)
    print(f"Chunker Demo Executed. Generated {len(chunks)} overlapping chunks.")
    return chunks

if __name__ == "__main__":
    run_chunker_demo()
examples/test_chunking_strategies_lib.py (1498 bytes)
import pytest
from examples.chunking_strategies_lib import (
    DocumentChunker
)

def test_sliding_window_chunking():
    chunker = DocumentChunker(chunk_size=5, overlap=2)
    text = "one two three four five six seven eight nine ten"
    chunks = chunker.sliding_window_chunk(text)

    assert len(chunks) == 3
    assert chunks[0]["text"] == "one two three four five"
    assert chunks[1]["text"] == "four five six seven eight"
    assert chunks[2]["text"] == "seven eight nine ten"

def test_markdown_header_chunking():
    chunker = DocumentChunker()
    md = """# Title
This is intro text.
## Section 1
Content of section 1.
## Section 2
Content of section 2."""
    
    chunks = chunker.markdown_header_chunk(md)
    assert len(chunks) == 3
    assert chunks[0]["header"] == "Title"
    assert chunks[0]["text"] == "This is intro text."
    assert chunks[1]["header"] == "Section 1"
    assert chunks[2]["header"] == "Section 2"

def test_parent_child_hierarchy():
    chunker = DocumentChunker()
    text = "word " * 300
    hierarchy = chunker.create_parent_child_hierarchy(text, parent_size=100, child_size=25)
    
    assert len(hierarchy) == 3
    assert len(hierarchy[0]["children"]) == 4
    assert hierarchy[0]["children"][0]["parent_id"] == "parent_0"

def test_empty_string_handling():
    chunker = DocumentChunker()
    assert chunker.sliding_window_chunk("") == []
    assert chunker.markdown_header_chunk("") == []
    assert chunker.create_parent_child_hierarchy("") == []
metadata.yml (432 bytes)
lesson_id: D263
day: 263
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/chunking_strategies_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/chunking_strategies_lib.py (735 bytes)
import re
from typing import List, Dict, Any

class DocumentChunker:
    def __init__(self, chunk_size: int = 200, overlap: int = 40):
        self.chunk_size = chunk_size
        self.overlap = overlap

    def sliding_window_chunk(self, text: str) -> List[Dict[str, Any]]:
        # TODO: Implement sliding window chunking with overlap
        pass

    def markdown_header_chunk(self, markdown_text: str) -> List[Dict[str, Any]]:
        # TODO: Implement Markdown header splitting with breadcrumb preservation
        pass

    def create_parent_child_hierarchy(self, text: str, parent_size: int = 400, child_size: int = 100) -> List[Dict[str, Any]]:
        # TODO: Implement parent-child hierarchical chunk mappings
        pass
starter/test_chunking_strategies_lib.py (1498 bytes)
import pytest
from examples.chunking_strategies_lib import (
    DocumentChunker
)

def test_sliding_window_chunking():
    chunker = DocumentChunker(chunk_size=5, overlap=2)
    text = "one two three four five six seven eight nine ten"
    chunks = chunker.sliding_window_chunk(text)

    assert len(chunks) == 3
    assert chunks[0]["text"] == "one two three four five"
    assert chunks[1]["text"] == "four five six seven eight"
    assert chunks[2]["text"] == "seven eight nine ten"

def test_markdown_header_chunking():
    chunker = DocumentChunker()
    md = """# Title
This is intro text.
## Section 1
Content of section 1.
## Section 2
Content of section 2."""
    
    chunks = chunker.markdown_header_chunk(md)
    assert len(chunks) == 3
    assert chunks[0]["header"] == "Title"
    assert chunks[0]["text"] == "This is intro text."
    assert chunks[1]["header"] == "Section 1"
    assert chunks[2]["header"] == "Section 2"

def test_parent_child_hierarchy():
    chunker = DocumentChunker()
    text = "word " * 300
    hierarchy = chunker.create_parent_child_hierarchy(text, parent_size=100, child_size=25)
    
    assert len(hierarchy) == 3
    assert len(hierarchy[0]["children"]) == 4
    assert hierarchy[0]["children"][0]["parent_id"] == "parent_0"

def test_empty_string_handling():
    chunker = DocumentChunker()
    assert chunker.sliding_window_chunk("") == []
    assert chunker.markdown_header_chunk("") == []
    assert chunker.create_parent_child_hierarchy("") == []
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 263 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_chunking_strategies_lib.py (1498 bytes)
import pytest
from examples.chunking_strategies_lib import (
    DocumentChunker
)

def test_sliding_window_chunking():
    chunker = DocumentChunker(chunk_size=5, overlap=2)
    text = "one two three four five six seven eight nine ten"
    chunks = chunker.sliding_window_chunk(text)

    assert len(chunks) == 3
    assert chunks[0]["text"] == "one two three four five"
    assert chunks[1]["text"] == "four five six seven eight"
    assert chunks[2]["text"] == "seven eight nine ten"

def test_markdown_header_chunking():
    chunker = DocumentChunker()
    md = """# Title
This is intro text.
## Section 1
Content of section 1.
## Section 2
Content of section 2."""
    
    chunks = chunker.markdown_header_chunk(md)
    assert len(chunks) == 3
    assert chunks[0]["header"] == "Title"
    assert chunks[0]["text"] == "This is intro text."
    assert chunks[1]["header"] == "Section 1"
    assert chunks[2]["header"] == "Section 2"

def test_parent_child_hierarchy():
    chunker = DocumentChunker()
    text = "word " * 300
    hierarchy = chunker.create_parent_child_hierarchy(text, parent_size=100, child_size=25)
    
    assert len(hierarchy) == 3
    assert len(hierarchy[0]["children"]) == 4
    assert hierarchy[0]["children"][0]["parent_id"] == "parent_0"

def test_empty_string_handling():
    chunker = DocumentChunker()
    assert chunker.sliding_window_chunk("") == []
    assert chunker.markdown_header_chunk("") == []
    assert chunker.create_parent_child_hierarchy("") == []

Troubleshooting

Troubleshooting: Day 263 - Chunking Strategies

Common Issues

  1. Empty chunks produced:
    • Cause: Consecutive newlines or empty header sections.
    • Fix: Check text_content.strip() before appending to chunks list.

Security notes

Security & Privacy: Day 263 - Chunking Strategies

Security Guidance

  • Sanitize and redact PII/credentials prior to document chunking and indexing.