LLMs and Generative AI β€Ί Embeddings and Vector Search β€Ί Day 263

Day 263: Chunking Strategies

Day 263 of 365 β€” Chunking Strategies

Master document segmentation and text chunking strategies for production RAG pipelines: evaluate Fixed-Size Chunking with sliding token overlap, Markdown/Code Structure-Aware Boundary Parsing, Semantic Similarity Chunking via cosine thresholding, and Parent-Document / Hierarchical Chunking; analyze context window trade-offs, retrieval fragmentation, boundary loss, and implement a production multi-strategy chunker in Python.

Course
LLMs and Generative AI
Category
Embeddings and Vector Search
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-263-chunking-strategies

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-263-chunking-strategies
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

The single most consequential preprocessing step in any Retrieval-Augmented Generation (RAG) pipeline is Document Chunking. Even the most powerful embedding model (such as text-embedding-3-large) and the most sophisticated LLM (such as Claude 3.7 Sonnet) will fail if your chunking strategy is flawed:

Document chunking is the art and science of optimizing semantic density: ensuring every retrieved text unit contains exactly enough information to answer a user inquiry with maximum precision and zero surrounding noise.

Today, you will master Fixed-Size Sliding Windows, Markdown Structure-Aware Boundary Parsing, Semantic Similarity Chunking, and Parent-Child Hierarchical Retrieval, and build a production-grade multi-strategy document segmenter in Python.


The idea in plain language

Diagram: why chunk size trades specificity against self-containment

Imagine you are slicing a long, 500-page historical encyclopedia to create study flashcards:


Historical background

Chunking strategies comparative pipeline diagram

  1. 1960s–1980s (Text Passage Retrieval): Early information retrieval systems (SMART, Okapi BM25) experimented with fixed-paragraph vs. sliding-window text boundaries.
  2. 2020 (Dense Passage Retrieval - DPR): Karpukhin et al. standardized the 100-word fixed passage convention for training dense dual-encoders on Wikipedia.
  3. 2023 (Lost in the Middle): Liu et al. revealed that LLMs struggle to recall facts buried inside overly large context chunks, prompting the industry to shift toward smaller, high-precision retrieval units.
  4. 2024 (Semantic & Hierarchical Chunking): Modern RAG frameworks (LlamaIndex, LangChain) introduced embedding-based semantic boundary detection and Parent-Document retrieval architectures.

What it is β€” and what it is not

What Document Chunking IS:

What it is NOT:


Why it was created and what problems it solves

Production chunking solves 4 critical failure modes:

  1. Embedding Context Window Overflow: Prevents documents from exceeding model token limits (e.g. 512 or 8,192 tokens).
  2. Boundary Semantic Loss: Sliding token overlaps guarantee that facts located at chunk transitions are never lost.
  3. Structured Document Integrity: Prevents Markdown tables, JSON payloads, and Python code functions from being split across chunk borders.
  4. Optimal Retrieval Precision: Aligns the granularity of the retrieved passage with the granularity of the user query.

How it works

Let us analyze the 4 foundational chunking strategies, their trade-offs, and mathematical boundary conditions.


1. Fixed-Size Chunking with Sliding Token Overlap

Diagram: how overlapping chunk boundaries prevent information loss at cuts

Sliding window overlap context preservation diagram

The simplest and most widely used baseline across early RAG architectures:


2. Markdown / Code Structure-Aware Chunking

Diagram: fixed-size splitting compared with structure-aware splitting

Structure-aware segmentation parses document syntax trees rather than treating text as a flat string:

  1. Header Hierarchy Splitting: Segment by # H1, ## H2, and ### H3 markers.

  2. Breadcrumb Metadata Enrichment: Inject the ancestral heading hierarchy into the chunk metadata payload so that the embedding model understands document context:

    {
      "document": "cloud_architecture.md",
      "breadcrumb": "Architecture > Security > Identity & Access Management",
      "text": "IAM roles enforce least-privilege access using temporary STS tokens..."
    }
  3. Table & Code Block Preservation: Identify fenced blocks (python ... ) and tables, ensuring they remain intact within a single chunk or split only along logical row lines.

  4. AST Code Parsing: For software repositories, parse source files using Abstract Syntax Trees (e.g. Tree-sitter or Python’s ast module) to keep whole class definitions, function signatures, and docstrings intact.


3. Semantic Similarity Chunking

Detects natural topic shifts by evaluating embedding vector distances across adjacent sentences:

  1. Split text into individual sentences \{s_1, s_2, ..., s_m\}.

  2. Group adjacent sentences into sliding windows (e.g. 3 sentences each) and compute their embeddings: e_i = \text{embed}(s_i).

  3. Compute the cosine distance between consecutive sentence embeddings: d_i = 1 - \cos(e_i, e_{i+1})

  4. Calculate the dynamic distance threshold (e.g. 90th percentile of all differences: \tau = \mu + 1.5\sigma).

  5. Split Boundary: Create a chunk split wherever d_i > \tau.

  6. Adaptive Window Aggregation: Merge small sub-threshold sentences together until they reach a minimum token threshold, guaranteeing that every semantic chunk maintains sufficient information density.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   SEMANTIC CHUNKING DISTANCE CURVE                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Cosine Dist                                                            β”‚
β”‚ 1.0 ─                                                                  β”‚
β”‚ 0.8 ─                 β–² [Split: Topic shift to Pricing]                β”‚
β”‚ 0.6 ─                / \                                               β”‚
β”‚ 0.4 ─   β–²           /   \          ─── Threshold tau (90th percentile) β”‚
β”‚ 0.2 ───/─\─────────/─────\────────                                     β”‚
β”‚ 0.0 └───┴───┴───┴───┴───┴───┴───┴───┴─── Sentence Index (1 to N)       β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

4. Parent-Document (Hierarchical) Retrieval

Diagram: how small chunks are matched but larger passages are supplied to the model

Solves the fundamental RAG tension: Small chunks are better for search matching, but large chunks are better for LLM reasoning!

  1. Index Generation:
    • Split document into large Parent Chunks (e.g. 1,024 tokens).
    • Split each Parent Chunk into 4 to 8 small Child Chunks (e.g. 128 tokens).
    • Link each Child Chunk to its parent_id.
  2. Retrieval Pipeline:
    • Vector search matches against the small 128-token Child Chunks.
    • The engine retrieves the matching child, looks up the corresponding parent_id, and feeds the full 1,024-token Parent Document into the LLM prompt context!
  3. De-duplication Buffer: When multiple child chunks belonging to the same parent document match the top-K query results, the retriever automatically deduplicates parent IDs, preventing redundant context injection.

5. Context Window Sizing Matrix

Recommended chunk sizes across common content types:

Content TypeStrategyChunk SizeOverlapNotes
API DocumentationMarkdown Hierarchy256–512 tokens32 tokensKeep method signature + description intact
Legal ContractsSentence / Paragraph512–1024 tokens128 tokensPrevent clause severance
Customer Support FAQQuestion/Answer Pairs128–256 tokens0 tokens1 Q&A pair per chunk
Source CodeAST / Function Tree300–800 tokens50 tokensSplit along function/class AST nodes

6. Context Window Inflation & Late Chunking

Diagram: chunking before embedding compared with embedding the whole document first

With modern LLMs supporting 1M+ context windows:


7. Token Count Estimation vs. Exact Tokenizer Counting

Never estimate token count using len(text) // 4:


8. Handling Tables and Tabular Data

Diagram: how tabular content is chunked without losing its structure

Tables require specialized handling:


9. Metadata Schema Best Practices

Diagram: the four metadata fields attached to each retrievable chunk

Always attach standard metadata keys to every chunk:


10. Summary: The Foundation of RAG Retrieval Quality


An everyday analogy

Diagram: cutting at fixed intervals compared with cutting at scene boundaries

Think of slicing a movie into short video clips:


Examples in practice

Let us build a Multi-Strategy Chunking Engine:

import re
from typing import List, Dict, Any

class DocumentChunker:
    def __init__(self, chunk_size: int = 200, overlap: int = 40):
        self.chunk_size = chunk_size
        self.overlap = overlap

    def sliding_window_chunk(self, text: str) -> List[Dict[str, Any]]:
        # Splits text into fixed-size word windows with overlap
        words = text.split()
        if not words:
            return []
        
        chunks = []
        stride = max(1, self.chunk_size - self.overlap)
        
        for i in range(0, len(words), stride):
            chunk_words = words[i:i + self.chunk_size]
            chunks.append({
                "chunk_id": len(chunks),
                "text": " ".join(chunk_words),
                "word_count": len(chunk_words),
                "start_idx": i,
                "end_idx": i + len(chunk_words)
            })
            if i + self.chunk_size >= len(words):
                break
        return chunks

    def markdown_header_chunk(self, markdown_text: str) -> List[Dict[str, Any]]:
        # Splits Markdown documents by # and ## headers, preserving breadcrumbs
        lines = markdown_text.split("\n")
        chunks = []
        current_header = "Introduction"
        current_lines = []

        for line in lines:
            if line.startswith("#"):
                if current_lines:
                    text_content = "\n".join(current_lines).strip()
                    if text_content:
                        chunks.append({
                            "chunk_id": len(chunks),
                            "header": current_header,
                            "text": text_content
                        })
                    current_lines = []
                current_header = line.strip("# ").strip()
            else:
                current_lines.append(line)

        if current_lines:
            text_content = "\n".join(current_lines).strip()
            if text_content:
                chunks.append({
                    "chunk_id": len(chunks),
                    "header": current_header,
                    "text": text_content
                })
        return chunks

Implications: security, privacy, performance, scalability, and cost

Diagram: the quality, cost, citation and rework implications of chunking strategy

  1. Context Window Cost: Feeding four 1,024-token chunks into an LLM costs 4,096 input tokens. Using fine-grained 256-token chunks reduces input costs by 75% while improving answer focus.
  2. PII Masking at Ingestion: Apply redaction and token anonymization before chunking to ensure sensitive data is not indexed across chunk boundaries.

Alternatives: free, open source, and commercial

LibraryChunking AlgorithmsPrimary LanguageBest For
LangChain (RecursiveSplitter)Character, Markdown, CodePython / TSGeneral multi-format chunking
LlamaIndex (NodeParsers)Hierarchical, Semantic, SentencePython / TSEnterprise RAG architectures
Unstructured.ioLayout-Aware PDF / VisionPythonComplex multi-column PDFs & OCR
ChonkieLightweight fast chunkingPythonHigh-throughput CPU preprocessing

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   CHUNKING STRATEGY COMPARISON MATRIX                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Strategy           β”‚ Compute Cost     β”‚ Boundary Safety β”‚ Best For     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Fixed Window       β”‚ O(1) Fast        β”‚ Low (Ruptures)  β”‚ Plain text   β”‚
β”‚ Markdown Header    β”‚ O(N) Regex       β”‚ High (Semantic) β”‚ Tech docs    β”‚
β”‚ Semantic Split     β”‚ High (Embeddings)β”‚ Very High       β”‚ Essays/Books β”‚
β”‚ Parent-Child Tree  β”‚ Moderate         β”‚ Maximum Context β”‚ Enterprise   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: choosing a chunking strategy by document type

When to Use Structure-Aware & Hierarchical Chunking:

When NOT to use:


The AI connection

Knowledge check

  1. What is the fundamental difference between context dilution and semantic fragmentation?
  2. Why is sliding token overlap necessary when using fixed-size chunking?
  3. How does Markdown Structure-Aware chunking enrich chunks with breadcrumb metadata?
  4. How does a Parent-Document retriever decouple retrieval precision from LLM generation context?
  5. Why should you use tokenizer token counts instead of character counts when sizing chunks?

Hands-on exercise

In this lab, you will build and test a Multi-Strategy Document Chunker in Python: implement Fixed-Size Sliding Windows with token overlaps, Markdown Header-Aware splitting with breadcrumb metadata injection, and Parent-Child hierarchical segment mapping.

Expected output

[Chunking Strategies Verification Suite]
Testing Sliding Window Chunker:
  Document (500 words) partitioned into 4 overlapping chunks [PASS]
  Overlap verification: 40 shared words confirmed between Chunk 0 and 1 [PASS]
Testing Markdown Header-Aware Splitter:
  Extracted 3 logical sections with breadcrumbs [PASS]
  Headers: ['Overview', 'Architecture', 'Security'] [PASS]
Testing Parent-Child Hierarchy Builder:
  Parent Doc (1024t) mapped to 4 Child Chunks (256t) [PASS]
Test Suite: 4 passed in 0.35s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement Code-Aware Chunking: Build a Python AST chunker that splits Python source files along def function and class boundaries without splitting function bodies.
  2. Build a Semantic Splitter that measures cosine distance drops between consecutive sentences using a local embedding model.

Extension challenge

Implement a Late Chunking Engine:

Quiz

Q1. What is the primary danger of using small fixed-size chunks (e.g. 100 tokens) without overlap in RAG systems?

  1. Text becomes encrypted
  2. The vector database crashes immediately
  3. The embedding dimension increases by 10x
  4. Facts get cut in half across a boundary
Show answer

Answer: D. Facts get cut in half across a boundary

Splitting across sentences severs context and relationship semantics between subject and predicate.

Q2. How does Markdown Structure-Aware Chunking preserve contextual grounding for downstream LLM generation?

  1. Split on heading levels and keep the breadcrumb
  2. By converting Markdown into binary bytecode
  3. By removing all punctuation
  4. By deleting Markdown code blocks
Show answer

Answer: A. Split on heading levels and keep the breadcrumb

Preserving heading breadcrumbs (e.g. Doc > Section > Subsection) grounds the LLM in the exact topic domain.

Q3. How does Semantic Chunking determine where to create a split in a document?

  1. By splitting after every 25th word
  2. Split where consecutive-sentence similarity drops
  3. By counting vowel frequency
  4. By using regular expression character counts alone
Show answer

Answer: B. Split where consecutive-sentence similarity drops

Semantic chunking identifies topical transition boundaries where vector similarity drops significantly.

Q4. What architectural problem does the Parent-Document (Hierarchical) Retriever solve?

  1. It generates audio files from text
  2. It compresses hard drives
  3. Small chunks match well; big parents give context
  4. It replaces the vector database entirely
Show answer

Answer: C. Small chunks match well; big parents give context

Small chunks yield high retrieval precision, while large parent docs eliminate missing-context hallucinations in the LLM.

Q5. Why is chunk size measured in tokenizer tokens (e.g. tiktoken/cl100k_base) rather than raw character or word counts?

  1. Because word counts are illegal in production
  2. Because characters do not exist in Python
  3. Because tokenizers use less memory
  4. Models cap tokens, and characters vary by language
Show answer

Answer: D. Models cap tokens, and characters vary by language

Embedding context windows and transformer attention compute strictly according to token counts.

Glossary

Chunking
The process of partitioning large documents into smaller, cohesive text segments suitable for embedding and retrieval.
Sliding Window Overlap
A technique where adjacent chunks share N trailing/leading tokens to preserve contextual continuity across boundaries.
Structure-Aware Chunking
Splitting documents based on intrinsic syntax markers such as Markdown headers, HTML tags, or Python class definitions.
Semantic Chunking
An adaptive splitting algorithm that detects topic transitions by measuring cosine similarity drops between sentences.
Parent-Document Retrieval
A two-stage pattern indexing small child chunks for vector search, but passing the larger parent document to the LLM.
Tokenization
Converting text strings into sub-word integer tokens using BPE algorithms (e.g. tiktoken, cl100k_base).
Context Dilution
When a chunk is too large, causing irrelevant information to dilute the core semantic signal in the embedding vector.
Semantic Fragmentation
When a chunk is too small, splitting an atomic fact across boundaries and rendering both fragments unretrievable.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.