LLMs and Generative AI βΊ Embeddings and Vector Search βΊ Day 263
Day 263: Chunking Strategies
Master document segmentation and text chunking strategies for production RAG pipelines: evaluate Fixed-Size Chunking with sliding token overlap, Markdown/Code Structure-Aware Boundary Parsing, Semantic Similarity Chunking via cosine thresholding, and Parent-Document / Hierarchical Chunking; analyze context window trade-offs, retrieval fragmentation, boundary loss, and implement a production multi-strategy chunker in Python.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-263-chunking-strategies
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/llms-and-generative-ai/day-263-chunking-strategies - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Differentiate between Fixed-Size, Markdown Structure-Aware, Semantic, and Hierarchical Chunking.
- Calculate token overlaps to prevent critical semantic boundary severance.
- Implement a Markdown heading-aware document segmenter that preserves section hierarchies in metadata.
- Implement a Semantic Splitter calculating cosine similarity drops between consecutive sentences.
- Design a Parent-Document retriever storing small child chunks for search and large parent contexts for LLM generation.
Prerequisites
Why this matters
The single most consequential preprocessing step in any Retrieval-Augmented Generation (RAG) pipeline is Document Chunking.
Even the most powerful embedding model (such as text-embedding-3-large) and the most sophisticated LLM (such as Claude 3.7 Sonnet) will fail if your chunking strategy is flawed:
- If your chunks are too large (e.g. 2,000 tokens), distinct facts blend together, causing context dilution where the embedding vector averages out into a generic, unhelpful representation.
- If your chunks are too small (e.g. 50 tokens), you suffer from semantic fragmentation, slicing critical sentences in half and stripping away essential subject pronouns and qualifying clauses.
- If you split across arbitrary character counts without respect for Markdown tables or code blocks, your LLM receives corrupted syntax and hallucinated schema relationships.
Document chunking is the art and science of optimizing semantic density: ensuring every retrieved text unit contains exactly enough information to answer a user inquiry with maximum precision and zero surrounding noise.
Today, you will master Fixed-Size Sliding Windows, Markdown Structure-Aware Boundary Parsing, Semantic Similarity Chunking, and Parent-Child Hierarchical Retrieval, and build a production-grade multi-strategy document segmenter in Python.
The idea in plain language
Imagine you are slicing a long, 500-page historical encyclopedia to create study flashcards:
- Naive Character Split (Scissors at every 200 words): You blindly cut every page every 200 words. Card #12 ends with: βOn July 4th, 1776, Thomas Jefferson drafted theβ¦β and Card #13 starts with: ββ¦declaration in Philadelphia.β Both cards are broken and confusing (Boundary Rupture).
- Structure-Aware Split (Cutting by Chapters & Headings): You cut only at logical section headers (e.g.
## The Revolutionary War > ### Draft Process), keeping paragraphs intact (Cohesive Semantic Blocks). - Parent-Child Flashcards: You write a tiny 2-sentence summary on the front of the card for quick searching, but tape the entire 3-page historical chapter to the back for reading (Hierarchical Context Preservation).
Historical background
- 1960sβ1980s (Text Passage Retrieval): Early information retrieval systems (SMART, Okapi BM25) experimented with fixed-paragraph vs. sliding-window text boundaries.
- 2020 (Dense Passage Retrieval - DPR): Karpukhin et al. standardized the 100-word fixed passage convention for training dense dual-encoders on Wikipedia.
- 2023 (Lost in the Middle): Liu et al. revealed that LLMs struggle to recall facts buried inside overly large context chunks, prompting the industry to shift toward smaller, high-precision retrieval units.
- 2024 (Semantic & Hierarchical Chunking): Modern RAG frameworks (LlamaIndex, LangChain) introduced embedding-based semantic boundary detection and Parent-Document retrieval architectures.
What it is β and what it is not
What Document Chunking IS:
- Intelligent Semantic Segmentation: Dividing unstructured text into discrete, semantically cohesive passages while injecting contextual metadata (breadcrumbs, timestamps, titles) into each chunk.
What it is NOT:
- Not Just String Slicing (
text[:500]): Arbitrary character slicing cuts multibyte UTF-8 characters and tears sentences midway through critical assertions.
Why it was created and what problems it solves
Production chunking solves 4 critical failure modes:
- Embedding Context Window Overflow: Prevents documents from exceeding model token limits (e.g. 512 or 8,192 tokens).
- Boundary Semantic Loss: Sliding token overlaps guarantee that facts located at chunk transitions are never lost.
- Structured Document Integrity: Prevents Markdown tables, JSON payloads, and Python code functions from being split across chunk borders.
- Optimal Retrieval Precision: Aligns the granularity of the retrieved passage with the granularity of the user query.
How it works
Let us analyze the 4 foundational chunking strategies, their trade-offs, and mathematical boundary conditions.
1. Fixed-Size Chunking with Sliding Token Overlap
The simplest and most widely used baseline across early RAG architectures:
-
Parameters:
chunk_size = 512tokens,chunk_overlap = 64tokens (typically 10% to 15% of chunk size). -
Stride Calculation:
stride = chunk_size - chunk_overlap = 448tokens. -
Mathematical Formulation:
\text{Chunk}_k = T[k \cdot \text{stride} : k \cdot \text{stride} + \text{chunk\_size}] -
Pros: Deterministic computational performance,
O(1)partitioning speed, and zero machine learning inference overhead during ingestion. -
Cons: Arbitrarily severs sentences, paragraphs, and list items across chunk boundaries, leading to contextual rupture unless compensated with adequate token overlap.
2. Markdown / Code Structure-Aware Chunking
Structure-aware segmentation parses document syntax trees rather than treating text as a flat string:
-
Header Hierarchy Splitting: Segment by
# H1,## H2, and### H3markers. -
Breadcrumb Metadata Enrichment: Inject the ancestral heading hierarchy into the chunk metadata payload so that the embedding model understands document context:
{ "document": "cloud_architecture.md", "breadcrumb": "Architecture > Security > Identity & Access Management", "text": "IAM roles enforce least-privilege access using temporary STS tokens..." } -
Table & Code Block Preservation: Identify fenced blocks (
python ...) and tables, ensuring they remain intact within a single chunk or split only along logical row lines. -
AST Code Parsing: For software repositories, parse source files using Abstract Syntax Trees (e.g. Tree-sitter or Pythonβs
astmodule) to keep whole class definitions, function signatures, and docstrings intact.
3. Semantic Similarity Chunking
Detects natural topic shifts by evaluating embedding vector distances across adjacent sentences:
-
Split text into individual sentences
\{s_1, s_2, ..., s_m\}. -
Group adjacent sentences into sliding windows (e.g. 3 sentences each) and compute their embeddings:
e_i = \text{embed}(s_i). -
Compute the cosine distance between consecutive sentence embeddings:
d_i = 1 - \cos(e_i, e_{i+1}) -
Calculate the dynamic distance threshold (e.g. 90th percentile of all differences:
\tau = \mu + 1.5\sigma). -
Split Boundary: Create a chunk split wherever
d_i > \tau. -
Adaptive Window Aggregation: Merge small sub-threshold sentences together until they reach a minimum token threshold, guaranteeing that every semantic chunk maintains sufficient information density.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SEMANTIC CHUNKING DISTANCE CURVE β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Cosine Dist β
β 1.0 β€ β
β 0.8 β€ β² [Split: Topic shift to Pricing] β
β 0.6 β€ / \ β
β 0.4 β€ β² / \ βββ Threshold tau (90th percentile) β
β 0.2 β€ββ/β\βββββββββ/βββββ\ββββββββ β
β 0.0 βββββ΄ββββ΄ββββ΄ββββ΄ββββ΄ββββ΄ββββ΄ββββ΄βββ Sentence Index (1 to N) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
4. Parent-Document (Hierarchical) Retrieval
Solves the fundamental RAG tension: Small chunks are better for search matching, but large chunks are better for LLM reasoning!
- Index Generation:
- Split document into large Parent Chunks (e.g. 1,024 tokens).
- Split each Parent Chunk into 4 to 8 small Child Chunks (e.g. 128 tokens).
- Link each Child Chunk to its
parent_id.
- Retrieval Pipeline:
- Vector search matches against the small 128-token Child Chunks.
- The engine retrieves the matching child, looks up the corresponding
parent_id, and feeds the full 1,024-token Parent Document into the LLM prompt context!
- De-duplication Buffer: When multiple child chunks belonging to the same parent document match the top-K query results, the retriever automatically deduplicates parent IDs, preventing redundant context injection.
5. Context Window Sizing Matrix
Recommended chunk sizes across common content types:
| Content Type | Strategy | Chunk Size | Overlap | Notes |
|---|---|---|---|---|
| API Documentation | Markdown Hierarchy | 256β512 tokens | 32 tokens | Keep method signature + description intact |
| Legal Contracts | Sentence / Paragraph | 512β1024 tokens | 128 tokens | Prevent clause severance |
| Customer Support FAQ | Question/Answer Pairs | 128β256 tokens | 0 tokens | 1 Q&A pair per chunk |
| Source Code | AST / Function Tree | 300β800 tokens | 50 tokens | Split along function/class AST nodes |
- Tuning chunk sizing parameters according to content taxonomy prevents domain-specific boundary distortion and preserves relational context.
6. Context Window Inflation & Late Chunking
With modern LLMs supporting 1M+ context windows:
- Late Chunking: Passing the entire 8,000-word document through a long-context embedding model (e.g.
jina-embeddings-v3), extracting token embeddings, and mean-pooling chunk slices after full-document cross-attention! - This gives each chunk awareness of the entire surrounding document before vectorization.
- Late chunking eliminates the loss of global document context while retaining the high retrieval precision of small chunk units.
7. Token Count Estimation vs. Exact Tokenizer Counting
Never estimate token count using len(text) // 4:
- Code snippets and multilingual text (Japanese, Arabic, German) exhibit token-to-character ratios ranging from 1:1 to 1:1.5.
- Always use the exact tokenizer corresponding to your embedding model (e.g.
tiktoken.get_encoding("cl100k_base")). - Exact tokenization ensures chunks fit within strict embedding model context windows without silent runtime truncation.
8. Handling Tables and Tabular Data
Tables require specialized handling:
-
Slicing a Markdown or HTML table midway destroys header associations and corrupts structural interpretation.
-
Convert table rows into self-contained natural language sentences:
"In year 2024, Q3 Revenue was $42M with a 15% YoY growth rate." -
This ensures high semantic vector alignment with natural user queries while preserving numerical accuracy.
9. Metadata Schema Best Practices
Always attach standard metadata keys to every chunk:
doc_id,chunk_id,chunk_index,total_chunks,source_url,heading_path,char_start,char_end.- Enriching metadata with timestamps and access control tags enables sub-second relational pre-filtering during vector queries.
10. Summary: The Foundation of RAG Retrieval Quality
- Document chunking transforms unstructured text corpora into modular, semantically rich units optimized for both dense vector retrieval and generative language model reasoning.
- Balancing chunk size, boundary detection, and parent-child hierarchies prevents both context dilution and semantic fragmentation.
- Modern RAG pipelines frequently combine multiple chunking strategies across heterogeneous document formats to achieve maximum retrieval recall and zero generation hallucination.
- In summary, choosing and implementing the optimal chunking strategy directly determines the quality, coherence, and accuracy of the context delivered to your LLM generator.
An everyday analogy
Think of slicing a movie into short video clips:
- Fixed Time Split (Every 30 seconds): A clip cuts off in the middle of a character saying: βThe treasure is hidden under theβ¦β and the next clip starts with: ββ¦giant rock.β Neither clip makes sense alone (Arbitrary Split).
- Scene-Aware Split (Semantic Split): You cut the film only when a scene changes from the pirate ship to the island beach (Natural Cohesive Unit).
- Trailer + Full Movie (Parent-Child): You search 10-second trailer clips to find the movie, but play the full 2-hour movie when you sit down to watch (Hierarchical Retrieval).
Examples in practice
Let us build a Multi-Strategy Chunking Engine:
import re
from typing import List, Dict, Any
class DocumentChunker:
def __init__(self, chunk_size: int = 200, overlap: int = 40):
self.chunk_size = chunk_size
self.overlap = overlap
def sliding_window_chunk(self, text: str) -> List[Dict[str, Any]]:
# Splits text into fixed-size word windows with overlap
words = text.split()
if not words:
return []
chunks = []
stride = max(1, self.chunk_size - self.overlap)
for i in range(0, len(words), stride):
chunk_words = words[i:i + self.chunk_size]
chunks.append({
"chunk_id": len(chunks),
"text": " ".join(chunk_words),
"word_count": len(chunk_words),
"start_idx": i,
"end_idx": i + len(chunk_words)
})
if i + self.chunk_size >= len(words):
break
return chunks
def markdown_header_chunk(self, markdown_text: str) -> List[Dict[str, Any]]:
# Splits Markdown documents by # and ## headers, preserving breadcrumbs
lines = markdown_text.split("\n")
chunks = []
current_header = "Introduction"
current_lines = []
for line in lines:
if line.startswith("#"):
if current_lines:
text_content = "\n".join(current_lines).strip()
if text_content:
chunks.append({
"chunk_id": len(chunks),
"header": current_header,
"text": text_content
})
current_lines = []
current_header = line.strip("# ").strip()
else:
current_lines.append(line)
if current_lines:
text_content = "\n".join(current_lines).strip()
if text_content:
chunks.append({
"chunk_id": len(chunks),
"header": current_header,
"text": text_content
})
return chunks
Implications: security, privacy, performance, scalability, and cost
- Context Window Cost: Feeding four 1,024-token chunks into an LLM costs 4,096 input tokens. Using fine-grained 256-token chunks reduces input costs by 75% while improving answer focus.
- PII Masking at Ingestion: Apply redaction and token anonymization before chunking to ensure sensitive data is not indexed across chunk boundaries.
Alternatives: free, open source, and commercial
| Library | Chunking Algorithms | Primary Language | Best For |
|---|---|---|---|
| LangChain (RecursiveSplitter) | Character, Markdown, Code | Python / TS | General multi-format chunking |
| LlamaIndex (NodeParsers) | Hierarchical, Semantic, Sentence | Python / TS | Enterprise RAG architectures |
| Unstructured.io | Layout-Aware PDF / Vision | Python | Complex multi-column PDFs & OCR |
| Chonkie | Lightweight fast chunking | Python | High-throughput CPU preprocessing |
Comparison with related concepts
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CHUNKING STRATEGY COMPARISON MATRIX β
ββββββββββββββββββββββ¬βββββββββββββββββββ¬ββββββββββββββββββ¬βββββββββββββββ€
β Strategy β Compute Cost β Boundary Safety β Best For β
ββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββΌβββββββββββββββ€
β Fixed Window β O(1) Fast β Low (Ruptures) β Plain text β
β Markdown Header β O(N) Regex β High (Semantic) β Tech docs β
β Semantic Split β High (Embeddings)β Very High β Essays/Books β
β Parent-Child Tree β Moderate β Maximum Context β Enterprise β
ββββββββββββββββββββββ΄βββββββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββ
When to use it β and when not to
When to Use Structure-Aware & Hierarchical Chunking:
- Documentation, technical knowledge bases, codebases, legal documents, and complex enterprise RAG pipelines.
When NOT to use:
- Micro-queries, short chat messages, or single-sentence FAQ databases where documents are already atomic units.
The AI connection
- Chunking decides retrieval quality more than the embedding model does, and it is the cheapest thing to get right.
- A chunk is the unit of retrieval, so it has to be simultaneously self-contained and specific β which pulls in opposite directions.
- Structure-aware splitting beats fixed-size splitting almost everywhere, because documents already carry the boundaries you want.
- Retrieving small and supplying large resolves the tension: match on a precise chunk, give the model its surrounding context.
- Tables and code need their own handling, since splitting either one mid-structure destroys the meaning entirely.
Knowledge check
- What is the fundamental difference between context dilution and semantic fragmentation?
- Why is sliding token overlap necessary when using fixed-size chunking?
- How does Markdown Structure-Aware chunking enrich chunks with breadcrumb metadata?
- How does a Parent-Document retriever decouple retrieval precision from LLM generation context?
- Why should you use tokenizer token counts instead of character counts when sizing chunks?
Hands-on exercise
In this lab, you will build and test a Multi-Strategy Document Chunker in Python: implement Fixed-Size Sliding Windows with token overlaps, Markdown Header-Aware splitting with breadcrumb metadata injection, and Parent-Child hierarchical segment mapping.
Expected output
[Chunking Strategies Verification Suite]
Testing Sliding Window Chunker:
Document (500 words) partitioned into 4 overlapping chunks [PASS]
Overlap verification: 40 shared words confirmed between Chunk 0 and 1 [PASS]
Testing Markdown Header-Aware Splitter:
Extracted 3 logical sections with breadcrumbs [PASS]
Headers: ['Overview', 'Architecture', 'Security'] [PASS]
Testing Parent-Child Hierarchy Builder:
Parent Doc (1024t) mapped to 4 Child Chunks (256t) [PASS]
Test Suite: 4 passed in 0.35s
Validate your work
Run the automated test runner:
./tests/run_tests.sh
Troubleshooting
- Ensure empty lines between Markdown headings do not generate empty string chunks.
Common mistakes
- Setting overlap equal to or greater than chunk size: This causes an infinite loop during sliding window iteration.
Practice assignment
- Implement Code-Aware Chunking: Build a Python AST chunker that splits Python source files along
deffunction andclassboundaries without splitting function bodies. - Build a Semantic Splitter that measures cosine distance drops between consecutive sentences using a local embedding model.
Extension challenge
Implement a Late Chunking Engine:
- Pass a 2,000-word document through a transformer model to compute token embeddings with full cross-attention, extract slice boundaries for 256-token chunks, compute mean-pooled chunk vectors, and compare retrieval accuracy against independently embedded chunks.
Quiz
Q1. What is the primary danger of using small fixed-size chunks (e.g. 100 tokens) without overlap in RAG systems?
- Text becomes encrypted
- The vector database crashes immediately
- The embedding dimension increases by 10x
- Facts get cut in half across a boundary
Show answer
Answer: D. Facts get cut in half across a boundary
Splitting across sentences severs context and relationship semantics between subject and predicate.
Q2. How does Markdown Structure-Aware Chunking preserve contextual grounding for downstream LLM generation?
- Split on heading levels and keep the breadcrumb
- By converting Markdown into binary bytecode
- By removing all punctuation
- By deleting Markdown code blocks
Show answer
Answer: A. Split on heading levels and keep the breadcrumb
Preserving heading breadcrumbs (e.g. Doc > Section > Subsection) grounds the LLM in the exact topic domain.
Q3. How does Semantic Chunking determine where to create a split in a document?
- By splitting after every 25th word
- Split where consecutive-sentence similarity drops
- By counting vowel frequency
- By using regular expression character counts alone
Show answer
Answer: B. Split where consecutive-sentence similarity drops
Semantic chunking identifies topical transition boundaries where vector similarity drops significantly.
Q4. What architectural problem does the Parent-Document (Hierarchical) Retriever solve?
- It generates audio files from text
- It compresses hard drives
- Small chunks match well; big parents give context
- It replaces the vector database entirely
Show answer
Answer: C. Small chunks match well; big parents give context
Small chunks yield high retrieval precision, while large parent docs eliminate missing-context hallucinations in the LLM.
Q5. Why is chunk size measured in tokenizer tokens (e.g. tiktoken/cl100k_base) rather than raw character or word counts?
- Because word counts are illegal in production
- Because characters do not exist in Python
- Because tokenizers use less memory
- Models cap tokens, and characters vary by language
Show answer
Answer: D. Models cap tokens, and characters vary by language
Embedding context windows and transformer attention compute strictly according to token counts.
Glossary
- Chunking
- The process of partitioning large documents into smaller, cohesive text segments suitable for embedding and retrieval.
- Sliding Window Overlap
- A technique where adjacent chunks share N trailing/leading tokens to preserve contextual continuity across boundaries.
- Structure-Aware Chunking
- Splitting documents based on intrinsic syntax markers such as Markdown headers, HTML tags, or Python class definitions.
- Semantic Chunking
- An adaptive splitting algorithm that detects topic transitions by measuring cosine similarity drops between sentences.
- Parent-Document Retrieval
- A two-stage pattern indexing small child chunks for vector search, but passing the larger parent document to the LLM.
- Tokenization
- Converting text strings into sub-word integer tokens using BPE algorithms (e.g. tiktoken, cl100k_base).
- Context Dilution
- When a chunk is too large, causing irrelevant information to dilute the core semantic signal in the embedding vector.
- Semantic Fragmentation
- When a chunk is too small, splitting an atomic fact across boundaries and rendering both fragments unretrievable.
Sources and further reading
- Dense Passage Retrieval for Open-Domain Question Answering β EMNLP 2020 (accessed 2026-08-30)
- Lost in the Middle: How Language Models Use Long Contexts β Transactions of the ACL (accessed 2026-08-30)
- LlamaIndex Advanced Chunking and Document Splitting Strategies β LlamaIndex Documentation (accessed 2026-08-30)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.