LLMs and Generative AI › Retrieval-Augmented Generation › Day 268
Day 268: A Minimal RAG System from Scratch
Construct a complete, production-clean RAG pipeline in pure Python without external database dependencies: implement document chunking, in-memory dense vector cosine scoring, sparse BM25 inverted indexing, prompt context assembly, and hallucination containment.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-268-a-minimal-rag-system-from-scratch
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
cd labs/sections/llms-and-generative-ai/day-268-a-minimal-rag-system-from-scratch - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Understand the end-to-end multi-stage RAG architecture and non-parametric memory bridging.
- Implement structure-aware document parsing and sliding window chunking.
- Build sparse BM25 and dense vector indices with Reciprocal Rank Fusion.
- Construct grounded system prompts that enforce verifiable inline bracket citations.
- Evaluate RAG pipelines quantitatively using the RAG Triad framework.
Prerequisites
Why this matters
You have used retrieval for four days without ever seeing it work. Yesterday’s lesson gave you the architecture; today you build the whole thing in about a hundred lines of pure Python, with no libraries at all.
That is not a stunt. You cannot debug what you have only ever imported. When a production pipeline returns a plausible answer sourced from the wrong document, the question is always the same — did retrieval fail, did fusion fail, or did the prompt fail? Someone who has written those three stages can answer in minutes. Someone who has only configured them is left restarting the vector database and hoping.
The whole apparatus reduces to arithmetic you already know:
chunk split text on word boundaries with an overlap
index count terms (BM25) and store vectors (embeddings)
retrieve score every chunk, keep the top k
fuse combine two rankings by position, not by score
assemble paste the survivors into a prompt with numbered labels
There is no step in that list that requires a framework. There is no step that cannot be tested. And once you have written it, every later decision — which vector store, which reranker, which chunk size — becomes a comparison against something you understand rather than a guess between two brand names.
The idea in plain language
A minimal RAG system is four small functions in a row, and the only interesting mathematics is the dot product.
Ask a librarian for “the paper about attention in translation”. They do two things at once without noticing. They look for those literal words on spines and in indexes — that is lexical search, and it is what BM25 does. They also think about what you meant, and fetch a paper titled “Neural Machine Translation by Jointly Learning to Align and Translate” that contains none of your words — that is semantic search, and it is what an embedding does.
Neither is sufficient. Lexical search fails on synonyms; semantic search fails on exact identifiers, error codes and part numbers. So you run both and merge the two lists. Then you hand the survivors to someone who can write, with the instruction: use only these pages, and cite them.
That is the entire system. Today you write it.
Historical background
The two halves of retrieval were invented decades apart and have only recently been used together.
-
1972 — Karen Spärck Jones publishes inverse document frequency: a term that appears everywhere tells you nothing, so weight it down. Every lexical search engine since rests on that one idea.
-
1994 — Robertson and Walker formalise BM25 at TREC-3. It is still a strong baseline in 2026, which is remarkable for a thirty-year-old ranking function with two tunable constants.
-
2013 — word2vec makes dense vectors cheap, and “similar meaning” becomes “small angle between two arrows”.
-
2019 — Sentence-BERT produces embeddings where cosine similarity between sentences is meaningful, not just between words.
-
2020 — Lewis et al. name Retrieval-Augmented Generation and Karpukhin et al. publish Dense Passage Retrieval, showing dense retrieval beating BM25 on open-domain question answering.
-
2009 — out of sequence, and worth it: Cormack et al. describe Reciprocal Rank Fusion, a method for merging rankings that needs no score calibration at all. It was designed for meta-search engines. It turned out to be exactly what hybrid retrieval needed a decade later.
The lesson of that timeline is that nothing here is new or proprietary. The pieces are old, well-documented, and small enough to write yourself.
What it is — and what it is not
It is a complete, honest, end-to-end RAG pipeline. Every stage a production system has, this has: chunking, lexical indexing, dense indexing, fusion, prompt assembly. The results are real results, and the failures are real failures.
It is not production infrastructure, and pretending otherwise would teach you the wrong lesson. Specifically it lacks:
| Missing | Why it does not matter today | When it starts to |
|---|---|---|
| Persistence | The index rebuilds in under a second | A corpus that takes minutes to embed |
| Approximate nearest neighbour | An exhaustive scan of 10,000 vectors is fast | Roughly 100,000 chunks and up |
| Concurrency | One query at a time is fine while learning | More than one user |
| A real embedding model | A hashed bag-of-words shows the mechanics | Any question about actual quality |
The last row deserves emphasis. Today’s “embeddings” are deterministic hashes, not learned representations. They demonstrate the plumbing of dense retrieval faithfully while telling you nothing about its quality. That distinction — between a mechanism that runs and a mechanism that works — is the one this course keeps returning to.
Why it was created and what problems it solves
Frameworks hide the seams, and the seams are where the bugs live.
Ask a working RAG team what actually went wrong in their last incident and you will rarely hear “the vector database was down”. You will hear things like these, every one of which lives in a seam between two stages:
-
The chunker split a table across two chunks, so no chunk contained both the column heading and the number underneath it.
-
The retriever returned excellent chunks and the prompt template truncated them at 2,000 characters, silently, from the end.
-
Two documents were near-duplicates, so the top three results were three copies of the same passage and the answer had one source wearing three hats.
-
The embedding of the query was computed with a different model version from the embeddings in the index, and every score was quietly meaningless.
None of those is visible from a configuration file. All of them are obvious to someone who has written the four functions and knows what each hands to the next. Building it once is the cheapest insurance available against a whole category of bug.
How it works
1. Chunking on word boundaries, with overlap
Text is split into windows of chunk_size words that overlap by overlap
words. The overlap exists because a fact can straddle a boundary: a sentence
ending “…the write cache costs” followed by “$3.75 per million tokens” is
useless if the two halves land in different chunks and only one is retrieved.
The loop must terminate on i + chunk_size >= len(words), or the final partial
window is emitted twice — a bug the lab’s tests catch deliberately.
2. BM25, in about fifteen lines
For a query term q and document d, BM25 scores
IDF(q) · ( f(q,d) · (k1 + 1) ) / ( f(q,d) + k1 · (1 - b + b · |d|/avgdl) )
f(q,d) is the term’s frequency in the document. k1 (typically 1.5) saturates
that frequency, so the twentieth occurrence of a word adds far less than the
second. b (typically 0.75) normalises for document length, so a long document
does not win simply by containing more words. IDF down-weights terms that
appear everywhere.
You do not need to memorise the formula. You do need to know that it produces unbounded positive scores — which is why the next stage cannot simply add it to a cosine similarity.
3. Dense retrieval as a dot product
Every chunk becomes a vector. Every query becomes a vector. Relevance is the cosine of the angle between them, and for unit-normalised vectors cosine similarity is the dot product — one multiply-add per dimension, nothing more.
Scores land in [-1, 1]. BM25’s do not. That mismatch is the whole reason the
next stage exists.
4. Reciprocal Rank Fusion
RRF throws the scores away and keeps only the positions:
RRF(d) = Σ 1 / (k + rank_i(d)) k = 60 by convention
A document ranked first in either list contributes 1/61. Ranked tenth,
1/70. Because only ranks enter the sum, BM25’s unbounded scale and cosine’s
bounded one never have to be reconciled — there is nothing to normalise and no
weight to tune. A document that both methods rank highly rises above one that
either ranks first alone, which is exactly the behaviour you want.
5. Prompt assembly
The survivors are numbered and pasted into the context with an instruction to answer only from them and to cite by number. This stage looks trivial and is where a surprising share of production failures happen: silent truncation, missing separators between chunks, and citation labels that do not match the order the chunks were pasted in.
An everyday analogy
Building this from scratch is assembling a bicycle from a box of parts before buying a motorbike.
Nobody commutes on the bicycle. But afterwards you know what a bottom bracket is, you know that the brakes rub because the wheel is not seated straight, and when the motorbike develops a noise you have a mental model of what is connected to what. The person who has only ever ridden has to hand it to a mechanic and believe whatever comes back.
The bicycle takes an afternoon. The understanding lasts.
Examples in practice
The four stages, at the size they actually are.
Chunking
def chunk(text: str, size: int = 120, overlap: int = 20) -> list[str]:
words, out, i = text.split(), [], 0
while i < len(words):
out.append(" ".join(words[i : i + size]))
if i + size >= len(words): # last window; stop or it repeats
break
i += size - overlap
return out
Lexical scoring
def bm25(query: str, docs: list[list[str]], k1=1.5, b=0.75) -> list[float]:
N, avgdl = len(docs), sum(len(d) for d in docs) / len(docs)
terms = re.findall(r"\w+", query.lower()) # lowercase, or nothing matches
scores = [0.0] * N
for t in terms:
n_t = sum(1 for d in docs if t in d)
if not n_t:
continue
idf = math.log(1 + (N - n_t + 0.5) / (n_t + 0.5))
for i, d in enumerate(docs):
f = d.count(t)
scores[i] += idf * (f * (k1 + 1)) / (f + k1 * (1 - b + b * len(d) / avgdl))
return scores
Fusion
def rrf(*rankings: list[int], k: int = 60) -> list[int]:
score: dict[int, float] = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
score[doc_id] = score.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(score, key=score.get, reverse=True)
Prompt assembly
def build_prompt(question: str, chunks: list[str]) -> str:
context = "\n\n".join(f"[{i}] {c}" for i, c in enumerate(chunks, start=1))
return (
"Answer using ONLY the context below. Cite sources as [1], [2].\n"
"If the context does not contain the answer, say so.\n\n"
f"{context}\n\nQuestion: {question}"
)
That is the system. Roughly sixty lines, four tests, no dependencies.
Implications: security, privacy, performance, scalability, and cost
Performance. Scoring is a linear scan: every query touches every chunk. At 10,000 chunks of 120 words that is milliseconds and entirely acceptable. The scan is what an approximate index later replaces, and Day 326 measures exactly what that trade buys and costs.
Scalability. Memory holds the whole corpus twice — once as text, once as vectors. A float32 vector of 384 dimensions is 1.5 KB, so 100,000 chunks is about 150 MB of vectors before the text. That is the point at which “keep it in a Python list” stops being a reasonable answer.
Security. Retrieval is an access-control boundary, and an in-memory list has none. Anything in the index is reachable by any query. In production the filter must be applied before chunks enter the context, not after — a model that has seen a passage cannot be asked to forget it.
Privacy. Chunks are copies. A document deleted at source remains in the index, and in the embeddings, until something removes it — which is why Day 325 treats freshness as a correctness property rather than a nicety.
Cost. Zero today, and that is itself instructive: every cost in a production RAG system comes from the embedding model, the vector store, or the generation call. The retrieval logic is free.
Alternatives: free, open source, and commercial
| Option | Licence | What it gives you | What it costs |
|---|---|---|---|
| Writing it yourself | — | Complete understanding; no dependency risk | You maintain it |
| LangChain | MIT, free | Chains, loaders, and a large integration surface | Abstraction depth; upgrades can move behaviour |
| LlamaIndex | MIT, free | Indexing and query engines aimed squarely at RAG | Opinionated structure |
| Haystack | Apache-2.0, free | Explicit, typed pipeline graphs | More ceremony for small jobs |
rank_bm25 | Apache-2.0, free | Just the lexical half, well tested | Nothing else |
The honest recommendation is neither “always build” nor “always import”. Build it once, today, so you can read a framework’s source and recognise what it is doing. Then import, because a maintained BM25 implementation has fixed edge cases yours has not met yet.
Comparison with related concepts
| Approach | Where knowledge lives | Update cost | Attribution |
|---|---|---|---|
| This pipeline | An external index | Re-index one document | Exact chunk, by number |
| Fine-tuning | Model weights | A full training run | None |
| Long-context stuffing | The prompt itself | Free | Whole document only |
| Plain keyword search | An inverted index | Re-index | Exact, but no synthesis |
Against fine-tuning: retrieval changes what the model knows; fine-tuning changes how it behaves. Day 274 draws that line properly.
Against long-context stuffing: if the corpus fits in the context window, stuffing is simpler and often better. Retrieval earns its complexity when the corpus does not fit, or when you need to say which passage an answer came from.
When to use it — and when not to
Use a hand-written pipeline when you are learning the mechanism, the corpus is small enough to scan exhaustively, you need zero dependencies, or you are building a reference implementation to test a framework against.
Reach for a library when the corpus outgrows memory, you need persistence across restarts, several people query at once, or you need real embeddings from a maintained model.
Do not use retrieval at all when the answer is already in the model, the corpus fits comfortably in the context window, or the question is about reasoning rather than facts. Retrieval adds two failure modes — retrieving the wrong thing, and retrieving nothing — and they are only worth accepting when grounding is genuinely required.
The AI connection
- Building the pipeline yourself is what makes the frameworks legible — every abstraction they offer maps onto something you wrote by hand.
- The core of retrieval is a few hundred lines, which is worth knowing before adopting a system that hides it.
- Debugging a framework you do not understand is far harder than debugging code you wrote, and retrieval fails in subtle ways.
- Every production system is this plus operations — the same stages, with scale, monitoring and permissions around them.
- Knowing the pieces lets you replace one at a time, which is how a system improves without being rewritten.
Knowledge check
quiz:
question: "Why does Reciprocal Rank Fusion combine BM25 and dense results by position rather than by score?"
options:
- "RRF is faster to compute than adding two floating-point numbers together"
- "BM25 scores are unbounded and cosine scores are not, so there is nothing to normalise"
- "RRF retrains the embedding model on each query it receives"
- "Rank positions use less memory than the scores they were derived from"
answer_index: 1
explanation: "BM25 produces unbounded positive scores while cosine similarity is bounded to [-1, 1]. RRF sums 1/(k + rank), so the two scales never have to be reconciled and there is no fusion weight to tune."
Hands-on exercise
In this exercise, you will build and evaluate a modular Python RAG pipeline that ingests a sample technical knowledge base, performs hybrid lexical-dense search, constructs a grounded system prompt, and validates answer grounding.
Expected output
When running your test suite, you should observe:
============================= test session starts ==============================
collected 4 items
tests/test_minimal_rag_lib.py::test_document_chunker PASSED [ 25%]
tests/test_minimal_rag_lib.py::test_sparse_bm25 PASSED [ 50%]
tests/test_minimal_rag_lib.py::test_hybrid_ranker PASSED [ 75%]
tests/test_minimal_rag_lib.py::test_prompt_synthesizer PASSED [100%]
============================== 4 passed in 0.04s ===============================
Validate your work
Run the pytest test runner script:
pytest tests/ -v
Troubleshooting
- Issue: BM25 returns zero relevance scores for exact keyword searches.
- Fix: Ensure input query terms are tokenized and lowercased using
re.findall(r'\w+', query.lower()).
- Fix: Ensure input query terms are tokenized and lowercased using
- Issue: Document chunker creates duplicate final chunks.
- Fix: Ensure sliding window loop terminates when
i + chunk_size >= len(words).
- Fix: Ensure sliding window loop terminates when
Common mistakes
- Naive String Slicing: Slicing raw strings every 1000 characters cuts words in half. Always chunk across token or whitespace boundaries.
- Ignoring Document Frequency in BM25: Forgetting to compute corpus-wide IDF makes common stopwords dominate search rankings.
Practice assignment
Take the pipeline you built and break it deliberately, one stage at a time, measuring what each break costs:
-
Set
overlap = 0and find a question whose answer now straddles a boundary. Record the question and both chunks. -
Replace RRF with a plain sum of BM25 and cosine scores. Find a query where the ranking is now dominated by BM25’s larger numbers.
-
Remove
.lower()from the BM25 tokeniser and measure how many queries stop matching anything at all.
Write one paragraph per break describing the symptom a user would report — not the cause. Getting from symptom to cause is the skill.
Extension challenge
Add a fifth stage: deduplication before assembly.
Near-duplicate chunks are common in real corpora — a changelog repeated in three release notes, a licence header on every file — and they waste context while making one source look like three independent confirmations.
Implement Jaccard similarity over token sets, drop any chunk that scores above 0.8 against a higher-ranked one, and measure two things: how many chunks are removed on your corpus, and whether any answer changes. Then argue for or against putting the stage before fusion instead of after it, using your own numbers.
Quiz
Q1. Why does Reciprocal Rank Fusion combine BM25 and dense results by position rather than by score?
- BM25 scores are unbounded and cosine scores are not, so there is nothing to normalise
- RRF is faster to compute than adding two floating-point numbers together
- RRF retrains the embedding model on each query it receives
- Rank positions use less memory than the scores they were derived from
Show answer
Answer: A. BM25 scores are unbounded and cosine scores are not, so there is nothing to normalise
BM25 produces unbounded positive scores while cosine similarity is bounded to [-1, 1]. RRF sums 1/(k + rank), so the two scales never have to be reconciled and there is no fusion weight to tune.
Q2. A chunker uses a sliding window but omits the terminating condition. What happens?
- The first window is silently dropped from the index
- The final partial window is emitted twice
- Every chunk is truncated to exactly the overlap length
- The function raises an IndexError on the last iteration
Show answer
Answer: B. The final partial window is emitted twice
Without the break, the loop advances past the end and re-emits the tail as a duplicate final chunk. The lab tests check for this specifically.
Q3. Today's pipeline uses hashed vectors rather than a learned embedding model. What does that limit?
- The number of chunks that can be held in memory at once
- The ability to run BM25 alongside the dense retriever
- Any conclusion about retrieval quality, though the plumbing is faithful
- Whether Reciprocal Rank Fusion can merge the two rankings
Show answer
Answer: C. Any conclusion about retrieval quality, though the plumbing is faithful
The mechanics of dense retrieval are demonstrated correctly, but hashed vectors carry no semantics, so the pipeline says nothing about how good real retrieval would be.
Q4. Why does the lesson argue for building the pipeline once before adopting a framework?
- Framework licences prohibit commercial use of retrieval code
- Frameworks are measurably slower than hand-written Python
- A hand-written pipeline scales further than a vector database
- The bugs live in the seams, which configuration hides
Show answer
Answer: D. The bugs live in the seams, which configuration hides
Most RAG incidents are seam failures: a table split across chunks, a silently truncated prompt, a query embedded with the wrong model version. None is visible from a config file.
Q5. What does the BM25 parameter b control?
- Length normalisation, so long documents do not win by containing more words
- The saturation point beyond which extra term occurrences stop counting
- How strongly inverse document frequency down-weights common terms
- The number of documents returned before fusion is applied
Show answer
Answer: A. Length normalisation, so long documents do not win by containing more words
b (typically 0.75) scales the document-length term. k1 saturates term frequency; IDF is a separate factor entirely.
Glossary
- Retrieval-Augmented Generation (RAG)
- An AI framework combining external knowledge retrieval from databases with generative LLM inference for grounded answers.
- Parametric Memory
- Knowledge encoded directly into the neural network weights during pre-training and fine-tuning.
- Non-Parametric Memory
- External knowledge stored in vector stores, databases, and document files accessed dynamically at inference time.
- Reciprocal Rank Fusion (RRF)
- A rank-based algorithm that aggregates heterogeneous search candidate rankings using reciprocal rank scoring.
- Okapi BM25
- A non-linear probabilistic ranking function used in sparse keyword search based on term frequency and inverted document frequency.
- Cross-Encoder Re-Ranker
- A neural model that processes query and document pairs jointly through full cross-attention to compute precise relevance scores.
- Bi-Encoder
- A neural architecture that encodes queries and documents into separate dense vector embeddings independently.
- Inline Citation
- A bracketed numerical marker ([1], [2]) linking a generated factual claim directly to a retrieved source document passage.
- RAG Triad
- An evaluation framework measuring Faithfulness, Answer Relevance, and Context Precision in RAG applications.
- Context Precision
- The proportion of retrieved context passages that contain ground-truth information relevant to answering the query.
Sources and further reading
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020) — NeurIPS / arXiv (accessed 2026-03-30)
- REALM: Retrieval-Augmented Language Model Pre-Training (Guu et al., 2020) — ICML / arXiv (accessed 2026-03-30)
- Introducing Contextual Retrieval — Anthropic Engineering (accessed 2026-08-31)
- Qdrant Vector Database Documentation & Hybrid Search — Qdrant Technologies (accessed 2026-03-30)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.