LLMs and Generative AI › Retrieval-Augmented Generation › Day 268

Day 268: A Minimal RAG System from Scratch

Day 268 of 365 — A Minimal RAG System from Scratch

Construct a complete, production-clean RAG pipeline in pure Python without external database dependencies: implement document chunking, in-memory dense vector cosine scoring, sparse BM25 inverted indexing, prompt context assembly, and hallucination containment.

Course
LLMs and Generative AI
Category
Retrieval-Augmented Generation
Reading time
≈ 35 min
Practical time
≈ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-268-a-minimal-rag-system-from-scratch

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-268-a-minimal-rag-system-from-scratch
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

You have used retrieval for four days without ever seeing it work. Yesterday’s lesson gave you the architecture; today you build the whole thing in about a hundred lines of pure Python, with no libraries at all.

That is not a stunt. You cannot debug what you have only ever imported. When a production pipeline returns a plausible answer sourced from the wrong document, the question is always the same — did retrieval fail, did fusion fail, or did the prompt fail? Someone who has written those three stages can answer in minutes. Someone who has only configured them is left restarting the vector database and hoping.

The whole apparatus reduces to arithmetic you already know:

  chunk        split text on word boundaries with an overlap
  index        count terms (BM25) and store vectors (embeddings)
  retrieve     score every chunk, keep the top k
  fuse         combine two rankings by position, not by score
  assemble     paste the survivors into a prompt with numbered labels

There is no step in that list that requires a framework. There is no step that cannot be tested. And once you have written it, every later decision — which vector store, which reranker, which chunk size — becomes a comparison against something you understand rather than a guess between two brand names.


The idea in plain language

Diagram: the five components of a minimal retrieval system

A minimal RAG system is four small functions in a row, and the only interesting mathematics is the dot product.

Ask a librarian for “the paper about attention in translation”. They do two things at once without noticing. They look for those literal words on spines and in indexes — that is lexical search, and it is what BM25 does. They also think about what you meant, and fetch a paper titled “Neural Machine Translation by Jointly Learning to Align and Translate” that contains none of your words — that is semantic search, and it is what an embedding does.

Neither is sufficient. Lexical search fails on synonyms; semantic search fails on exact identifiers, error codes and part numbers. So you run both and merge the two lists. Then you hand the survivors to someone who can write, with the instruction: use only these pages, and cite them.

That is the entire system. Today you write it.


Historical background

Diagram: the historical progression of retrieval pipeline implementations

The two halves of retrieval were invented decades apart and have only recently been used together.

The lesson of that timeline is that nothing here is new or proprietary. The pieces are old, well-documented, and small enough to write yourself.


What it is — and what it is not

Diagram: a hand-built retrieval pipeline compared with a framework

It is a complete, honest, end-to-end RAG pipeline. Every stage a production system has, this has: chunking, lexical indexing, dense indexing, fusion, prompt assembly. The results are real results, and the failures are real failures.

It is not production infrastructure, and pretending otherwise would teach you the wrong lesson. Specifically it lacks:

MissingWhy it does not matter todayWhen it starts to
PersistenceThe index rebuilds in under a secondA corpus that takes minutes to embed
Approximate nearest neighbourAn exhaustive scan of 10,000 vectors is fastRoughly 100,000 chunks and up
ConcurrencyOne query at a time is fine while learningMore than one user
A real embedding modelA hashed bag-of-words shows the mechanicsAny question about actual quality

The last row deserves emphasis. Today’s “embeddings” are deterministic hashes, not learned representations. They demonstrate the plumbing of dense retrieval faithfully while telling you nothing about its quality. That distinction — between a mechanism that runs and a mechanism that works — is the one this course keeps returning to.


Why it was created and what problems it solves

Frameworks hide the seams, and the seams are where the bugs live.

Ask a working RAG team what actually went wrong in their last incident and you will rarely hear “the vector database was down”. You will hear things like these, every one of which lives in a seam between two stages:

None of those is visible from a configuration file. All of them are obvious to someone who has written the four functions and knows what each hands to the next. Building it once is the cheapest insurance available against a whole category of bug.


How it works

Minimal In-Memory RAG Pipeline diagram

1. Chunking on word boundaries, with overlap

Diagram: the four decisions encoded in a simple chunking function

Text is split into windows of chunk_size words that overlap by overlap words. The overlap exists because a fact can straddle a boundary: a sentence ending “…the write cache costs” followed by “$3.75 per million tokens” is useless if the two halves land in different chunks and only one is retrieved.

The loop must terminate on i + chunk_size >= len(words), or the final partial window is emitted twice — a bug the lab’s tests catch deliberately.

2. BM25, in about fifteen lines

Diagram: the three components of a term-based relevance score

For a query term q and document d, BM25 scores

  IDF(q) · ( f(q,d) · (k1 + 1) ) / ( f(q,d) + k1 · (1 - b + b · |d|/avgdl) )

f(q,d) is the term’s frequency in the document. k1 (typically 1.5) saturates that frequency, so the twentieth occurrence of a word adds far less than the second. b (typically 0.75) normalises for document length, so a long document does not win simply by containing more words. IDF down-weights terms that appear everywhere.

You do not need to memorise the formula. You do need to know that it produces unbounded positive scores — which is why the next stage cannot simply add it to a cosine similarity.

3. Dense retrieval as a dot product

Diagram: how dense retrieval reduces to a single matrix operation

Every chunk becomes a vector. Every query becomes a vector. Relevance is the cosine of the angle between them, and for unit-normalised vectors cosine similarity is the dot product — one multiply-add per dimension, nothing more.

Scores land in [-1, 1]. BM25’s do not. That mismatch is the whole reason the next stage exists.

4. Reciprocal Rank Fusion

Diagram: how two ranked lists are fused into one in a few lines of code

RRF throws the scores away and keeps only the positions:

  RRF(d) = Σ  1 / (k + rank_i(d))          k = 60 by convention

A document ranked first in either list contributes 1/61. Ranked tenth, 1/70. Because only ranks enter the sum, BM25’s unbounded scale and cosine’s bounded one never have to be reconciled — there is nothing to normalise and no weight to tune. A document that both methods rank highly rises above one that either ranks first alone, which is exactly the behaviour you want.

5. Prompt assembly

Diagram: how retrieved passages are assembled into a grounded prompt in code

The survivors are numbered and pasted into the context with an instruction to answer only from them and to cite by number. This stage looks trivial and is where a surprising share of production failures happen: silent truncation, missing separators between chunks, and citation labels that do not match the order the chunks were pasted in.


An everyday analogy

Diagram: using a machine as a black box compared with having taken it apart

Building this from scratch is assembling a bicycle from a box of parts before buying a motorbike.

Nobody commutes on the bicycle. But afterwards you know what a bottom bracket is, you know that the brakes rub because the wheel is not seated straight, and when the motorbike develops a noise you have a mental model of what is connected to what. The person who has only ever ridden has to hand it to a mechanic and believe whatever comes back.

The bicycle takes an afternoon. The understanding lasts.


Examples in practice

The four stages, at the size they actually are.

Chunking

def chunk(text: str, size: int = 120, overlap: int = 20) -> list[str]:
    words, out, i = text.split(), [], 0
    while i < len(words):
        out.append(" ".join(words[i : i + size]))
        if i + size >= len(words):      # last window; stop or it repeats
            break
        i += size - overlap
    return out

Lexical scoring

def bm25(query: str, docs: list[list[str]], k1=1.5, b=0.75) -> list[float]:
    N, avgdl = len(docs), sum(len(d) for d in docs) / len(docs)
    terms = re.findall(r"\w+", query.lower())      # lowercase, or nothing matches
    scores = [0.0] * N
    for t in terms:
        n_t = sum(1 for d in docs if t in d)
        if not n_t:
            continue
        idf = math.log(1 + (N - n_t + 0.5) / (n_t + 0.5))
        for i, d in enumerate(docs):
            f = d.count(t)
            scores[i] += idf * (f * (k1 + 1)) / (f + k1 * (1 - b + b * len(d) / avgdl))
    return scores

Fusion

def rrf(*rankings: list[int], k: int = 60) -> list[int]:
    score: dict[int, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            score[doc_id] = score.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(score, key=score.get, reverse=True)

Naive RAG Prompt Synthesis Flow diagram

Prompt assembly

def build_prompt(question: str, chunks: list[str]) -> str:
    context = "\n\n".join(f"[{i}] {c}" for i, c in enumerate(chunks, start=1))
    return (
        "Answer using ONLY the context below. Cite sources as [1], [2].\n"
        "If the context does not contain the answer, say so.\n\n"
        f"{context}\n\nQuestion: {question}"
    )

That is the system. Roughly sixty lines, four tests, no dependencies.


Implications: security, privacy, performance, scalability, and cost

Diagram: the four capabilities a production retrieval system adds to a minimal one

Performance. Scoring is a linear scan: every query touches every chunk. At 10,000 chunks of 120 words that is milliseconds and entirely acceptable. The scan is what an approximate index later replaces, and Day 326 measures exactly what that trade buys and costs.

Scalability. Memory holds the whole corpus twice — once as text, once as vectors. A float32 vector of 384 dimensions is 1.5 KB, so 100,000 chunks is about 150 MB of vectors before the text. That is the point at which “keep it in a Python list” stops being a reasonable answer.

Security. Retrieval is an access-control boundary, and an in-memory list has none. Anything in the index is reachable by any query. In production the filter must be applied before chunks enter the context, not after — a model that has seen a passage cannot be asked to forget it.

Privacy. Chunks are copies. A document deleted at source remains in the index, and in the embeddings, until something removes it — which is why Day 325 treats freshness as a correctness property rather than a nicety.

Cost. Zero today, and that is itself instructive: every cost in a production RAG system comes from the embedding model, the vector store, or the generation call. The retrieval logic is free.


Alternatives: free, open source, and commercial

OptionLicenceWhat it gives youWhat it costs
Writing it yourself—Complete understanding; no dependency riskYou maintain it
LangChainMIT, freeChains, loaders, and a large integration surfaceAbstraction depth; upgrades can move behaviour
LlamaIndexMIT, freeIndexing and query engines aimed squarely at RAGOpinionated structure
HaystackApache-2.0, freeExplicit, typed pipeline graphsMore ceremony for small jobs
rank_bm25Apache-2.0, freeJust the lexical half, well testedNothing else

The honest recommendation is neither “always build” nor “always import”. Build it once, today, so you can read a framework’s source and recognise what it is doing. Then import, because a maintained BM25 implementation has fixed edge cases yours has not met yet.


Diagram: a minimal retrieval implementation compared with a production one

ApproachWhere knowledge livesUpdate costAttribution
This pipelineAn external indexRe-index one documentExact chunk, by number
Fine-tuningModel weightsA full training runNone
Long-context stuffingThe prompt itselfFreeWhole document only
Plain keyword searchAn inverted indexRe-indexExact, but no synthesis

Against fine-tuning: retrieval changes what the model knows; fine-tuning changes how it behaves. Day 274 draws that line properly.

Against long-context stuffing: if the corpus fits in the context window, stuffing is simpler and often better. Retrieval earns its complexity when the corpus does not fit, or when you need to say which passage an answer came from.


When to use it — and when not to

Diagram: when a hand-built retrieval system is the right choice

Use a hand-written pipeline when you are learning the mechanism, the corpus is small enough to scan exhaustively, you need zero dependencies, or you are building a reference implementation to test a framework against.

Reach for a library when the corpus outgrows memory, you need persistence across restarts, several people query at once, or you need real embeddings from a maintained model.

Do not use retrieval at all when the answer is already in the model, the corpus fits comfortably in the context window, or the question is about reasoning rather than facts. Retrieval adds two failure modes — retrieving the wrong thing, and retrieving nothing — and they are only worth accepting when grounding is genuinely required.


The AI connection

Knowledge check

quiz:
  question: "Why does Reciprocal Rank Fusion combine BM25 and dense results by position rather than by score?"
  options:
    - "RRF is faster to compute than adding two floating-point numbers together"
    - "BM25 scores are unbounded and cosine scores are not, so there is nothing to normalise"
    - "RRF retrains the embedding model on each query it receives"
    - "Rank positions use less memory than the scores they were derived from"
  answer_index: 1
  explanation: "BM25 produces unbounded positive scores while cosine similarity is bounded to [-1, 1]. RRF sums 1/(k + rank), so the two scales never have to be reconciled and there is no fusion weight to tune."

Hands-on exercise

In this exercise, you will build and evaluate a modular Python RAG pipeline that ingests a sample technical knowledge base, performs hybrid lexical-dense search, constructs a grounded system prompt, and validates answer grounding.

Expected output

When running your test suite, you should observe:

============================= test session starts ==============================
collected 4 items

tests/test_minimal_rag_lib.py::test_document_chunker PASSED              [ 25%]
tests/test_minimal_rag_lib.py::test_sparse_bm25 PASSED                   [ 50%]
tests/test_minimal_rag_lib.py::test_hybrid_ranker PASSED                 [ 75%]
tests/test_minimal_rag_lib.py::test_prompt_synthesizer PASSED            [100%]

============================== 4 passed in 0.04s ===============================

Validate your work

Run the pytest test runner script:

pytest tests/ -v

Troubleshooting

Common mistakes


Practice assignment

Take the pipeline you built and break it deliberately, one stage at a time, measuring what each break costs:

  1. Set overlap = 0 and find a question whose answer now straddles a boundary. Record the question and both chunks.

  2. Replace RRF with a plain sum of BM25 and cosine scores. Find a query where the ranking is now dominated by BM25’s larger numbers.

  3. Remove .lower() from the BM25 tokeniser and measure how many queries stop matching anything at all.

Write one paragraph per break describing the symptom a user would report — not the cause. Getting from symptom to cause is the skill.


Extension challenge

Add a fifth stage: deduplication before assembly.

Near-duplicate chunks are common in real corpora — a changelog repeated in three release notes, a licence header on every file — and they waste context while making one source look like three independent confirmations.

Implement Jaccard similarity over token sets, drop any chunk that scores above 0.8 against a higher-ranked one, and measure two things: how many chunks are removed on your corpus, and whether any answer changes. Then argue for or against putting the stage before fusion instead of after it, using your own numbers.


Quiz

Q1. Why does Reciprocal Rank Fusion combine BM25 and dense results by position rather than by score?

  1. BM25 scores are unbounded and cosine scores are not, so there is nothing to normalise
  2. RRF is faster to compute than adding two floating-point numbers together
  3. RRF retrains the embedding model on each query it receives
  4. Rank positions use less memory than the scores they were derived from
Show answer

Answer: A. BM25 scores are unbounded and cosine scores are not, so there is nothing to normalise

BM25 produces unbounded positive scores while cosine similarity is bounded to [-1, 1]. RRF sums 1/(k + rank), so the two scales never have to be reconciled and there is no fusion weight to tune.

Q2. A chunker uses a sliding window but omits the terminating condition. What happens?

  1. The first window is silently dropped from the index
  2. The final partial window is emitted twice
  3. Every chunk is truncated to exactly the overlap length
  4. The function raises an IndexError on the last iteration
Show answer

Answer: B. The final partial window is emitted twice

Without the break, the loop advances past the end and re-emits the tail as a duplicate final chunk. The lab tests check for this specifically.

Q3. Today's pipeline uses hashed vectors rather than a learned embedding model. What does that limit?

  1. The number of chunks that can be held in memory at once
  2. The ability to run BM25 alongside the dense retriever
  3. Any conclusion about retrieval quality, though the plumbing is faithful
  4. Whether Reciprocal Rank Fusion can merge the two rankings
Show answer

Answer: C. Any conclusion about retrieval quality, though the plumbing is faithful

The mechanics of dense retrieval are demonstrated correctly, but hashed vectors carry no semantics, so the pipeline says nothing about how good real retrieval would be.

Q4. Why does the lesson argue for building the pipeline once before adopting a framework?

  1. Framework licences prohibit commercial use of retrieval code
  2. Frameworks are measurably slower than hand-written Python
  3. A hand-written pipeline scales further than a vector database
  4. The bugs live in the seams, which configuration hides
Show answer

Answer: D. The bugs live in the seams, which configuration hides

Most RAG incidents are seam failures: a table split across chunks, a silently truncated prompt, a query embedded with the wrong model version. None is visible from a config file.

Q5. What does the BM25 parameter b control?

  1. Length normalisation, so long documents do not win by containing more words
  2. The saturation point beyond which extra term occurrences stop counting
  3. How strongly inverse document frequency down-weights common terms
  4. The number of documents returned before fusion is applied
Show answer

Answer: A. Length normalisation, so long documents do not win by containing more words

b (typically 0.75) scales the document-length term. k1 saturates term frequency; IDF is a separate factor entirely.

Glossary

Retrieval-Augmented Generation (RAG)
An AI framework combining external knowledge retrieval from databases with generative LLM inference for grounded answers.
Parametric Memory
Knowledge encoded directly into the neural network weights during pre-training and fine-tuning.
Non-Parametric Memory
External knowledge stored in vector stores, databases, and document files accessed dynamically at inference time.
Reciprocal Rank Fusion (RRF)
A rank-based algorithm that aggregates heterogeneous search candidate rankings using reciprocal rank scoring.
Okapi BM25
A non-linear probabilistic ranking function used in sparse keyword search based on term frequency and inverted document frequency.
Cross-Encoder Re-Ranker
A neural model that processes query and document pairs jointly through full cross-attention to compute precise relevance scores.
Bi-Encoder
A neural architecture that encodes queries and documents into separate dense vector embeddings independently.
Inline Citation
A bracketed numerical marker ([1], [2]) linking a generated factual claim directly to a retrieved source document passage.
RAG Triad
An evaluation framework measuring Faithfulness, Answer Relevance, and Context Precision in RAG applications.
Context Precision
The proportion of retrieved context passages that contain ground-truth information relevant to answering the query.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.