LLMs and Generative AI › Retrieval-Augmented Generation › Day 271

Day 271: Advanced RAG Patterns

Day 271 of 365 — Advanced RAG Patterns

Elevate retrieval precision beyond naive vector search: implement Multi-Query generation, sub-question decomposition, HyDE (Hypothetical Document Embeddings), and Parent-Document retrieval with small-to-big context expansion.

Course
LLMs and Generative AI
Category
Retrieval-Augmented Generation
Reading time
≈ 35 min
Practical time
≈ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-271-advanced-rag-patterns

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-271-advanced-rag-patterns
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Days 268 to 270 assumed something that is usually false: that the user’s question resembles the passage that answers it.

It frequently does not. A question and its answer are different kinds of text.

  question   "why is my build slow"
  answer     "Layer cache invalidation occurs when any instruction above a
              COPY directive is modified, forcing rebuild of all subsequent
              layers."

Almost no shared vocabulary. BM25 finds nothing. A dense retriever does better but is still comparing a six-word fragment against a technical paragraph — short queries have unstable embeddings, because there is very little signal to average over.

Every technique today attacks that gap from a different side. Generate several phrasings of the question. Generate a fake answer and search with that instead. Search small and return large. None is exotic; each is a few lines, and each targets a specific, measurable failure of the naive approach.


The idea in plain language

Diagram: where advanced retrieval patterns intervene in the pipeline

The trick is to stop searching with the question.

A researcher handed “why is my build slow” does not type those words into a library catalogue. They think for a moment about what a document answering it would look like — it would mention caching, layers, invalidation, dependencies — and they search for that. They are searching with an imagined answer.

That is HyDE: have the model write a hypothetical answer, embed the hypothetical answer, and search with it. The hypothetical may be factually wrong. It does not matter, because it is never shown to anyone. It is being used purely as a better-shaped query — a paragraph of the right kind of language, sitting in roughly the right neighbourhood of the vector space.

The second idea is just as simple. Ask the question three different ways and take the union. Where a single phrasing might miss, three rarely all miss.


Historical background

Diagram: the historical progression of retrieval augmentation patterns

The consistent thread across fifty years is that the query as typed is rarely the best query available.


What it is — and what it is not

It is a set of independent, composable techniques applied at query time. Each addresses a distinct failure of naive retrieval, and each can be measured on its own.

It is not a package deal, and applying all of them is usually wrong. Each costs latency, and several cost an extra model call per query.

TechniqueThe failure it fixesCost
Multi-queryOne phrasing misses1 model call, N searches
HyDEQuestion and answer look different1 model call
Small-to-bigPrecise match, insufficient contextNone — an index-time choice
Self-queryFilterable constraints buried in prose1 model call
RerankingGood recall, poor ordering1 cross-encoder pass

The discipline that matters is measuring before adopting. “Advanced” is not a synonym for “better” — on a corpus where the naive approach already retrieves the right chunk, every one of these adds latency and changes nothing.


Why it was created and what problems it solves

Four specific, recognisable failures:

The vocabulary mismatch. The user says “slow build”; the document says “layer cache invalidation”. No lexical overlap, weak semantic overlap. Multi-query and HyDE both attack this.

The unstable short query. A three-word query produces an embedding dominated by whichever word happens to carry the most weight. Averaging several phrasings, or embedding a full paragraph instead, stabilises it.

The granularity conflict. Small chunks match precisely and lack the surrounding context needed to answer. Large chunks carry context and match diffusely. Small-to-big resolves this by refusing to choose: search small, return the parent.

The buried constraint. “What did we ship in Q3 2024?” contains a date filter disguised as prose. Semantic search over a date range is hopeless; a metadata filter is trivial. Self-querying extracts the filter and applies it properly.

Recognising which of the four you actually have is most of the work. Applying the matching technique is the easy part.


How it works

Advanced RAG Query Expansion and HyDE diagram

1. Multi-query expansion

Diagram: how a single question is expanded into several search queries

Ask the model for three alternative phrasings, run all of them, and fuse the result lists with Reciprocal Rank Fusion — the same function from Day 268, now merging four rankings instead of two.

  "why is my build slow"
       ├── "Docker build cache invalidation causes"
       ├── "reasons a container image rebuild takes a long time"
       └── "how layer ordering affects build duration"

A chunk retrieved by two phrasings outranks one retrieved by a single phrasing. That is exactly the consensus signal RRF was designed to capture.

2. HyDE

Diagram: how a hypothetical answer is used as the search query

Ask the model to answer the question from its own knowledge, embed that answer, and search with its vector. Discard the hypothetical text afterwards.

The hypothetical may contain errors. It is never returned to the user and never enters the final context — its only job is to be a paragraph-shaped query in the right region of the embedding space. The retrieved passages are real; only the probe was invented.

3. Small-to-big retrieval

Diagram: how a small matched chunk is expanded to its surrounding context

Index small chunks — a sentence or two — for precision. Store each one’s parent: the surrounding section or the whole document. Search over the small units, then return the parents.

This is the cheapest technique here because it costs nothing at query time. It is an indexing decision made once, and it dissolves the precision-versus-context trade rather than balancing it.

4. Self-querying

Diagram: how structured filters are extracted from a natural language question

Have the model extract structured filters from the question before searching:

{ "semantic": "attention mechanisms", "filters": { "year": { "$gte": 2020 } } }

Apply the filter as a filter, and search semantically over what survives. A date range is a database operation; asking an embedding to represent “after 2020” is asking it to do arithmetic it cannot do.

5. Reranking

Diagram: how reranking reorders retrieved candidates before generation

Retrieval optimises recall; reranking optimises order. A cross-encoder reads query and passage together — full attention across both — and scores relevance far more accurately than a dot product between two independently computed vectors.

It is too slow to run over the corpus and ideal over 50 candidates. Retrieve 50 cheaply, rerank to 5 accurately. Note the ordering constraint from Day 326: a reranker can only reorder what retrieval already returned. If the right document was never a candidate, no amount of reranking will surface it.


An everyday analogy

Diagram: searching for the words asked compared with searching for what was meant

A junior researcher types your exact words into the catalogue and reports back that there is nothing.

A senior one does something different before searching at all. They rephrase the question three ways because they know the catalogue’s vocabulary differs from yours. They think about what a paper answering this would be titled and search for that instead. They search chapter titles but hand you the whole chapter, because a sentence out of context is no use. And when you say “recent work”, they convert that to year >= 2020 rather than hoping the catalogue understands the word “recent”.

Same catalogue, same query, very different results — because the senior researcher does not treat your question as the query.


Examples in practice

Parent-Document Hierarchical Retrieval Tree diagram

Multi-query with RRF

def multi_query(question: str, k: int = 5) -> list[Chunk]:
    variants = [question] + generate_paraphrases(question, n=3)
    rankings = [dense_search(v, k=20) for v in variants]
    return rrf(*rankings)[:k]          # rrf() is Day 268's, unchanged

HyDE

HYDE_PROMPT = (
    "Write a short factual paragraph that would answer this question. "
    "Do not hedge, do not say you are unsure — accuracy is not required.\n\n"
    "Question: {q}"
)

def hyde_search(question: str, k: int = 5) -> list[Chunk]:
    hypothetical = llm(HYDE_PROMPT.format(q=question))
    # The text is thrown away; only its vector is used.
    return dense_search_by_vector(embed(hypothetical), k=k)

Small-to-big

def small_to_big(question: str, k: int = 3) -> list[str]:
    hits = dense_search(question, k=k * 4)        # precise, narrow units
    parents, seen = [], set()
    for h in hits:
        if h.parent_id not in seen:               # dedupe: siblings share a parent
            seen.add(h.parent_id)
            parents.append(PARENTS[h.parent_id])
        if len(parents) == k:
            break
    return parents

Measuring whether any of it helped

for name, retrieve in [("naive", naive), ("multi", multi_query), ("hyde", hyde_search)]:
    hits = sum(gold_id in {c.id for c in retrieve(q)} for q, gold_id in eval_set)
    print(f"{name:6} recall@5 = {hits / len(eval_set):.2f}")

That last block is the important one. Every technique above is a hypothesis until this table says otherwise.


Implications: security, privacy, performance, scalability, and cost

Diagram: the quality, latency, cost and complexity implications of advanced retrieval patterns

Performance. Multi-query multiplies searches by the number of variants and adds a model call before any search begins. HyDE adds a model call on the critical path — typically 200–800 ms, which for an interactive assistant is often the single largest latency item.

Cost. Each query-time model call is a real per-query cost. At a million queries a month, a small model at $0.15 per million tokens generating 100 tokens per call is roughly $15 — trivial. The same pattern with a frontier model is not.

Scalability. Small-to-big increases index size modestly (small chunks plus parent references) and costs nothing at query time. It is usually the best value in the list and it is the one people reach for last.

Security. Self-querying lets model output construct a database filter. Treat the extracted filter as untrusted input: validate field names against an allow-list and types against a schema. A model that can write arbitrary filters can write {"user_id": {"$ne": null}}.

Privacy. HyDE sends the user’s question to a generation model before any retrieval happens. In a deployment where retrieval is local but generation is hosted, that is a data-egress decision made implicitly by a retrieval technique.


Alternatives: free, open source, and commercial

TechniqueAvailabilityAdds latencyTypical recall gain
Multi-queryAny model1 call + N searchesModerate, consistent
HyDEAny model1 callLarge where vocabulary differs
Small-to-bigIndex-time onlyNoneLarge where context matters
Self-queryAny model1 callLarge where filters exist, none otherwise
Cross-encoder rerankbge-reranker, Cohere Rerank50–200 msLarge on ordering

Ranked by value per unit of complexity, the order is usually: small-to-big first because it is free at query time, then reranking because the gain is reliable, then HyDE, then multi-query, and self-query only when the corpus actually has metadata worth filtering on.


Diagram: the cost and benefit of four advanced retrieval patterns

ApproachWhen it actsWhat it changes
Better chunkingIndex timeWhat can be retrieved at all
Query expansionQuery timeWhat is retrieved for this question
RerankingAfter retrievalThe order of what was retrieved
Larger context windowGenerationHow much of it the model can read

These stack rather than compete, and they have a strict precedence. No amount of query expansion recovers a fact that chunking destroyed, and no reranker promotes a document retrieval never returned. Fix them in that order — index, then query, then ranking — or you will tune the wrong stage.


When to use it — and when not to

Diagram: choosing a retrieval pattern by the observed failure

Reach for these when measured recall is the bottleneck, users phrase questions very differently from the source material, the corpus mixes short identifiers with long prose, or answers need more context than the matching chunk contains.

Do not reach for them when you have not measured a baseline. Every technique here is justified by a number, and adopting them by reputation is how a pipeline acquires 800 ms of latency and no improvement.

Prefer the cheap ones first. Small-to-big and reranking cost no extra model call. Exhaust the free options before adding a generation call to the critical path of every query.


The AI connection

Knowledge check

quiz:
  question: "In HyDE, why does it not matter if the hypothetical answer contains factual errors?"
  options:
    - "It is never shown to anyone; only its vector is used as the query"
    - "The errors are corrected by the reranking stage before generation"
    - "The model verifies the hypothetical against the corpus before searching"
    - "Errors are removed by lowercasing and stripping punctuation"
  answer_index: 0
  explanation: "The hypothetical exists to be a paragraph-shaped probe in roughly the right region of embedding space. It is discarded after embedding and never enters the final context."

Hands-on exercise

In this exercise, you will build and evaluate a modular Python RAG pipeline that ingests a sample technical knowledge base, performs hybrid lexical-dense search, constructs a grounded system prompt, and validates answer grounding.

Expected output

When running your test suite, you should observe:

============================= test session starts ==============================
collected 4 items

tests/test_advanced_rag_lib.py::test_document_chunker PASSED              [ 25%]
tests/test_advanced_rag_lib.py::test_sparse_bm25 PASSED                   [ 50%]
tests/test_advanced_rag_lib.py::test_hybrid_ranker PASSED                 [ 75%]
tests/test_advanced_rag_lib.py::test_prompt_synthesizer PASSED            [100%]

============================== 4 passed in 0.04s ===============================

Validate your work

Run the pytest test runner script:

pytest tests/ -v

Troubleshooting

Common mistakes


Practice assignment

Build a twenty-question evaluation set against your Day 268 corpus, recording for each question the chunk id that genuinely answers it.

Measure recall@5 for four configurations: naive dense search, multi-query, HyDE, and small-to-big. Report four numbers and the latency of each.

Then find one question where an advanced technique performs worse than naive retrieval, and explain why. There will be one — HyDE hurts when the model’s hypothetical wanders into a different topic, and multi-query hurts when a paraphrase drifts. Knowing when a technique backfires is worth more than knowing its average gain.


Extension challenge

Implement adaptive retrieval: choose the technique per query rather than applying one to all.

Route on cheap signals — query length, presence of a date or identifier, whether the naive search’s top score clears a confidence threshold. Short vague queries go to HyDE; queries containing an exact identifier go straight to BM25; queries with a date go to self-query.

Measure recall and mean latency against always-HyDE and always-naive. Report whether the routing paid for its own complexity. If it did not, say so — a negative result honestly measured is the point of the exercise.


Quiz

Q1. In HyDE, why does it not matter if the hypothetical answer contains factual errors?

  1. Errors are removed by lowercasing and stripping punctuation
  2. The errors are corrected by the reranking stage before generation
  3. The model verifies the hypothetical against the corpus before searching
  4. It is never shown to anyone; only its vector is used as the query
Show answer

Answer: D. It is never shown to anyone; only its vector is used as the query

The hypothetical exists to be a paragraph-shaped probe in roughly the right region of embedding space. It is discarded after embedding.

Q2. Which advanced technique costs nothing at query time?

  1. Small-to-big retrieval, which is an indexing decision
  2. Multi-query expansion, because the variants run in parallel
  3. HyDE, because the hypothetical answer is short
  4. Self-querying, because filters are applied by the database
Show answer

Answer: A. Small-to-big retrieval, which is an indexing decision

Small-to-big is decided once when the index is built. Every other technique adds a model call or extra searches per query.

Q3. Why does asking an embedding to handle papers after 2020 work badly?

  1. Embedding models are trained only on text published before 2020
  2. A date range is a filter, and vectors cannot do arithmetic
  3. The year is tokenised into digits that carry no meaning
  4. Semantic search returns results in chronological order already
Show answer

Answer: B. A date range is a filter, and vectors cannot do arithmetic

Self-querying extracts the constraint and applies it as a database filter, leaving the embedding to do the part it is good at.

Q4. What is the strict precedence when fixing a retrieval problem?

  1. Query expansion, then ranking, then chunking
  2. Ranking, then chunking, then query expansion
  3. Chunking, then query expansion, then ranking
  4. All three can be tuned independently in any order
Show answer

Answer: C. Chunking, then query expansion, then ranking

No query expansion recovers a fact that chunking destroyed, and no reranker promotes a document retrieval never returned.

Q5. Why measure a baseline before adopting these techniques?

  1. Without one, the embedding model cannot be initialised
  2. The techniques are only licensed for use after benchmarking
  3. Baselines are required to compute Reciprocal Rank Fusion
  4. Advanced is not a synonym for better, and each adds latency
Show answer

Answer: D. Advanced is not a synonym for better, and each adds latency

On a corpus where naive retrieval already finds the right chunk, every technique adds cost and changes nothing.

Glossary

Retrieval-Augmented Generation (RAG)
An AI framework combining external knowledge retrieval from databases with generative LLM inference for grounded answers.
Parametric Memory
Knowledge encoded directly into the neural network weights during pre-training and fine-tuning.
Non-Parametric Memory
External knowledge stored in vector stores, databases, and document files accessed dynamically at inference time.
Reciprocal Rank Fusion (RRF)
A rank-based algorithm that aggregates heterogeneous search candidate rankings using reciprocal rank scoring.
Okapi BM25
A non-linear probabilistic ranking function used in sparse keyword search based on term frequency and inverted document frequency.
Cross-Encoder Re-Ranker
A neural model that processes query and document pairs jointly through full cross-attention to compute precise relevance scores.
Bi-Encoder
A neural architecture that encodes queries and documents into separate dense vector embeddings independently.
Inline Citation
A bracketed numerical marker ([1], [2]) linking a generated factual claim directly to a retrieved source document passage.
RAG Triad
An evaluation framework measuring Faithfulness, Answer Relevance, and Context Precision in RAG applications.
Context Precision
The proportion of retrieved context passages that contain ground-truth information relevant to answering the query.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.