LLMs and Generative AI › Embeddings and Vector Search › Day 266

Hands-on lab — Day 266: Semantic Search over Your Own Notes

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/e2e_rag_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/e2e_rag_lib.py
examples/test_e2e_rag_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/e2e_rag_lib.py
starter/test_e2e_rag_lib.py
tests/run_tests.sh
tests/test_e2e_rag_lib.py
troubleshooting.md

Lab README

Lab: Day 266 -- End-to-End RAG System

Lesson

Day number: 266 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: End-to-End RAG System Architecture, Ingestion, Generation, and Citations.

Purpose

Build and test a complete Modular End-to-End RAG System in Python. Implement document ingestion, candidate retrieval, confidence threshold fallback gating, citation-enforced prompt synthesis, and grounded answer generation.

Learning objectives

  • Synthesize ingestion, retrieval, re-ranking, and prompt synthesis into a single unified pipeline.
  • Implement confidence threshold gating to prevent hallucinations.
  • Enforce source citations and claim verification.
  • Structure modular, production-ready Python RAG architectures.

Prerequisites

  • Day 265 (Evaluating RAG with RAGAS and TruLens).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Pure Python standard libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/e2e_rag_lib.py: Student scaffold file.
  • examples/e2e_rag_lib.py: Complete reference implementation.
  • tests/test_e2e_rag_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/e2e_rag_lib.py

What the commands do

  • Ingests documentation corpus.
  • Executes query retrieval and prompt synthesis.
  • Generates answer with verified citations.
  • Triggers fallback refusal for unknown queries.
  • Runs unit test assertions.

Expected output

Known Query Answer: According to documentation, API gateway tokens expire after 3600 seconds. [1].
Unknown Query Answer: I do not have sufficient documentation to answer this question accurately.

Validation steps

  1. Verify high-confidence queries return grounded answers with [1] citation tags.
  2. Confirm unknown queries trigger graceful refusal messages.
  3. Validate prompt synthesis correctly formats source metadata blocks.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Refusal on valid queries: Lower the confidence_threshold parameter (e.g. from 0.50 to 0.20).

Security notes

All data is processed in-memory within local Python runtime.

Extension exercises

  1. Implement OpenTelemetry distributed tracing spans.
  2. Add multi-turn conversation history buffer.

Expected output

FIELDS.md

# Expected Output Fields: Day 266

- `Known Query Answer`: Grounded response with citation.
- `Unknown Query Answer`: Fallback message.

examples-run.txt

Known Query Answer: According to documentation, API gateway tokens expire after 3600 seconds. [1].
Unknown Query Answer: I do not have sufficient documentation to answer this question accurately.

measured-values.txt

Known Query Answer: According to documentation, API gateway tokens expire after 3600 seconds. [1].
Unknown Query Answer: I do not have sufficient documentation to answer this question accurately.

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_e2e_rag_lib.py::test_rag_ingestion_and_retrieval PASSED              [ 25%]
tests/test_e2e_rag_lib.py::test_high_confidence_generation_with_citation PASSED [ 50%]
tests/test_e2e_rag_lib.py::test_low_confidence_fallback_refusal PASSED          [ 75%]
tests/test_e2e_rag_lib.py::test_prompt_synthesis_formatting PASSED             [100%]

============================== 4 passed in 0.28s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/e2e_rag_lib.py (3081 bytes)
from typing import List, Dict, Any, Optional

class EndToEndRAGSystem:
    def __init__(self, confidence_threshold: float = 0.30):
        self.confidence_threshold = confidence_threshold
        self.documents: List[Dict[str, Any]] = []

    def ingest_corpus(self, docs: List[Dict[str, Any]]):
        self.documents = docs

    def retrieve(self, query: str, top_k: int = 3) -> List[Dict[str, Any]]:
        q_words = set(query.lower().split())
        scored = []
        for doc in self.documents:
            d_words = set(doc["text"].lower().split())
            overlap = len(q_words.intersection(d_words))
            score = overlap / max(1, len(q_words))
            scored.append((doc, score))
        
        scored.sort(key=lambda x: x[1], reverse=True)
        return [{"doc": doc, "score": score} for doc, score in scored[:top_k]]

    def synthesize_prompt(self, query: str, retrieved_items: List[Dict[str, Any]]) -> str:
        context_blocks = []
        for idx, item in enumerate(retrieved_items, 1):
            context_blocks.append(f"[{idx}] Source: {item['doc'].get('title', 'Doc')}\n{item['doc']['text']}")
        
        context_str = "\n\n".join(context_blocks)
        prompt = (
            "You are a helpful assistant. Answer the question STRICTLY using the context below. "
            "Cite sources using [1], [2]. If unknown, say you do not know.\n\n"
            f"Context:\n{context_str}\n\n"
            f"Question: {query}\n"
            "Answer:"
        )
        return prompt

    def query(self, user_query: str) -> Dict[str, Any]:
        retrieved = self.retrieve(user_query, top_k=3)
        if not retrieved or retrieved[0]["score"] < self.confidence_threshold:
            return {
                "answer": "I do not have sufficient documentation to answer this question accurately.",
                "citations": [],
                "retrieved_count": len(retrieved),
                "confidence": 0.0
            }

        prompt = self.synthesize_prompt(user_query, retrieved)
        top_doc = retrieved[0]["doc"]
        simulated_answer = f"According to documentation, {top_doc['text']} [1]."
        citations = [{"source_id": 1, "title": top_doc.get("title", "Doc")}]

        return {
            "answer": simulated_answer,
            "citations": citations,
            "retrieved_count": len(retrieved),
            "confidence": retrieved[0]["score"],
            "prompt_used": prompt
        }

def run_rag_demo():
    rag = EndToEndRAGSystem(confidence_threshold=0.30)
    docs = [
        {"title": "Auth Guide", "text": "API gateway tokens expire after 3600 seconds."},
        {"title": "Limits Guide", "text": "Standard accounts are rate limited to 500 RPM."}
    ]
    rag.ingest_corpus(docs)
    res_success = rag.query("API gateway tokens")
    res_fallback = rag.query("unrelated rocket science")

    print(f"Known Query Answer: {res_success['answer']}")
    print(f"Unknown Query Answer: {res_fallback['answer']}")
    return res_success, res_fallback

if __name__ == "__main__":
    run_rag_demo()
examples/test_e2e_rag_lib.py (1587 bytes)
import pytest
from examples.e2e_rag_lib import (
    EndToEndRAGSystem
)

def test_rag_ingestion_and_retrieval():
    rag = EndToEndRAGSystem()
    docs = [
        {"title": "Doc 1", "text": "PostgreSQL 16 logical replication guide"},
        {"title": "Doc 2", "text": "Redis caching and eviction policies"}
    ]
    rag.ingest_corpus(docs)
    results = rag.retrieve("PostgreSQL replication", top_k=1)
    assert len(results) == 1
    assert results[0]["doc"]["title"] == "Doc 1"
    assert results[0]["score"] > 0.0

def test_high_confidence_generation_with_citation():
    rag = EndToEndRAGSystem(confidence_threshold=0.20)
    docs = [{"title": "SLA", "text": "Uptime guarantee is 99.99%"}]
    rag.ingest_corpus(docs)
    response = rag.query("Uptime guarantee")
    assert "99.99%" in response["answer"]
    assert "[1]" in response["answer"]
    assert len(response["citations"]) == 1

def test_low_confidence_fallback_refusal():
    rag = EndToEndRAGSystem(confidence_threshold=0.50)
    docs = [{"title": "Doc A", "text": "Apples are red"}]
    rag.ingest_corpus(docs)
    response = rag.query("quantum mechanics wave equations")
    assert "I do not have sufficient documentation" in response["answer"]
    assert response["confidence"] == 0.0

def test_prompt_synthesis_formatting():
    rag = EndToEndRAGSystem()
    items = [{"doc": {"title": "Test", "text": "Sample text"}, "score": 1.0}]
    prompt = rag.synthesize_prompt("Test question", items)
    assert "Sample text" in prompt
    assert "[1] Source: Test" in prompt
    assert "Question: Test question" in prompt
metadata.yml (420 bytes)
lesson_id: D266
day: 266
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/e2e_rag_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/e2e_rag_lib.py (831 bytes)
from typing import List, Dict, Any, Optional

class EndToEndRAGSystem:
    def __init__(self, confidence_threshold: float = 0.30):
        self.confidence_threshold = confidence_threshold
        self.documents: List[Dict[str, Any]] = []

    def ingest_corpus(self, docs: List[Dict[str, Any]]):
        # TODO: Store documents for retrieval
        pass

    def retrieve(self, query: str, top_k: int = 3) -> List[Dict[str, Any]]:
        # TODO: Calculate query-document relevance scores
        pass

    def synthesize_prompt(self, query: str, retrieved_items: List[Dict[str, Any]]) -> str:
        # TODO: Format prompt with context passages and citation rules
        pass

    def query(self, user_query: str) -> Dict[str, Any]:
        # TODO: Execute retrieve, check confidence threshold, and generate answer
        pass
starter/test_e2e_rag_lib.py (1587 bytes)
import pytest
from examples.e2e_rag_lib import (
    EndToEndRAGSystem
)

def test_rag_ingestion_and_retrieval():
    rag = EndToEndRAGSystem()
    docs = [
        {"title": "Doc 1", "text": "PostgreSQL 16 logical replication guide"},
        {"title": "Doc 2", "text": "Redis caching and eviction policies"}
    ]
    rag.ingest_corpus(docs)
    results = rag.retrieve("PostgreSQL replication", top_k=1)
    assert len(results) == 1
    assert results[0]["doc"]["title"] == "Doc 1"
    assert results[0]["score"] > 0.0

def test_high_confidence_generation_with_citation():
    rag = EndToEndRAGSystem(confidence_threshold=0.20)
    docs = [{"title": "SLA", "text": "Uptime guarantee is 99.99%"}]
    rag.ingest_corpus(docs)
    response = rag.query("Uptime guarantee")
    assert "99.99%" in response["answer"]
    assert "[1]" in response["answer"]
    assert len(response["citations"]) == 1

def test_low_confidence_fallback_refusal():
    rag = EndToEndRAGSystem(confidence_threshold=0.50)
    docs = [{"title": "Doc A", "text": "Apples are red"}]
    rag.ingest_corpus(docs)
    response = rag.query("quantum mechanics wave equations")
    assert "I do not have sufficient documentation" in response["answer"]
    assert response["confidence"] == 0.0

def test_prompt_synthesis_formatting():
    rag = EndToEndRAGSystem()
    items = [{"doc": {"title": "Test", "text": "Sample text"}, "score": 1.0}]
    prompt = rag.synthesize_prompt("Test question", items)
    assert "Sample text" in prompt
    assert "[1] Source: Test" in prompt
    assert "Question: Test question" in prompt
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 266 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_e2e_rag_lib.py (1587 bytes)
import pytest
from examples.e2e_rag_lib import (
    EndToEndRAGSystem
)

def test_rag_ingestion_and_retrieval():
    rag = EndToEndRAGSystem()
    docs = [
        {"title": "Doc 1", "text": "PostgreSQL 16 logical replication guide"},
        {"title": "Doc 2", "text": "Redis caching and eviction policies"}
    ]
    rag.ingest_corpus(docs)
    results = rag.retrieve("PostgreSQL replication", top_k=1)
    assert len(results) == 1
    assert results[0]["doc"]["title"] == "Doc 1"
    assert results[0]["score"] > 0.0

def test_high_confidence_generation_with_citation():
    rag = EndToEndRAGSystem(confidence_threshold=0.20)
    docs = [{"title": "SLA", "text": "Uptime guarantee is 99.99%"}]
    rag.ingest_corpus(docs)
    response = rag.query("Uptime guarantee")
    assert "99.99%" in response["answer"]
    assert "[1]" in response["answer"]
    assert len(response["citations"]) == 1

def test_low_confidence_fallback_refusal():
    rag = EndToEndRAGSystem(confidence_threshold=0.50)
    docs = [{"title": "Doc A", "text": "Apples are red"}]
    rag.ingest_corpus(docs)
    response = rag.query("quantum mechanics wave equations")
    assert "I do not have sufficient documentation" in response["answer"]
    assert response["confidence"] == 0.0

def test_prompt_synthesis_formatting():
    rag = EndToEndRAGSystem()
    items = [{"doc": {"title": "Test", "text": "Sample text"}, "score": 1.0}]
    prompt = rag.synthesize_prompt("Test question", items)
    assert "Sample text" in prompt
    assert "[1] Source: Test" in prompt
    assert "Question: Test question" in prompt

Troubleshooting

Troubleshooting: Day 266 - End-to-End RAG System

Common Issues

  1. Low confidence on exact matches:
    • Cause: Query words split differences.
    • Fix: Lowercase and strip punctuation before computing word overlap.

Security notes

Security & Privacy: Day 266 - End-to-End RAG System

Security Guidance

  • Restrict prompt context injection to verified document payloads.