LLMs and Generative AI › Embeddings and Vector Search › Day 266
Hands-on lab — Day 266: Semantic Search over Your Own Notes
- ← Back to the Day 266 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-266-semantic-search-over-your-own-notes/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/e2e_rag_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/e2e_rag_lib.py examples/test_e2e_rag_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/e2e_rag_lib.py starter/test_e2e_rag_lib.py tests/run_tests.sh tests/test_e2e_rag_lib.py troubleshooting.md
Lab README
Lab: Day 266 -- End-to-End RAG System
Lesson
Day number: 266 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: End-to-End RAG System Architecture, Ingestion, Generation, and Citations.
Purpose
Build and test a complete Modular End-to-End RAG System in Python. Implement document ingestion, candidate retrieval, confidence threshold fallback gating, citation-enforced prompt synthesis, and grounded answer generation.
Learning objectives
- Synthesize ingestion, retrieval, re-ranking, and prompt synthesis into a single unified pipeline.
- Implement confidence threshold gating to prevent hallucinations.
- Enforce source citations and claim verification.
- Structure modular, production-ready Python RAG architectures.
Prerequisites
- Day 265 (Evaluating RAG with RAGAS and TruLens).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 512 MB RAM.
- 50 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Pure Python standard libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/e2e_rag_lib.py: Student scaffold file.examples/e2e_rag_lib.py: Complete reference implementation.tests/test_e2e_rag_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/e2e_rag_lib.py
What the commands do
- Ingests documentation corpus.
- Executes query retrieval and prompt synthesis.
- Generates answer with verified citations.
- Triggers fallback refusal for unknown queries.
- Runs unit test assertions.
Expected output
Known Query Answer: According to documentation, API gateway tokens expire after 3600 seconds. [1].
Unknown Query Answer: I do not have sufficient documentation to answer this question accurately.
Validation steps
- Verify high-confidence queries return grounded answers with
[1]citation tags. - Confirm unknown queries trigger graceful refusal messages.
- Validate prompt synthesis correctly formats source metadata blocks.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Refusal on valid queries: Lower the
confidence_thresholdparameter (e.g. from 0.50 to 0.20).
Security notes
All data is processed in-memory within local Python runtime.
Extension exercises
- Implement OpenTelemetry distributed tracing spans.
- Add multi-turn conversation history buffer.
Navigation
- Lesson title: Semantic Search over Your Own Notes
- Day number: 266 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-266-semantic-search-over-your-own-notes
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-266-semantic-search-over-your-own-noteswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 266
- `Known Query Answer`: Grounded response with citation.
- `Unknown Query Answer`: Fallback message.
examples-run.txt
Known Query Answer: According to documentation, API gateway tokens expire after 3600 seconds. [1].
Unknown Query Answer: I do not have sufficient documentation to answer this question accurately.
measured-values.txt
Known Query Answer: According to documentation, API gateway tokens expire after 3600 seconds. [1].
Unknown Query Answer: I do not have sufficient documentation to answer this question accurately.
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 4 items
tests/test_e2e_rag_lib.py::test_rag_ingestion_and_retrieval PASSED [ 25%]
tests/test_e2e_rag_lib.py::test_high_confidence_generation_with_citation PASSED [ 50%]
tests/test_e2e_rag_lib.py::test_low_confidence_fallback_refusal PASSED [ 75%]
tests/test_e2e_rag_lib.py::test_prompt_synthesis_formatting PASSED [100%]
============================== 4 passed in 0.28s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/e2e_rag_lib.py (3081 bytes)
from typing import List, Dict, Any, Optional
class EndToEndRAGSystem:
def __init__(self, confidence_threshold: float = 0.30):
self.confidence_threshold = confidence_threshold
self.documents: List[Dict[str, Any]] = []
def ingest_corpus(self, docs: List[Dict[str, Any]]):
self.documents = docs
def retrieve(self, query: str, top_k: int = 3) -> List[Dict[str, Any]]:
q_words = set(query.lower().split())
scored = []
for doc in self.documents:
d_words = set(doc["text"].lower().split())
overlap = len(q_words.intersection(d_words))
score = overlap / max(1, len(q_words))
scored.append((doc, score))
scored.sort(key=lambda x: x[1], reverse=True)
return [{"doc": doc, "score": score} for doc, score in scored[:top_k]]
def synthesize_prompt(self, query: str, retrieved_items: List[Dict[str, Any]]) -> str:
context_blocks = []
for idx, item in enumerate(retrieved_items, 1):
context_blocks.append(f"[{idx}] Source: {item['doc'].get('title', 'Doc')}\n{item['doc']['text']}")
context_str = "\n\n".join(context_blocks)
prompt = (
"You are a helpful assistant. Answer the question STRICTLY using the context below. "
"Cite sources using [1], [2]. If unknown, say you do not know.\n\n"
f"Context:\n{context_str}\n\n"
f"Question: {query}\n"
"Answer:"
)
return prompt
def query(self, user_query: str) -> Dict[str, Any]:
retrieved = self.retrieve(user_query, top_k=3)
if not retrieved or retrieved[0]["score"] < self.confidence_threshold:
return {
"answer": "I do not have sufficient documentation to answer this question accurately.",
"citations": [],
"retrieved_count": len(retrieved),
"confidence": 0.0
}
prompt = self.synthesize_prompt(user_query, retrieved)
top_doc = retrieved[0]["doc"]
simulated_answer = f"According to documentation, {top_doc['text']} [1]."
citations = [{"source_id": 1, "title": top_doc.get("title", "Doc")}]
return {
"answer": simulated_answer,
"citations": citations,
"retrieved_count": len(retrieved),
"confidence": retrieved[0]["score"],
"prompt_used": prompt
}
def run_rag_demo():
rag = EndToEndRAGSystem(confidence_threshold=0.30)
docs = [
{"title": "Auth Guide", "text": "API gateway tokens expire after 3600 seconds."},
{"title": "Limits Guide", "text": "Standard accounts are rate limited to 500 RPM."}
]
rag.ingest_corpus(docs)
res_success = rag.query("API gateway tokens")
res_fallback = rag.query("unrelated rocket science")
print(f"Known Query Answer: {res_success['answer']}")
print(f"Unknown Query Answer: {res_fallback['answer']}")
return res_success, res_fallback
if __name__ == "__main__":
run_rag_demo()
examples/test_e2e_rag_lib.py (1587 bytes)
import pytest
from examples.e2e_rag_lib import (
EndToEndRAGSystem
)
def test_rag_ingestion_and_retrieval():
rag = EndToEndRAGSystem()
docs = [
{"title": "Doc 1", "text": "PostgreSQL 16 logical replication guide"},
{"title": "Doc 2", "text": "Redis caching and eviction policies"}
]
rag.ingest_corpus(docs)
results = rag.retrieve("PostgreSQL replication", top_k=1)
assert len(results) == 1
assert results[0]["doc"]["title"] == "Doc 1"
assert results[0]["score"] > 0.0
def test_high_confidence_generation_with_citation():
rag = EndToEndRAGSystem(confidence_threshold=0.20)
docs = [{"title": "SLA", "text": "Uptime guarantee is 99.99%"}]
rag.ingest_corpus(docs)
response = rag.query("Uptime guarantee")
assert "99.99%" in response["answer"]
assert "[1]" in response["answer"]
assert len(response["citations"]) == 1
def test_low_confidence_fallback_refusal():
rag = EndToEndRAGSystem(confidence_threshold=0.50)
docs = [{"title": "Doc A", "text": "Apples are red"}]
rag.ingest_corpus(docs)
response = rag.query("quantum mechanics wave equations")
assert "I do not have sufficient documentation" in response["answer"]
assert response["confidence"] == 0.0
def test_prompt_synthesis_formatting():
rag = EndToEndRAGSystem()
items = [{"doc": {"title": "Test", "text": "Sample text"}, "score": 1.0}]
prompt = rag.synthesize_prompt("Test question", items)
assert "Sample text" in prompt
assert "[1] Source: Test" in prompt
assert "Question: Test question" in prompt
metadata.yml (420 bytes)
lesson_id: D266
day: 266
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/e2e_rag_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/e2e_rag_lib.py (831 bytes)
from typing import List, Dict, Any, Optional
class EndToEndRAGSystem:
def __init__(self, confidence_threshold: float = 0.30):
self.confidence_threshold = confidence_threshold
self.documents: List[Dict[str, Any]] = []
def ingest_corpus(self, docs: List[Dict[str, Any]]):
# TODO: Store documents for retrieval
pass
def retrieve(self, query: str, top_k: int = 3) -> List[Dict[str, Any]]:
# TODO: Calculate query-document relevance scores
pass
def synthesize_prompt(self, query: str, retrieved_items: List[Dict[str, Any]]) -> str:
# TODO: Format prompt with context passages and citation rules
pass
def query(self, user_query: str) -> Dict[str, Any]:
# TODO: Execute retrieve, check confidence threshold, and generate answer
pass
starter/test_e2e_rag_lib.py (1587 bytes)
import pytest
from examples.e2e_rag_lib import (
EndToEndRAGSystem
)
def test_rag_ingestion_and_retrieval():
rag = EndToEndRAGSystem()
docs = [
{"title": "Doc 1", "text": "PostgreSQL 16 logical replication guide"},
{"title": "Doc 2", "text": "Redis caching and eviction policies"}
]
rag.ingest_corpus(docs)
results = rag.retrieve("PostgreSQL replication", top_k=1)
assert len(results) == 1
assert results[0]["doc"]["title"] == "Doc 1"
assert results[0]["score"] > 0.0
def test_high_confidence_generation_with_citation():
rag = EndToEndRAGSystem(confidence_threshold=0.20)
docs = [{"title": "SLA", "text": "Uptime guarantee is 99.99%"}]
rag.ingest_corpus(docs)
response = rag.query("Uptime guarantee")
assert "99.99%" in response["answer"]
assert "[1]" in response["answer"]
assert len(response["citations"]) == 1
def test_low_confidence_fallback_refusal():
rag = EndToEndRAGSystem(confidence_threshold=0.50)
docs = [{"title": "Doc A", "text": "Apples are red"}]
rag.ingest_corpus(docs)
response = rag.query("quantum mechanics wave equations")
assert "I do not have sufficient documentation" in response["answer"]
assert response["confidence"] == 0.0
def test_prompt_synthesis_formatting():
rag = EndToEndRAGSystem()
items = [{"doc": {"title": "Test", "text": "Sample text"}, "score": 1.0}]
prompt = rag.synthesize_prompt("Test question", items)
assert "Sample text" in prompt
assert "[1] Source: Test" in prompt
assert "Question: Test question" in prompt
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 266 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_e2e_rag_lib.py (1587 bytes)
import pytest
from examples.e2e_rag_lib import (
EndToEndRAGSystem
)
def test_rag_ingestion_and_retrieval():
rag = EndToEndRAGSystem()
docs = [
{"title": "Doc 1", "text": "PostgreSQL 16 logical replication guide"},
{"title": "Doc 2", "text": "Redis caching and eviction policies"}
]
rag.ingest_corpus(docs)
results = rag.retrieve("PostgreSQL replication", top_k=1)
assert len(results) == 1
assert results[0]["doc"]["title"] == "Doc 1"
assert results[0]["score"] > 0.0
def test_high_confidence_generation_with_citation():
rag = EndToEndRAGSystem(confidence_threshold=0.20)
docs = [{"title": "SLA", "text": "Uptime guarantee is 99.99%"}]
rag.ingest_corpus(docs)
response = rag.query("Uptime guarantee")
assert "99.99%" in response["answer"]
assert "[1]" in response["answer"]
assert len(response["citations"]) == 1
def test_low_confidence_fallback_refusal():
rag = EndToEndRAGSystem(confidence_threshold=0.50)
docs = [{"title": "Doc A", "text": "Apples are red"}]
rag.ingest_corpus(docs)
response = rag.query("quantum mechanics wave equations")
assert "I do not have sufficient documentation" in response["answer"]
assert response["confidence"] == 0.0
def test_prompt_synthesis_formatting():
rag = EndToEndRAGSystem()
items = [{"doc": {"title": "Test", "text": "Sample text"}, "score": 1.0}]
prompt = rag.synthesize_prompt("Test question", items)
assert "Sample text" in prompt
assert "[1] Source: Test" in prompt
assert "Question: Test question" in prompt
Troubleshooting
Troubleshooting: Day 266 - End-to-End RAG System
Common Issues
- Low confidence on exact matches:
- Cause: Query words split differences.
- Fix: Lowercase and strip punctuation before computing word overlap.
Security notes
Security & Privacy: Day 266 - End-to-End RAG System
Security Guidance
- Restrict prompt context injection to verified document payloads.