LLMs and Generative AI › Embeddings and Vector Search › Day 264
Hands-on lab — Day 264: Hybrid Search and Rerankers
- ← Back to the Day 264 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-264-hybrid-search-and-rerankers/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/hybrid_search_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/hybrid_search_lib.py examples/test_hybrid_search_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/hybrid_search_lib.py starter/test_hybrid_search_lib.py tests/run_tests.sh tests/test_hybrid_search_lib.py troubleshooting.md
Lab README
Lab: Day 264 -- Hybrid Search and Re-Ranking
Lesson
Day number: 264 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: Hybrid Search, BM25, Reciprocal Rank Fusion (RRF), and Re-Ranking.
Purpose
Build and test an End-to-End Hybrid Search & Re-Ranking Pipeline in Python. Implement BM25 lexical keyword scoring, Reciprocal Rank Fusion (RRF) rank combination, and evaluate candidate merging.
Learning objectives
- Implement BM25 term frequency saturation and document length normalization.
- Execute Reciprocal Rank Fusion (RRF) to merge heterogeneous candidate rankings.
- Explain the trade-offs between Bi-Encoders and Cross-Encoders.
- Build a robust two-stage retrieval pipeline.
Prerequisites
- Day 263 (Chunking Strategies).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 512 MB RAM.
- 50 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Pure Python standard math libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/hybrid_search_lib.py: Student scaffold file.examples/hybrid_search_lib.py: Complete reference implementation.tests/test_hybrid_search_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/hybrid_search_lib.py
What the commands do
- Indexes document corpus with BM25 term weights.
- Executes BM25 keyword search.
- Merges candidate lists with RRF rank fusion.
- Runs unit test assertions.
Expected output
Hybrid Demo Executed. Top Match: Error 504 Gateway Timeout on API gateway with RRF score 0.0325
Validation steps
- Verify BM25 ranks exact keyword matches highest.
- Confirm RRF scores combine inverse rank positions.
- Validate candidate deduplication during fusion.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Zero IDF scores: Add the smoothing constant 1.0 to the logarithmic IDF formula.
Security notes
All computations execute locally in memory.
Extension exercises
- Implement linear convex alpha score fusion.
- Integrate a local FlashRank ONNX cross-encoder.
Navigation
- Lesson title: Hybrid Search and Rerankers
- Day number: 264 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-264-hybrid-search-and-rerankers
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-264-hybrid-search-and-rerankerswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 264
- `Hybrid Demo`: Status of hybrid search execution.
examples-run.txt
Hybrid Demo Executed. Top Match: Error 504 Gateway Timeout on API gateway with RRF score 0.0325
measured-values.txt
Hybrid Demo: Success
Top Match: Error 504 Gateway Timeout on API gateway
RRF Score: 0.0325
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 4 items
tests/test_hybrid_search_lib.py::test_bm25_exact_keyword_matching PASSED [ 25%]
tests/test_hybrid_search_lib.py::test_reciprocal_rank_fusion_logic PASSED [ 50%]
tests/test_hybrid_search_lib.py::test_empty_query_bm25 PASSED [ 75%]
tests/test_hybrid_search_lib.py::test_rrf_top_k_bounds PASSED [100%]
============================== 4 passed in 0.29s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/hybrid_search_lib.py (3228 bytes)
import math
from collections import Counter
from typing import List, Dict, Any, Tuple
class HybridSearchEngine:
def __init__(self, rrf_k: int = 60):
self.rrf_k = rrf_k
self.documents: List[Dict[str, Any]] = []
self.doc_term_freqs: List[Counter] = []
self.doc_lengths: List[int] = []
self.avg_doc_len: float = 0.0
self.idf: Dict[str, float] = {}
def index_documents(self, docs: List[Dict[str, Any]]):
self.documents = docs
self.doc_term_freqs = []
self.doc_lengths = []
df: Counter = Counter()
for doc in docs:
tokens = doc["text"].lower().split()
tf = Counter(tokens)
self.doc_term_freqs.append(tf)
self.doc_lengths.append(len(tokens))
for term in tf.keys():
df[term] += 1
n_docs = len(docs)
self.avg_doc_len = sum(self.doc_lengths) / max(1, n_docs)
self.idf = {term: math.log((n_docs - count + 0.5) / (count + 0.5) + 1.0) for term, count in df.items()}
def bm25_search(self, query: str, top_k: int = 10, k1: float = 1.5, b: float = 0.75) -> List[Tuple[int, float]]:
q_tokens = query.lower().split()
scores = []
for idx, tf in enumerate(self.doc_term_freqs):
score = 0.0
d_len = self.doc_lengths[idx]
for term in q_tokens:
if term in tf:
term_idf = self.idf.get(term, 0.0)
freq = tf[term]
numerator = freq * (k1 + 1.0)
denominator = freq + k1 * (1.0 - b + b * (d_len / self.avg_doc_len))
score += term_idf * (numerator / denominator)
scores.append((idx, score))
scores.sort(key=lambda x: x[1], reverse=True)
return scores[:top_k]
def reciprocal_rank_fusion(self, sparse_results: List[Tuple[int, float]], dense_results: List[Tuple[int, float]], top_k: int = 5) -> List[Tuple[Dict[str, Any], float]]:
rrf_scores: Dict[int, float] = Counter()
for rank, (doc_idx, _) in enumerate(sparse_results):
rrf_scores[doc_idx] += 1.0 / (self.rrf_k + (rank + 1))
for rank, (doc_idx, _) in enumerate(dense_results):
rrf_scores[doc_idx] += 1.0 / (self.rrf_k + (rank + 1))
ranked_items = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
return [(self.documents[doc_idx], score) for doc_idx, score in ranked_items[:top_k]]
def run_hybrid_demo():
engine = HybridSearchEngine()
docs = [
{"id": 0, "text": "Error 504 Gateway Timeout on API gateway"},
{"id": 1, "text": "Latency optimization and server sluggish performance tuning"},
{"id": 2, "text": "Database indexing and Postgres connection pool configuration"}
]
engine.index_documents(docs)
sparse_res = engine.bm25_search("error 504 gateway", top_k=2)
dense_res = [(1, 0.95), (0, 0.82)]
hybrid_res = engine.reciprocal_rank_fusion(sparse_res, dense_res, top_k=2)
print(f"Hybrid Demo Executed. Top Match: {hybrid_res[0][0]['text']} with RRF score {hybrid_res[0][1]:.4f}")
return hybrid_res
if __name__ == "__main__":
run_hybrid_demo()
examples/test_hybrid_search_lib.py (1505 bytes)
import pytest
from examples.hybrid_search_lib import (
HybridSearchEngine
)
def test_bm25_exact_keyword_matching():
engine = HybridSearchEngine()
docs = [
{"id": 0, "text": "fatal error 504 gateway timeout"},
{"id": 1, "text": "normal server response ok"},
{"id": 2, "text": "database migration in progress"}
]
engine.index_documents(docs)
results = engine.bm25_search("fatal error", top_k=1)
assert len(results) == 1
assert results[0][0] == 0
assert results[0][1] > 0.0
def test_reciprocal_rank_fusion_logic():
engine = HybridSearchEngine(rrf_k=60)
engine.documents = [{"id": 0}, {"id": 1}, {"id": 2}]
sparse = [(0, 10.5), (1, 5.2)]
dense = [(1, 0.95), (0, 0.85)]
# Doc 0: 1/61 + 1/62 = 0.01639 + 0.01612 = 0.0325
# Doc 1: 1/62 + 1/61 = 0.0325
results = engine.reciprocal_rank_fusion(sparse, dense, top_k=2)
assert len(results) == 2
assert results[0][1] > 0.03
def test_empty_query_bm25():
engine = HybridSearchEngine()
docs = [{"id": 0, "text": "hello world"}]
engine.index_documents(docs)
results = engine.bm25_search("unmatched term", top_k=1)
assert results[0][1] == 0.0
def test_rrf_top_k_bounds():
engine = HybridSearchEngine()
engine.documents = [{"id": i} for i in range(10)]
sparse = [(i, 1.0) for i in range(10)]
dense = [(i, 1.0) for i in range(10)]
results = engine.reciprocal_rank_fusion(sparse, dense, top_k=3)
assert len(results) == 3
metadata.yml (426 bytes)
lesson_id: D264
day: 264
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/hybrid_search_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/hybrid_search_lib.py (797 bytes)
import math
from collections import Counter
from typing import List, Dict, Any, Tuple
class HybridSearchEngine:
def __init__(self, rrf_k: int = 60):
self.rrf_k = rrf_k
self.documents: List[Dict[str, Any]] = []
def index_documents(self, docs: List[Dict[str, Any]]):
# TODO: Compute term frequencies, document lengths, and IDF
pass
def bm25_search(self, query: str, top_k: int = 10, k1: float = 1.5, b: float = 0.75) -> List[Tuple[int, float]]:
# TODO: Calculate BM25 scores
pass
def reciprocal_rank_fusion(self, sparse_results: List[Tuple[int, float]], dense_results: List[Tuple[int, float]], top_k: int = 5) -> List[Tuple[Dict[str, Any], float]]:
# TODO: Merge rankings using RRF formulation: 1 / (k + rank)
pass
starter/test_hybrid_search_lib.py (1505 bytes)
import pytest
from examples.hybrid_search_lib import (
HybridSearchEngine
)
def test_bm25_exact_keyword_matching():
engine = HybridSearchEngine()
docs = [
{"id": 0, "text": "fatal error 504 gateway timeout"},
{"id": 1, "text": "normal server response ok"},
{"id": 2, "text": "database migration in progress"}
]
engine.index_documents(docs)
results = engine.bm25_search("fatal error", top_k=1)
assert len(results) == 1
assert results[0][0] == 0
assert results[0][1] > 0.0
def test_reciprocal_rank_fusion_logic():
engine = HybridSearchEngine(rrf_k=60)
engine.documents = [{"id": 0}, {"id": 1}, {"id": 2}]
sparse = [(0, 10.5), (1, 5.2)]
dense = [(1, 0.95), (0, 0.85)]
# Doc 0: 1/61 + 1/62 = 0.01639 + 0.01612 = 0.0325
# Doc 1: 1/62 + 1/61 = 0.0325
results = engine.reciprocal_rank_fusion(sparse, dense, top_k=2)
assert len(results) == 2
assert results[0][1] > 0.03
def test_empty_query_bm25():
engine = HybridSearchEngine()
docs = [{"id": 0, "text": "hello world"}]
engine.index_documents(docs)
results = engine.bm25_search("unmatched term", top_k=1)
assert results[0][1] == 0.0
def test_rrf_top_k_bounds():
engine = HybridSearchEngine()
engine.documents = [{"id": i} for i in range(10)]
sparse = [(i, 1.0) for i in range(10)]
dense = [(i, 1.0) for i in range(10)]
results = engine.reciprocal_rank_fusion(sparse, dense, top_k=3)
assert len(results) == 3
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 264 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_hybrid_search_lib.py (1505 bytes)
import pytest
from examples.hybrid_search_lib import (
HybridSearchEngine
)
def test_bm25_exact_keyword_matching():
engine = HybridSearchEngine()
docs = [
{"id": 0, "text": "fatal error 504 gateway timeout"},
{"id": 1, "text": "normal server response ok"},
{"id": 2, "text": "database migration in progress"}
]
engine.index_documents(docs)
results = engine.bm25_search("fatal error", top_k=1)
assert len(results) == 1
assert results[0][0] == 0
assert results[0][1] > 0.0
def test_reciprocal_rank_fusion_logic():
engine = HybridSearchEngine(rrf_k=60)
engine.documents = [{"id": 0}, {"id": 1}, {"id": 2}]
sparse = [(0, 10.5), (1, 5.2)]
dense = [(1, 0.95), (0, 0.85)]
# Doc 0: 1/61 + 1/62 = 0.01639 + 0.01612 = 0.0325
# Doc 1: 1/62 + 1/61 = 0.0325
results = engine.reciprocal_rank_fusion(sparse, dense, top_k=2)
assert len(results) == 2
assert results[0][1] > 0.03
def test_empty_query_bm25():
engine = HybridSearchEngine()
docs = [{"id": 0, "text": "hello world"}]
engine.index_documents(docs)
results = engine.bm25_search("unmatched term", top_k=1)
assert results[0][1] == 0.0
def test_rrf_top_k_bounds():
engine = HybridSearchEngine()
engine.documents = [{"id": i} for i in range(10)]
sparse = [(i, 1.0) for i in range(10)]
dense = [(i, 1.0) for i in range(10)]
results = engine.reciprocal_rank_fusion(sparse, dense, top_k=3)
assert len(results) == 3
Troubleshooting
Troubleshooting: Day 264 - Hybrid Search and Re-Ranking
Common Issues
- Division by zero during average doc length calculation:
- Cause: Empty document list indexed.
- Fix: Use max(1, n_docs) in divisor.
Security notes
Security & Privacy: Day 264 - Hybrid Search and Re-Ranking
Security Guidance
- Filter sensitive documents before re-ranking to prevent prompt leakage.