LLMs and Generative AI › Embeddings and Vector Search › Day 264

Hands-on lab — Day 264: Hybrid Search and Rerankers

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/hybrid_search_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/hybrid_search_lib.py
examples/test_hybrid_search_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/hybrid_search_lib.py
starter/test_hybrid_search_lib.py
tests/run_tests.sh
tests/test_hybrid_search_lib.py
troubleshooting.md

Lab README

Lab: Day 264 -- Hybrid Search and Re-Ranking

Lesson

Day number: 264 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: Hybrid Search, BM25, Reciprocal Rank Fusion (RRF), and Re-Ranking.

Purpose

Build and test an End-to-End Hybrid Search & Re-Ranking Pipeline in Python. Implement BM25 lexical keyword scoring, Reciprocal Rank Fusion (RRF) rank combination, and evaluate candidate merging.

Learning objectives

  • Implement BM25 term frequency saturation and document length normalization.
  • Execute Reciprocal Rank Fusion (RRF) to merge heterogeneous candidate rankings.
  • Explain the trade-offs between Bi-Encoders and Cross-Encoders.
  • Build a robust two-stage retrieval pipeline.

Prerequisites

  • Day 263 (Chunking Strategies).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Pure Python standard math libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/hybrid_search_lib.py: Student scaffold file.
  • examples/hybrid_search_lib.py: Complete reference implementation.
  • tests/test_hybrid_search_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/hybrid_search_lib.py

What the commands do

  • Indexes document corpus with BM25 term weights.
  • Executes BM25 keyword search.
  • Merges candidate lists with RRF rank fusion.
  • Runs unit test assertions.

Expected output

Hybrid Demo Executed. Top Match: Error 504 Gateway Timeout on API gateway with RRF score 0.0325

Validation steps

  1. Verify BM25 ranks exact keyword matches highest.
  2. Confirm RRF scores combine inverse rank positions.
  3. Validate candidate deduplication during fusion.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Zero IDF scores: Add the smoothing constant 1.0 to the logarithmic IDF formula.

Security notes

All computations execute locally in memory.

Extension exercises

  1. Implement linear convex alpha score fusion.
  2. Integrate a local FlashRank ONNX cross-encoder.

Expected output

FIELDS.md

# Expected Output Fields: Day 264

- `Hybrid Demo`: Status of hybrid search execution.

examples-run.txt

Hybrid Demo Executed. Top Match: Error 504 Gateway Timeout on API gateway with RRF score 0.0325

measured-values.txt

Hybrid Demo: Success
Top Match: Error 504 Gateway Timeout on API gateway
RRF Score: 0.0325

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_hybrid_search_lib.py::test_bm25_exact_keyword_matching PASSED        [ 25%]
tests/test_hybrid_search_lib.py::test_reciprocal_rank_fusion_logic PASSED       [ 50%]
tests/test_hybrid_search_lib.py::test_empty_query_bm25 PASSED                  [ 75%]
tests/test_hybrid_search_lib.py::test_rrf_top_k_bounds PASSED                  [100%]

============================== 4 passed in 0.29s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/hybrid_search_lib.py (3228 bytes)
import math
from collections import Counter
from typing import List, Dict, Any, Tuple

class HybridSearchEngine:
    def __init__(self, rrf_k: int = 60):
        self.rrf_k = rrf_k
        self.documents: List[Dict[str, Any]] = []
        self.doc_term_freqs: List[Counter] = []
        self.doc_lengths: List[int] = []
        self.avg_doc_len: float = 0.0
        self.idf: Dict[str, float] = {}

    def index_documents(self, docs: List[Dict[str, Any]]):
        self.documents = docs
        self.doc_term_freqs = []
        self.doc_lengths = []
        df: Counter = Counter()

        for doc in docs:
            tokens = doc["text"].lower().split()
            tf = Counter(tokens)
            self.doc_term_freqs.append(tf)
            self.doc_lengths.append(len(tokens))
            for term in tf.keys():
                df[term] += 1

        n_docs = len(docs)
        self.avg_doc_len = sum(self.doc_lengths) / max(1, n_docs)
        self.idf = {term: math.log((n_docs - count + 0.5) / (count + 0.5) + 1.0) for term, count in df.items()}

    def bm25_search(self, query: str, top_k: int = 10, k1: float = 1.5, b: float = 0.75) -> List[Tuple[int, float]]:
        q_tokens = query.lower().split()
        scores = []

        for idx, tf in enumerate(self.doc_term_freqs):
            score = 0.0
            d_len = self.doc_lengths[idx]
            for term in q_tokens:
                if term in tf:
                    term_idf = self.idf.get(term, 0.0)
                    freq = tf[term]
                    numerator = freq * (k1 + 1.0)
                    denominator = freq + k1 * (1.0 - b + b * (d_len / self.avg_doc_len))
                    score += term_idf * (numerator / denominator)
            scores.append((idx, score))

        scores.sort(key=lambda x: x[1], reverse=True)
        return scores[:top_k]

    def reciprocal_rank_fusion(self, sparse_results: List[Tuple[int, float]], dense_results: List[Tuple[int, float]], top_k: int = 5) -> List[Tuple[Dict[str, Any], float]]:
        rrf_scores: Dict[int, float] = Counter()

        for rank, (doc_idx, _) in enumerate(sparse_results):
            rrf_scores[doc_idx] += 1.0 / (self.rrf_k + (rank + 1))

        for rank, (doc_idx, _) in enumerate(dense_results):
            rrf_scores[doc_idx] += 1.0 / (self.rrf_k + (rank + 1))

        ranked_items = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
        return [(self.documents[doc_idx], score) for doc_idx, score in ranked_items[:top_k]]

def run_hybrid_demo():
    engine = HybridSearchEngine()
    docs = [
        {"id": 0, "text": "Error 504 Gateway Timeout on API gateway"},
        {"id": 1, "text": "Latency optimization and server sluggish performance tuning"},
        {"id": 2, "text": "Database indexing and Postgres connection pool configuration"}
    ]
    engine.index_documents(docs)
    sparse_res = engine.bm25_search("error 504 gateway", top_k=2)
    dense_res = [(1, 0.95), (0, 0.82)]
    hybrid_res = engine.reciprocal_rank_fusion(sparse_res, dense_res, top_k=2)
    print(f"Hybrid Demo Executed. Top Match: {hybrid_res[0][0]['text']} with RRF score {hybrid_res[0][1]:.4f}")
    return hybrid_res

if __name__ == "__main__":
    run_hybrid_demo()
examples/test_hybrid_search_lib.py (1505 bytes)
import pytest
from examples.hybrid_search_lib import (
    HybridSearchEngine
)

def test_bm25_exact_keyword_matching():
    engine = HybridSearchEngine()
    docs = [
        {"id": 0, "text": "fatal error 504 gateway timeout"},
        {"id": 1, "text": "normal server response ok"},
        {"id": 2, "text": "database migration in progress"}
    ]
    engine.index_documents(docs)
    results = engine.bm25_search("fatal error", top_k=1)
    
    assert len(results) == 1
    assert results[0][0] == 0
    assert results[0][1] > 0.0

def test_reciprocal_rank_fusion_logic():
    engine = HybridSearchEngine(rrf_k=60)
    engine.documents = [{"id": 0}, {"id": 1}, {"id": 2}]
    sparse = [(0, 10.5), (1, 5.2)]
    dense = [(1, 0.95), (0, 0.85)]
    
    # Doc 0: 1/61 + 1/62 = 0.01639 + 0.01612 = 0.0325
    # Doc 1: 1/62 + 1/61 = 0.0325
    results = engine.reciprocal_rank_fusion(sparse, dense, top_k=2)
    assert len(results) == 2
    assert results[0][1] > 0.03

def test_empty_query_bm25():
    engine = HybridSearchEngine()
    docs = [{"id": 0, "text": "hello world"}]
    engine.index_documents(docs)
    results = engine.bm25_search("unmatched term", top_k=1)
    assert results[0][1] == 0.0

def test_rrf_top_k_bounds():
    engine = HybridSearchEngine()
    engine.documents = [{"id": i} for i in range(10)]
    sparse = [(i, 1.0) for i in range(10)]
    dense = [(i, 1.0) for i in range(10)]
    results = engine.reciprocal_rank_fusion(sparse, dense, top_k=3)
    assert len(results) == 3
metadata.yml (426 bytes)
lesson_id: D264
day: 264
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/hybrid_search_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/hybrid_search_lib.py (797 bytes)
import math
from collections import Counter
from typing import List, Dict, Any, Tuple

class HybridSearchEngine:
    def __init__(self, rrf_k: int = 60):
        self.rrf_k = rrf_k
        self.documents: List[Dict[str, Any]] = []

    def index_documents(self, docs: List[Dict[str, Any]]):
        # TODO: Compute term frequencies, document lengths, and IDF
        pass

    def bm25_search(self, query: str, top_k: int = 10, k1: float = 1.5, b: float = 0.75) -> List[Tuple[int, float]]:
        # TODO: Calculate BM25 scores
        pass

    def reciprocal_rank_fusion(self, sparse_results: List[Tuple[int, float]], dense_results: List[Tuple[int, float]], top_k: int = 5) -> List[Tuple[Dict[str, Any], float]]:
        # TODO: Merge rankings using RRF formulation: 1 / (k + rank)
        pass
starter/test_hybrid_search_lib.py (1505 bytes)
import pytest
from examples.hybrid_search_lib import (
    HybridSearchEngine
)

def test_bm25_exact_keyword_matching():
    engine = HybridSearchEngine()
    docs = [
        {"id": 0, "text": "fatal error 504 gateway timeout"},
        {"id": 1, "text": "normal server response ok"},
        {"id": 2, "text": "database migration in progress"}
    ]
    engine.index_documents(docs)
    results = engine.bm25_search("fatal error", top_k=1)
    
    assert len(results) == 1
    assert results[0][0] == 0
    assert results[0][1] > 0.0

def test_reciprocal_rank_fusion_logic():
    engine = HybridSearchEngine(rrf_k=60)
    engine.documents = [{"id": 0}, {"id": 1}, {"id": 2}]
    sparse = [(0, 10.5), (1, 5.2)]
    dense = [(1, 0.95), (0, 0.85)]
    
    # Doc 0: 1/61 + 1/62 = 0.01639 + 0.01612 = 0.0325
    # Doc 1: 1/62 + 1/61 = 0.0325
    results = engine.reciprocal_rank_fusion(sparse, dense, top_k=2)
    assert len(results) == 2
    assert results[0][1] > 0.03

def test_empty_query_bm25():
    engine = HybridSearchEngine()
    docs = [{"id": 0, "text": "hello world"}]
    engine.index_documents(docs)
    results = engine.bm25_search("unmatched term", top_k=1)
    assert results[0][1] == 0.0

def test_rrf_top_k_bounds():
    engine = HybridSearchEngine()
    engine.documents = [{"id": i} for i in range(10)]
    sparse = [(i, 1.0) for i in range(10)]
    dense = [(i, 1.0) for i in range(10)]
    results = engine.reciprocal_rank_fusion(sparse, dense, top_k=3)
    assert len(results) == 3
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 264 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_hybrid_search_lib.py (1505 bytes)
import pytest
from examples.hybrid_search_lib import (
    HybridSearchEngine
)

def test_bm25_exact_keyword_matching():
    engine = HybridSearchEngine()
    docs = [
        {"id": 0, "text": "fatal error 504 gateway timeout"},
        {"id": 1, "text": "normal server response ok"},
        {"id": 2, "text": "database migration in progress"}
    ]
    engine.index_documents(docs)
    results = engine.bm25_search("fatal error", top_k=1)
    
    assert len(results) == 1
    assert results[0][0] == 0
    assert results[0][1] > 0.0

def test_reciprocal_rank_fusion_logic():
    engine = HybridSearchEngine(rrf_k=60)
    engine.documents = [{"id": 0}, {"id": 1}, {"id": 2}]
    sparse = [(0, 10.5), (1, 5.2)]
    dense = [(1, 0.95), (0, 0.85)]
    
    # Doc 0: 1/61 + 1/62 = 0.01639 + 0.01612 = 0.0325
    # Doc 1: 1/62 + 1/61 = 0.0325
    results = engine.reciprocal_rank_fusion(sparse, dense, top_k=2)
    assert len(results) == 2
    assert results[0][1] > 0.03

def test_empty_query_bm25():
    engine = HybridSearchEngine()
    docs = [{"id": 0, "text": "hello world"}]
    engine.index_documents(docs)
    results = engine.bm25_search("unmatched term", top_k=1)
    assert results[0][1] == 0.0

def test_rrf_top_k_bounds():
    engine = HybridSearchEngine()
    engine.documents = [{"id": i} for i in range(10)]
    sparse = [(i, 1.0) for i in range(10)]
    dense = [(i, 1.0) for i in range(10)]
    results = engine.reciprocal_rank_fusion(sparse, dense, top_k=3)
    assert len(results) == 3

Troubleshooting

Troubleshooting: Day 264 - Hybrid Search and Re-Ranking

Common Issues

  1. Division by zero during average doc length calculation:
    • Cause: Empty document list indexed.
    • Fix: Use max(1, n_docs) in divisor.

Security notes

Security & Privacy: Day 264 - Hybrid Search and Re-Ranking

Security Guidance

  • Filter sensitive documents before re-ranking to prevent prompt leakage.