LLMs and Generative AI › Retrieval-Augmented Generation › Day 270

Hands-on lab — Day 270: Citations and Grounded Answers

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/grounded_citations_lib.py

Test

./tests/run_tests.sh

File tree

examples/grounded_citations_lib.py
examples/test_grounded_citations_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/grounded_citations_lib.py
starter/test_grounded_citations_lib.py
tests/run_tests.sh
tests/test_grounded_citations_lib.py
troubleshooting.md

Lab README

Lab: Day 270 -- Citations and Grounded Answers

Lesson

Day number: 270 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: Sentence-Level Citation Grounding and Claim Attribution.

Purpose

Build and test a modular Python RAG implementation. Implement document chunking, sparse BM25 indexing, dense vector search, Reciprocal Rank Fusion, grounded prompt synthesis, and inline citation attribution.

Learning objectives

  • Implement sliding-window document chunking preserving document metadata.
  • Build an in-memory Okapi BM25 inverted index with token frequency and IDF weighting.
  • Implement Reciprocal Rank Fusion (RRF) for hybrid retrieval candidate scoring.
  • Synthesize prompt context blocks enforcing verifiable inline bracket citations.
  • Validate RAG grounding and retrieval precision with automated unit tests.

Prerequisites

  • Day 269 (RAG over PDFs and Messy Documents).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 512 MB RAM.
  • 50 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Pure Python standard libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/grounded_citations_lib.py: Student scaffold file.
  • examples/grounded_citations_lib.py: Complete reference implementation.
  • tests/test_grounded_citations_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/grounded_citations_lib.py

What the commands do

  • Chunks sample documents into token overlapping windows.
  • Indexes chunks in BM25 inverted index.
  • Executes keyword query retrieval.
  • Formats grounded system prompt with numbered citation tags.
  • Runs Pytest unit validation suite.

Expected output

Ingested 1 chunks.
Top Result Score: 1.2500
Synthesized Prompt Length: 390 characters.

Validation steps

  1. Verify document chunking preserves metadata and title strings.
  2. Confirm BM25 search scores target documents highest for matching keywords.
  3. Validate Reciprocal Rank Fusion correctly prioritizes documents appearing in multiple candidate lists.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Zero scores on BM25: Verify queries are tokenized using regex word boundaries (re.findall(r'\w+', query.lower())).

Security notes

All data is processed in-memory within local Python runtime.

Extension exercises

  1. Implement in-memory cosine vector similarity search.
  2. Add support for PDF extraction using pdfplumber.

Expected output

FIELDS.md

# Expected Output Fields: Day 270

- `Ingested chunks`: Total number of chunks indexed.
- `Top Result Score`: BM25 score of highest ranked document.
- `Synthesized Prompt Length`: Total character length of generated prompt.

examples-run.txt

Ingested 1 chunks.
Top Result Score: 0.9808
Synthesized Prompt Length: 391 characters.

measured-values.txt

metric_name=bm25_top_score, value=0.9808
metric_name=chunk_count, value=1
metric_name=prompt_length, value=391

run-output.txt

Ingested 1 chunks.
Top Result Score: 0.6987
Synthesized Prompt Length: 550 characters.

starter-run.txt

Running starter scaffold...

test-output.txt

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_grounded_citations_lib.py::test_document_chunker PASSED                         [ 25%]
tests/test_grounded_citations_lib.py::test_sparse_bm25 PASSED                              [ 50%]
tests/test_grounded_citations_lib.py::test_hybrid_ranker PASSED                            [ 75%]
tests/test_grounded_citations_lib.py::test_prompt_synthesizer PASSED                       [100%]

============================== 4 passed in 0.05s ===============================

Source files

examples/grounded_citations_lib.py (4756 bytes)
import math
import re
from collections import Counter
from typing import List, Dict, Any, Tuple

class DocumentChunker:
    def __init__(self, chunk_size: int = 150, overlap: int = 30):
        self.chunk_size = chunk_size
        self.overlap = overlap

    def chunk_document(self, doc_id: str, title: str, text: str) -> List[Dict[str, Any]]:
        words = text.split()
        chunks = []
        step = max(1, self.chunk_size - self.overlap)
        for i in range(0, len(words), step):
            chunk_words = words[i:i + self.chunk_size]
            if not chunk_words:
                break
            chunks.append({
                "chunk_id": f"{doc_id}#chunk_{len(chunks)}",
                "doc_id": doc_id,
                "title": title,
                "text": " ".join(chunk_words),
                "start_word": i,
                "end_word": i + len(chunk_words)
            })
            if i + self.chunk_size >= len(words):
                break
        return chunks

class SparseBM25:
    def __init__(self, k1: float = 1.5, b: float = 0.75):
        self.k1 = k1
        self.b = b
        self.chunks: List[Dict[str, Any]] = []
        self.doc_freqs: List[Counter] = []
        self.doc_lengths: List[int] = []
        self.avg_dl: float = 0.0
        self.idf: Dict[str, float] = {}

    def index(self, chunks: List[Dict[str, Any]]):
        self.chunks = chunks
        self.doc_freqs = []
        self.doc_lengths = []
        df_counter = Counter()

        for chunk in chunks:
            tokens = re.findall(r'\w+', chunk["text"].lower())
            tf = Counter(tokens)
            self.doc_freqs.append(tf)
            self.doc_lengths.append(len(tokens))
            for term in tf.keys():
                df_counter[term] += 1

        n_docs = len(chunks)
        self.avg_dl = sum(self.doc_lengths) / max(1, n_docs)
        self.idf = {
            term: math.log((n_docs - freq + 0.5) / (freq + 0.5) + 1.0)
            for term, freq in df_counter.items()
        }

    def search(self, query: str, top_k: int = 10) -> List[Tuple[int, float]]:
        q_tokens = re.findall(r'\w+', query.lower())
        scores = []

        for idx, tf in enumerate(self.doc_freqs):
            dl = self.doc_lengths[idx]
            score = 0.0
            for term in q_tokens:
                if term in tf:
                    term_idf = self.idf.get(term, 0.0)
                    freq = tf[term]
                    numerator = freq * (self.k1 + 1.0)
                    denominator = freq + self.k1 * (1.0 - self.b + self.b * (dl / max(1.0, self.avg_dl)))
                    score += term_idf * (numerator / denominator)
            scores.append((idx, score))

        scores.sort(key=lambda x: x[1], reverse=True)
        return scores[:top_k]

class HybridRanker:
    def __init__(self, rrf_k: int = 60):
        self.rrf_k = rrf_k

    def fuse_ranks(self, list_a: List[int], list_b: List[int]) -> List[Tuple[int, float]]:
        scores = Counter()
        for rank, doc_idx in enumerate(list_a, 1):
            scores[doc_idx] += 1.0 / (self.rrf_k + rank)
        for rank, doc_idx in enumerate(list_b, 1):
            scores[doc_idx] += 1.0 / (self.rrf_k + rank)
        return sorted(scores.items(), key=lambda x: x[1], reverse=True)

class PromptSynthesizer:
    @staticmethod
    def format_context(query: str, chunks: List[Dict[str, Any]]) -> str:
        blocks = []
        for i, c in enumerate(chunks, 1):
            blocks.append(f"[{i}] Title: {c['title']}\nContent: {c['text']}")
        joined = "\n\n".join(blocks)
        return (
            "You are a helpful technical assistant. Answer the user question using ONLY the provided context.\n"
            "Include inline bracketed citations matching document numbers, e.g. [1] or [2].\n\n"
            f"Context:\n{joined}\n\nQuestion: {query}\nAnswer:"
        )

if __name__ == "__main__":
    chunker = DocumentChunker(chunk_size=50, overlap=10)
    sample_text = (
        "Anthropic prompt caching allows developers to cache frequently used context. "
        "Cache writes cost $3.75 per million tokens for Claude 3.5 Sonnet, while cache reads cost $0.30 per million tokens. "
        "The minimum cacheable prompt length is 1024 tokens for Sonnet and Haiku models."
    )
    chunks = chunker.chunk_document("doc_01", "Claude Prompt Caching", sample_text)
    bm25 = SparseBM25()
    bm25.index(chunks)
    results = bm25.search("pricing prompt caching", top_k=1)
    prompt = PromptSynthesizer.format_context("What is prompt caching pricing?", [chunks[results[0][0]]])
    print(f"Ingested {len(chunks)} chunks.")
    print(f"Top Result Score: {results[0][1]:.4f}")
    print(f"Synthesized Prompt Length: {len(prompt)} characters.")
examples/test_grounded_citations_lib.py (1781 bytes)
import pytest
from grounded_citations_lib import DocumentChunker, SparseBM25, HybridRanker, PromptSynthesizer

def test_document_chunker():
    chunker = DocumentChunker(chunk_size=10, overlap=2)
    sample_text = " ".join([f"word{i}" for i in range(25)])
    chunks = chunker.chunk_document("doc1", "Test Doc", sample_text)
    assert len(chunks) >= 3
    assert chunks[0]["doc_id"] == "doc1"
    assert chunks[0]["title"] == "Test Doc"
    assert len(chunks[0]["text"].split()) == 10

def test_sparse_bm25():
    bm25 = SparseBM25()
    chunks = [
        {"chunk_id": "c1", "title": "Doc 1", "text": "Claude 3.5 Sonnet supports prompt caching and tool use."},
        {"chunk_id": "c2", "title": "Doc 2", "text": "PostgreSQL database configuration for pgvector storage."},
        {"chunk_id": "c3", "title": "Doc 3", "text": "Prompt caching reduces API cost by ninety percent."}
    ]
    bm25.index(chunks)
    results = bm25.search("prompt caching", top_k=2)
    assert len(results) == 2
    top_doc_idx = results[0][0]
    assert top_doc_idx in (0, 2)
    assert results[0][1] > 0.0

def test_hybrid_ranker():
    ranker = HybridRanker(rrf_k=60)
    sparse_ranks = [0, 1, 2]
    dense_ranks = [2, 0, 3]
    fused = ranker.fuse_ranks(sparse_ranks, dense_ranks)
    assert len(fused) == 4
    # Item 0 appears at rank 1 in list_a and rank 2 in list_b
    assert fused[0][0] == 0

def test_prompt_synthesizer():
    chunks = [
        {"chunk_id": "c1", "title": "Pricing", "text": "Prompt cache read tokens cost $0.30 per million."}
    ]
    prompt = PromptSynthesizer.format_context("What is prompt cache read cost?", chunks)
    assert "[1] Title: Pricing" in prompt
    assert "$0.30 per million" in prompt
    assert "Question: What is prompt cache read cost?" in prompt
metadata.yml (431 bytes)
lesson_id: D270
day: 270
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/grounded_citations_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (14 bytes)
pytest>=8.0.0
starter/grounded_citations_lib.py (1316 bytes)
from typing import List, Dict, Any, Tuple

class DocumentChunker:
    def __init__(self, chunk_size: int = 150, overlap: int = 30):
        self.chunk_size = chunk_size
        self.overlap = overlap

    def chunk_document(self, doc_id: str, title: str, text: str) -> List[Dict[str, Any]]:
        # Student implementation: split text into overlapping windows
        return []

class SparseBM25:
    def __init__(self, k1: float = 1.5, b: float = 0.75):
        self.k1 = k1
        self.b = b

    def index(self, chunks: List[Dict[str, Any]]):
        # Student implementation: compute token frequencies and IDFs
        pass

    def search(self, query: str, top_k: int = 10) -> List[Tuple[int, float]]:
        # Student implementation: score candidate chunks
        return []

class HybridRanker:
    def __init__(self, rrf_k: int = 60):
        self.rrf_k = rrf_k

    def fuse_ranks(self, list_a: List[int], list_b: List[int]) -> List[Tuple[int, float]]:
        # Student implementation: reciprocal rank fusion
        return []

class PromptSynthesizer:
    @staticmethod
    def format_context(query: str, chunks: List[Dict[str, Any]]) -> str:
        # Student implementation: format prompt with numbered citations
        return ""

if __name__ == "__main__":
    print("Running starter scaffold...")
starter/test_grounded_citations_lib.py (1781 bytes)
import pytest
from grounded_citations_lib import DocumentChunker, SparseBM25, HybridRanker, PromptSynthesizer

def test_document_chunker():
    chunker = DocumentChunker(chunk_size=10, overlap=2)
    sample_text = " ".join([f"word{i}" for i in range(25)])
    chunks = chunker.chunk_document("doc1", "Test Doc", sample_text)
    assert len(chunks) >= 3
    assert chunks[0]["doc_id"] == "doc1"
    assert chunks[0]["title"] == "Test Doc"
    assert len(chunks[0]["text"].split()) == 10

def test_sparse_bm25():
    bm25 = SparseBM25()
    chunks = [
        {"chunk_id": "c1", "title": "Doc 1", "text": "Claude 3.5 Sonnet supports prompt caching and tool use."},
        {"chunk_id": "c2", "title": "Doc 2", "text": "PostgreSQL database configuration for pgvector storage."},
        {"chunk_id": "c3", "title": "Doc 3", "text": "Prompt caching reduces API cost by ninety percent."}
    ]
    bm25.index(chunks)
    results = bm25.search("prompt caching", top_k=2)
    assert len(results) == 2
    top_doc_idx = results[0][0]
    assert top_doc_idx in (0, 2)
    assert results[0][1] > 0.0

def test_hybrid_ranker():
    ranker = HybridRanker(rrf_k=60)
    sparse_ranks = [0, 1, 2]
    dense_ranks = [2, 0, 3]
    fused = ranker.fuse_ranks(sparse_ranks, dense_ranks)
    assert len(fused) == 4
    # Item 0 appears at rank 1 in list_a and rank 2 in list_b
    assert fused[0][0] == 0

def test_prompt_synthesizer():
    chunks = [
        {"chunk_id": "c1", "title": "Pricing", "text": "Prompt cache read tokens cost $0.30 per million."}
    ]
    prompt = PromptSynthesizer.format_context("What is prompt cache read cost?", chunks)
    assert "[1] Title: Pricing" in prompt
    assert "$0.30 per million" in prompt
    assert "Question: What is prompt cache read cost?" in prompt
tests/run_tests.sh (135 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH="$(cd "$(dirname "$0")/.." && pwd)/examples"
python3 tests/test_grounded_citations_lib.py
tests/test_grounded_citations_lib.py (1960 bytes)
from grounded_citations_lib import DocumentChunker, SparseBM25, HybridRanker, PromptSynthesizer

def test_document_chunker():
    chunker = DocumentChunker(chunk_size=10, overlap=2)
    text = "one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen"
    chunks = chunker.chunk_document("doc_01", "Numbers", text)
    assert len(chunks) >= 2
    assert chunks[0]["chunk_id"] == "doc_01#chunk_0"
    assert chunks[0]["title"] == "Numbers"
    assert len(chunks[0]["text"].split()) == 10

def test_sparse_bm25():
    bm25 = SparseBM25()
    chunks = [
        {"chunk_id": "c1", "title": "Caching", "text": "Anthropic prompt caching reduces latency and token costs."},
        {"chunk_id": "c2", "title": "Tools", "text": "Tool calling allows LLMs to execute structured JSON schemas."},
        {"chunk_id": "c3", "title": "Vision", "text": "Multimodal vision models inspect images and diagrams."}
    ]
    bm25.index(chunks)
    results = bm25.search("prompt caching latency", top_k=2)
    assert len(results) == 2
    assert results[0][0] == 0
    assert results[0][1] > 0.0

def test_hybrid_ranker():
    ranker = HybridRanker(rrf_k=60)
    list_a = [0, 1, 2]
    list_b = [1, 0, 3]
    fused = ranker.fuse_ranks(list_a, list_b)
    top_docs = [idx for idx, score in fused[:2]]
    assert 0 in top_docs
    assert 1 in top_docs

def test_prompt_synthesizer():
    chunks = [
        {"chunk_id": "c1", "title": "Prompt Cache Guide", "text": "Cache write costs $3.75 per million tokens."}
    ]
    prompt = PromptSynthesizer.format_context("What is the cost?", chunks)
    assert "[1] Title: Prompt Cache Guide" in prompt
    assert "Cache write costs $3.75 per million tokens." in prompt
    assert "Question: What is the cost?" in prompt

if __name__ == "__main__":
    test_document_chunker()
    test_sparse_bm25()
    test_hybrid_ranker()
    test_prompt_synthesizer()
    print("All 4 unit tests passed successfully.")

Troubleshooting

Troubleshooting: Day 270 - Citations and Grounded Answers

Common Issues

  1. Zero scores on BM25:
    • Cause: Query tokens do not match document tokens.
    • Fix: Normalize text using regex word extraction and lowercasing.

Security notes

Security & Privacy: Day 270 - Citations and Grounded Answers

Security Guidance

  • Restrict prompt context injection to verified document payloads.
  • Sanitize user query strings to prevent prompt injection jailbreaks.