AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 325

Hands-on lab — Day 325: Keeping Indexes Fresh

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/freshness_demo.py

Test

bash tests/run_tests.sh

File tree

examples/freshness_demo.py
examples/freshness.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/freshness.py
tests/run_tests.sh
tests/test_freshness.py
troubleshooting.md

Lab README

Lab — Day 325: Keeping Indexes Fresh

Lesson

  • Lesson title: Keeping Indexes Fresh
  • Day number: 325 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-325-keeping-indexes-fresh
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-325-keeping-indexes-fresh when the site is running.

Purpose

Build a freshness checker that measures how far an index has drifted from its source, rather than assuming it has not. You classify every document into one of four drift categories, measure staleness as a distribution, guard deletion against a failed source enumeration, and reconcile until the two sides agree.

Everything runs offline against in-memory state, using a logical clock so the results are deterministic.

Learning objectives

  • Classify drift into fresh, missing, stale and orphaned, and explain why they differ in consequence.
  • Compare over the union of both id sets, and explain why a source-driven scan cannot find an orphan.
  • Report staleness as a percentile over per-document ages rather than a count.
  • Refuse a reconciliation that would delete an implausible share of the index.
  • Show that a correct reconciliation converges on a second run.

Prerequisites

  • Day 323, "Data Ingestion Pipelines" — the content hash used as a version here comes from there.
  • Day 324, "Document Processing at Scale".
  • Comfortable with Python dictionaries, sets and enums.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.

Required software

  • Python 3.10 or newer.
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no hosted service. The lesson discusses Elasticsearch aliases, Debezium and Dagster freshness policies as real-world options; none is required here.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/freshness.py         your work: compare, staleness_age, percentile,
                             reconcile, safe_to_delete
examples/freshness.py        reference implementation
examples/freshness_demo.py   compares a drifted pair and reconciles it
tests/test_freshness.py      grouped by concern
tests/run_tests.sh           suite entry point
expected-output/             real captured output and measured values
requirements/                pinned dependency

How to run

python3 examples/freshness_demo.py   # classify, measure, guard, reconcile
bash tests/run_tests.sh              # run the suite

To work on the exercise, edit starter/freshness.py, then copy it over the reference:

cp starter/freshness.py examples/freshness.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/freshness_demo.py builds a source and an index that disagree in every possible way, prints one line per document, reports staleness ages and a p95, checks the delete guard, reconciles, and compares again to show convergence.

bash tests/run_tests.sh runs pytest over seventeen tests grouped by concern — classification, staleness, reconciliation and the delete guard.

Expected output

doc-1    stale     index at v1, source at v2
doc-2    fresh
doc-3    stale     index at v4, source at v5
doc-4    missing   at source, not indexed
doc-9    orphaned  deleted at source, still indexed
fresh=1 missing=1 stale=2 orphaned=1
staleness ages: {'doc-1': 10, 'doc-3': 2}
p95 staleness: 10 ticks
safe to delete: True
indexed=1 updated=2 deleted=1
after reconcile: fresh=4 missing=0 stale=0 orphaned=0

doc-9 is the document to notice: it is not at the source at all, so any process starting from "for each document in the source" walks straight past it. expected-output/FIELDS.md explains each part.

Validation steps

  1. bash tests/run_tests.sh reports 17 passed.
  2. Reconcile twice. The second run must report indexed=0 updated=0 deleted=0 — a correct reconciliation converges.
  3. Build a state with an empty source and twenty indexed documents. safe_to_delete must return False; that shape is a broken enumeration, not a mass deletion.

Tests

Seventeen tests in four groups:

  • classification — each category is identified; every document lands in exactly one; the comparison walks the union rather than the source.
  • staleness — ages cover only stale documents, are never negative, and the percentile uses nearest rank (ceil, not round).
  • reconcile — every category is fixed, the result is idempotent, and deletes can be withheld.
  • delete guard — an implausible mass deletion is blocked, a normal one is allowed, an empty index is safe, and the threshold is configurable.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md for each failure this lab is designed to produce.

Security notes

See security.md. In short: no network, no credentials, no API key. The delete guard is itself a security control, and deletion propagation is where right-to-erasure is actually honoured.

Extension exercises

  1. Per-partition reconciliation. Reconcile one partition per run, track when each was last checked, and apply the delete guard per partition rather than globally.
  2. A freshness objective. State a target such as p95 staleness under ten ticks with zero orphans, and report an error budget against it over a simulated week of intermittent failures. Report honestly whether it was met.
  3. Adaptive scheduling. Track how often each document changes and check volatile documents more frequently, then measure whether p95 staleness actually improved for the same number of checks.
  • Lesson
  • Previous: Day 324 — Document Processing at Scale
  • Next: Day 326 — Scaling Retrieval

Expected output

FIELDS.md

# Reading the freshness output

## The per-document lines

    doc-1    stale     index at v1, source at v2

| Part | Meaning |
| --- | --- |
| id | The document. |
| drift | One of `fresh`, `missing`, `stale`, `orphaned` — every document lands in exactly one. |
| detail | For `stale`, both versions. For `missing` and `orphaned`, which side has it. `fresh` has none. |

## The summary line

    fresh=1 missing=1 stale=2 orphaned=1

Counts per category. `orphaned` is the one to watch: it is the only category a source-driven scan cannot produce, and the only one that can mean deleted content is still being quoted.

## Staleness

    staleness ages: {'doc-1': 10, 'doc-3': 2}
    p95 staleness: 10 ticks

Ages, not counts. Both documents are stale, but `doc-1` has been stale five times longer — the count treats them identically and the ages do not. Ticks are a logical clock so the tests are deterministic; in production this is wall-clock time.

## The guard and the reconcile

    safe to delete: True
    indexed=1 updated=2 deleted=1
    after reconcile: fresh=4 missing=0 stale=0 orphaned=0

One orphan out of four indexed documents is 25 percent, exactly at the default threshold, so deletion is permitted. Had the source enumeration returned nothing, all four would have looked orphaned, the fraction would have been 100 percent, and the guard would have refused the run outright.

The final line is the convergence check: everything fresh, nothing left in any other category.

examples-run.txt

doc-1    stale     index at v1, source at v2
doc-2    fresh
doc-3    stale     index at v4, source at v5
doc-4    missing   at source, not indexed
doc-9    orphaned  deleted at source, still indexed
fresh=1 missing=1 stale=2 orphaned=1
staleness ages: {'doc-1': 10, 'doc-3': 2}
p95 staleness: 10 ticks
safe to delete: True
indexed=1 updated=2 deleted=1
after reconcile: fresh=4 missing=0 stale=0 orphaned=0

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/freshness_demo.py -> 11 lines, exit 0
  bash tests/run_tests.sh            -> 17 passed in 0.01s

Values worth checking against your own implementation:

  staleness ages     {'doc-1': 10, 'doc-3': 2}
  p95 staleness      10   (nearest rank over two observations)
  safe to delete     True (1 orphan of 4 indexed = 25%, exactly at the default
                     threshold, so it is permitted)
  reconcile          indexed=1 updated=2 deleted=1
  after reconcile    fresh=4 missing=0 stale=0 orphaned=0

A second reconcile against the same source reports indexed=0 updated=0
deleted=0. A reconciler that does not converge is disagreeing with its own
writer about what "same version" means.

run-output.txt

doc-1    stale     index at v1, source at v2
doc-2    fresh
doc-3    stale     index at v4, source at v5
doc-4    missing   at source, not indexed
doc-9    orphaned  deleted at source, still indexed
fresh=1 missing=1 stale=2 orphaned=1
staleness ages: {'doc-1': 10, 'doc-3': 2}
p95 staleness: 10 ticks
safe to delete: True
indexed=1 updated=2 deleted=1
after reconcile: fresh=4 missing=0 stale=0 orphaned=0

starter-run.txt

An untouched starter fails every test:

    17 failed in 0.06s

All five stubbed functions raise NotImplementedError, and every test depends on
at least one of them.

test-output.txt

.................                                                        [100%]
17 passed in 0.01s

test-run.txt

.................                                                        [100%]
17 passed in 0.01s

Source files

examples/freshness_demo.py (1693 bytes)
#!/usr/bin/env python3
"""Detect drift between a source and an index, then reconcile it."""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from freshness import (
    Drift,
    Record,
    compare,
    percentile,
    reconcile,
    safe_to_delete,
    staleness_age,
)


def build_state():
    source = {
        "doc-1": Record("doc-1", "v2", updated_at=10),
        "doc-2": Record("doc-2", "v1", updated_at=2),
        "doc-3": Record("doc-3", "v5", updated_at=18),
        "doc-4": Record("doc-4", "v1", updated_at=4),
    }
    index = {
        "doc-1": Record("doc-1", "v1", updated_at=3),
        "doc-2": Record("doc-2", "v1", updated_at=2),
        "doc-3": Record("doc-3", "v4", updated_at=11),
        "doc-9": Record("doc-9", "v1", updated_at=1),
    }
    return source, index


def main() -> int:
    source, index = build_state()
    now = 20

    report = compare(source, index, now=now)
    for finding in report.findings:
        line = f"{finding.doc_id:<8} {finding.drift.value}"
        if finding.detail:
            line = f"{finding.doc_id:<8} {finding.drift.value:<9} {finding.detail}"
        print(line)
    print(report.summary())

    ages = staleness_age(source, index, now=now)
    print(f"staleness ages: {ages}")
    print(f"p95 staleness: {percentile(list(ages.values()), 95)} ticks")

    print(f"safe to delete: {safe_to_delete(report)}")
    result = reconcile(source, index, report)
    print(result.summary())

    after = compare(source, index, now=now)
    print(f"after reconcile: {after.summary()}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/freshness.py (6426 bytes)
"""Index freshness: detecting drift between a source and an index, and fixing it.

Offline and standard-library only. The source and index are in-memory, but the
drift categories are the real ones, and the point of the lab is that they are
genuinely different problems with genuinely different fixes.

Four ways an index drifts from its source:

  missing    at the source, absent from the index    -> a user cannot find it
  stale      in both, but the index holds an old copy -> answers cite old text
  orphaned   in the index, deleted at the source      -> the serious one
  fresh      in both and identical                    -> nothing to do

Orphaned is the category that matters most and gets noticed least. A missing
document produces a complaint. An orphaned one produces a confident answer from
content that was deleted -- sometimes for legal reasons.
"""

from __future__ import annotations

import math
from dataclasses import dataclass, field
from enum import Enum


class Drift(str, Enum):
    FRESH = "fresh"
    MISSING = "missing"
    STALE = "stale"
    ORPHANED = "orphaned"


@dataclass(frozen=True)
class Record:
    """A document as either side holds it."""

    doc_id: str
    version: str  # a content hash in production; a short token here
    updated_at: int  # logical clock, so tests are deterministic


@dataclass
class Finding:
    doc_id: str
    drift: Drift
    detail: str = ""


@dataclass
class FreshnessReport:
    findings: list[Finding] = field(default_factory=list)
    checked_at: int = 0

    def by_drift(self, drift: Drift) -> list[Finding]:
        return [f for f in self.findings if f.drift is drift]

    def count(self, drift: Drift) -> int:
        return len(self.by_drift(drift))

    def summary(self) -> str:
        return (
            f"fresh={self.count(Drift.FRESH)} missing={self.count(Drift.MISSING)} "
            f"stale={self.count(Drift.STALE)} orphaned={self.count(Drift.ORPHANED)}"
        )


def compare(source: dict[str, Record], index: dict[str, Record], *, now: int = 0) -> FreshnessReport:
    """Classify every document on either side into exactly one drift category.

    The union of both key sets matters, not just the source: iterating the
    source alone can never find an orphan, because an orphan is precisely a
    document the source no longer mentions. That asymmetry is why deletion
    propagation is so often missed.
    """
    findings: list[Finding] = []
    for doc_id in sorted(set(source) | set(index)):
        src = source.get(doc_id)
        idx = index.get(doc_id)
        if src is not None and idx is None:
            findings.append(Finding(doc_id, Drift.MISSING, "at source, not indexed"))
        elif src is None and idx is not None:
            findings.append(Finding(doc_id, Drift.ORPHANED, "deleted at source, still indexed"))
        elif src.version != idx.version:
            findings.append(
                Finding(doc_id, Drift.STALE, f"index at {idx.version}, source at {src.version}")
            )
        else:
            findings.append(Finding(doc_id, Drift.FRESH))
    return FreshnessReport(findings=findings, checked_at=now)


def staleness_age(source: dict[str, Record], index: dict[str, Record], *, now: int) -> dict[str, int]:
    """How long each stale document has been stale, in logical ticks.

    Freshness is a distribution, not a boolean. A p95 staleness age is what you
    can actually put in an objective; "the index is fresh" is not measurable.
    """
    ages: dict[str, int] = {}
    for doc_id, src in source.items():
        idx = index.get(doc_id)
        if idx is not None and idx.version != src.version:
            ages[doc_id] = max(0, now - src.updated_at)
    return ages


def percentile(values: list[int], p: float) -> int:
    """Nearest-rank percentile. Returns 0 for an empty input.

    The rank is `ceil(p/100 * N)`, which is the standard nearest-rank
    definition. `round` is wrong here and wrong in a way that is easy to miss:
    Python rounds halves to even, so `round(2.5)` is 2, and the p50 of
    [1, 2, 3, 4, 100] comes out as 2 rather than 3.
    """
    if not values:
        return 0
    ordered = sorted(values)
    rank = max(1, min(len(ordered), math.ceil(p / 100 * len(ordered))))
    return ordered[rank - 1]


@dataclass
class ReconcileResult:
    indexed: list[str] = field(default_factory=list)
    updated: list[str] = field(default_factory=list)
    deleted: list[str] = field(default_factory=list)

    def summary(self) -> str:
        return (
            f"indexed={len(self.indexed)} updated={len(self.updated)} "
            f"deleted={len(self.deleted)}"
        )


def reconcile(
    source: dict[str, Record],
    index: dict[str, Record],
    report: FreshnessReport,
    *,
    allow_deletes: bool = True,
) -> ReconcileResult:
    """Bring the index into agreement with the source.

    `allow_deletes` exists because deletion is the irreversible half. A
    reconciliation run against a source that failed to enumerate correctly --
    an auth error returning an empty list, say -- would otherwise delete the
    entire index. Production systems gate deletes behind a sanity check; see
    `safe_to_delete`.
    """
    result = ReconcileResult()
    for finding in report.findings:
        if finding.drift is Drift.MISSING:
            index[finding.doc_id] = source[finding.doc_id]
            result.indexed.append(finding.doc_id)
        elif finding.drift is Drift.STALE:
            index[finding.doc_id] = source[finding.doc_id]
            result.updated.append(finding.doc_id)
        elif finding.drift is Drift.ORPHANED and allow_deletes:
            del index[finding.doc_id]
            result.deleted.append(finding.doc_id)
    return result


def safe_to_delete(report: FreshnessReport, *, max_fraction: float = 0.25) -> bool:
    """Refuse a reconciliation that would delete an implausible share of the index.

    If the source enumeration silently returned nothing, every indexed document
    looks orphaned and a naive reconcile empties the index. The guard is crude
    on purpose: a threshold a human can reason about beats a clever heuristic
    nobody trusts at 3 a.m.
    """
    indexed_total = report.count(Drift.FRESH) + report.count(Drift.STALE) + report.count(Drift.ORPHANED)
    if indexed_total == 0:
        return True
    return (report.count(Drift.ORPHANED) / indexed_total) <= max_fraction
metadata.yml (730 bytes)
lesson_id: D325
day: 325
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/freshness_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3
  examples/freshness_demo.py reproduced the documented output exactly; bash tests/run_tests.sh -> 17
  passed in 0.01s. The untouched starter was verified to fail all 17. requires_network is false: source
  and index are in-memory dictionaries and no test opens a socket.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/freshness.py (6573 bytes)
"""Index freshness: detecting drift between a source and an index, and fixing it.

Offline and standard-library only. The source and index are in-memory, but the
drift categories are the real ones, and the point of the lab is that they are
genuinely different problems with genuinely different fixes.

Four ways an index drifts from its source:

  missing    at the source, absent from the index    -> a user cannot find it
  stale      in both, but the index holds an old copy -> answers cite old text
  orphaned   in the index, deleted at the source      -> the serious one
  fresh      in both and identical                    -> nothing to do

Orphaned is the category that matters most and gets noticed least. A missing
document produces a complaint. An orphaned one produces a confident answer from
content that was deleted -- sometimes for legal reasons.
"""

from __future__ import annotations

import math
from dataclasses import dataclass, field
from enum import Enum


class Drift(str, Enum):
    FRESH = "fresh"
    MISSING = "missing"
    STALE = "stale"
    ORPHANED = "orphaned"


@dataclass(frozen=True)
class Record:
    """A document as either side holds it."""

    doc_id: str
    version: str  # a content hash in production; a short token here
    updated_at: int  # logical clock, so tests are deterministic


@dataclass
class Finding:
    doc_id: str
    drift: Drift
    detail: str = ""


@dataclass
class FreshnessReport:
    findings: list[Finding] = field(default_factory=list)
    checked_at: int = 0

    def by_drift(self, drift: Drift) -> list[Finding]:
        return [f for f in self.findings if f.drift is drift]

    def count(self, drift: Drift) -> int:
        return len(self.by_drift(drift))

    def summary(self) -> str:
        return (
            f"fresh={self.count(Drift.FRESH)} missing={self.count(Drift.MISSING)} "
            f"stale={self.count(Drift.STALE)} orphaned={self.count(Drift.ORPHANED)}"
        )


def compare(source: dict[str, Record], index: dict[str, Record], *, now: int = 0) -> FreshnessReport:
    """Classify every document on either side into exactly one drift category.

    The union of both key sets matters, not just the source: iterating the
    source alone can never find an orphan, because an orphan is precisely a
    document the source no longer mentions. That asymmetry is why deletion
    propagation is so often missed.
    """
    # TASK 1: classify every document into exactly one drift category.
    # Iterate sorted(set(source) | set(index)) -- the UNION. Iterating the
    # source alone can never find an orphan, because an orphan is precisely a
    # document the source no longer mentions.
    #   at source, not indexed        -> Drift.MISSING,  "at source, not indexed"
    #   indexed, not at source        -> Drift.ORPHANED, "deleted at source, still indexed"
    #   both, versions differ         -> Drift.STALE,    f"index at {i.version}, source at {s.version}"
    #   both, versions equal          -> Drift.FRESH,    no detail
    raise NotImplementedError("implement compare")

def staleness_age(source: dict[str, Record], index: dict[str, Record], *, now: int) -> dict[str, int]:
    """How long each stale document has been stale, in logical ticks.

    Freshness is a distribution, not a boolean. A p95 staleness age is what you
    can actually put in an objective; "the index is fresh" is not measurable.
    """
    # TASK 2: for each document present in BOTH sides with differing
    # versions, return now - source.updated_at, floored at 0. Documents that
    # are fresh, missing or orphaned do not appear in the result.
    raise NotImplementedError("implement staleness_age")

def percentile(values: list[int], p: float) -> int:
    """Nearest-rank percentile. Returns 0 for an empty input.

    The rank is `ceil(p/100 * N)`, which is the standard nearest-rank
    definition. `round` is wrong here and wrong in a way that is easy to miss:
    Python rounds halves to even, so `round(2.5)` is 2, and the p50 of
    [1, 2, 3, 4, 100] comes out as 2 rather than 3.
    """
    # TASK 3: nearest-rank percentile; return 0 for an empty list.
    # rank = ceil(p/100 * N), clamped to [1, N], then return ordered[rank-1].
    # Use math.ceil, NOT round: Python rounds halves to even, so round(2.5) is
    # 2 and the p50 of [1,2,3,4,100] would come out as 2 rather than 3.
    raise NotImplementedError("implement percentile")

@dataclass
class ReconcileResult:
    indexed: list[str] = field(default_factory=list)
    updated: list[str] = field(default_factory=list)
    deleted: list[str] = field(default_factory=list)

    def summary(self) -> str:
        return (
            f"indexed={len(self.indexed)} updated={len(self.updated)} "
            f"deleted={len(self.deleted)}"
        )


def reconcile(
    source: dict[str, Record],
    index: dict[str, Record],
    report: FreshnessReport,
    *,
    allow_deletes: bool = True,
) -> ReconcileResult:
    """Bring the index into agreement with the source.

    `allow_deletes` exists because deletion is the irreversible half. A
    reconciliation run against a source that failed to enumerate correctly --
    an auth error returning an empty list, say -- would otherwise delete the
    entire index. Production systems gate deletes behind a sanity check; see
    `safe_to_delete`.
    """
    # TASK 4: apply the report.
    #   MISSING  -> index[doc_id] = source[doc_id];  result.indexed.append(id)
    #   STALE    -> index[doc_id] = source[doc_id];  result.updated.append(id)
    #   ORPHANED -> del index[doc_id] IF allow_deletes; result.deleted.append(id)
    # Reconciling twice against an unchanged source must do nothing the second
    # time.
    raise NotImplementedError("implement reconcile")

def safe_to_delete(report: FreshnessReport, *, max_fraction: float = 0.25) -> bool:
    """Refuse a reconciliation that would delete an implausible share of the index.

    If the source enumeration silently returned nothing, every indexed document
    looks orphaned and a naive reconcile empties the index. The guard is crude
    on purpose: a threshold a human can reason about beats a clever heuristic
    nobody trusts at 3 a.m.
    """
    # TASK 5: return False when the orphaned share of the INDEXED total
    # (fresh + stale + orphaned) exceeds max_fraction. An empty index is safe.
    # This is what stops a failed source enumeration -- one that returned
    # nothing but looked successful -- from emptying the whole index.
    raise NotImplementedError("implement safe_to_delete")
tests/run_tests.sh (205 bytes)
#!/usr/bin/env bash
# Run the day 325 index freshness test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_freshness.py (5273 bytes)
"""Grouped by drift category, so a failure names which one is misclassified.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from freshness import (  # noqa: E402
    Drift,
    Record,
    compare,
    percentile,
    reconcile,
    safe_to_delete,
    staleness_age,
)


def rec(doc_id: str, version: str, updated_at: int = 0) -> Record:
    return Record(doc_id, version, updated_at)


# ------------------------------------------------------------ classification


def test_identical_documents_are_fresh():
    src = {"a": rec("a", "v1")}
    idx = {"a": rec("a", "v1")}
    report = compare(src, idx)
    assert report.count(Drift.FRESH) == 1
    assert report.summary() == "fresh=1 missing=0 stale=0 orphaned=0"


def test_document_only_at_source_is_missing():
    report = compare({"a": rec("a", "v1")}, {})
    assert report.by_drift(Drift.MISSING)[0].doc_id == "a"


def test_differing_version_is_stale():
    report = compare({"a": rec("a", "v2")}, {"a": rec("a", "v1")})
    finding = report.by_drift(Drift.STALE)[0]
    assert finding.doc_id == "a"
    assert "v1" in finding.detail and "v2" in finding.detail


def test_document_only_in_index_is_orphaned():
    # The category a source-only scan can never find, because an orphan is
    # exactly a document the source no longer mentions.
    report = compare({}, {"gone": rec("gone", "v1")})
    assert report.by_drift(Drift.ORPHANED)[0].doc_id == "gone"


def test_every_document_lands_in_exactly_one_category():
    src = {"a": rec("a", "v1"), "b": rec("b", "v2")}
    idx = {"b": rec("b", "v1"), "c": rec("c", "v1")}
    report = compare(src, idx)
    ids = [f.doc_id for f in report.findings]
    assert sorted(ids) == ["a", "b", "c"]
    assert len(ids) == len(set(ids))


def test_comparison_walks_the_union_not_just_the_source():
    # Iterating the source alone would report a clean index here.
    report = compare({}, {"x": rec("x", "v1"), "y": rec("y", "v1")})
    assert report.count(Drift.ORPHANED) == 2


# ----------------------------------------------------------------- staleness


def test_staleness_age_measures_only_stale_documents():
    src = {"a": rec("a", "v2", updated_at=10), "b": rec("b", "v1", updated_at=3)}
    idx = {"a": rec("a", "v1", updated_at=4), "b": rec("b", "v1", updated_at=3)}
    ages = staleness_age(src, idx, now=20)
    assert set(ages) == {"a"}
    assert ages["a"] == 10


def test_staleness_age_is_never_negative():
    src = {"a": rec("a", "v2", updated_at=30)}
    idx = {"a": rec("a", "v1", updated_at=1)}
    assert staleness_age(src, idx, now=10)["a"] == 0


def test_percentile_of_empty_is_zero():
    assert percentile([], 95) == 0


def test_percentile_picks_a_real_observation():
    values = [1, 2, 3, 4, 100]
    assert percentile(values, 95) == 100
    assert percentile(values, 50) == 3
    assert percentile(values, 100) == 100


# --------------------------------------------------------------- reconcile


def test_reconcile_fixes_every_category():
    src = {"a": rec("a", "v1"), "b": rec("b", "v2")}
    idx = {"b": rec("b", "v1"), "c": rec("c", "v1")}
    report = compare(src, idx)
    result = reconcile(src, idx, report)

    assert result.indexed == ["a"]
    assert result.updated == ["b"]
    assert result.deleted == ["c"]
    assert compare(src, idx).summary() == "fresh=2 missing=0 stale=0 orphaned=0"


def test_reconcile_is_idempotent():
    src = {"a": rec("a", "v1"), "b": rec("b", "v2")}
    idx = {"b": rec("b", "v1"), "c": rec("c", "v1")}
    reconcile(src, idx, compare(src, idx))
    snapshot = dict(idx)

    second = reconcile(src, idx, compare(src, idx))
    assert second.summary() == "indexed=0 updated=0 deleted=0"
    assert idx == snapshot


def test_deletes_can_be_withheld():
    src: dict[str, Record] = {}
    idx = {"gone": rec("gone", "v1")}
    result = reconcile(src, idx, compare(src, idx), allow_deletes=False)
    assert result.deleted == []
    assert "gone" in idx


# ------------------------------------------------------------- delete guard


def test_guard_blocks_an_implausible_mass_deletion():
    # The source enumeration returned nothing -- an auth failure, say. Every
    # indexed document now looks orphaned.
    idx = {f"doc-{i}": rec(f"doc-{i}", "v1") for i in range(20)}
    report = compare({}, idx)
    assert report.count(Drift.ORPHANED) == 20
    assert safe_to_delete(report) is False


def test_guard_allows_a_normal_amount_of_deletion():
    src = {f"doc-{i}": rec(f"doc-{i}", "v1") for i in range(19)}
    idx = dict(src)
    idx["doc-gone"] = rec("doc-gone", "v1")
    report = compare(src, idx)
    assert report.count(Drift.ORPHANED) == 1
    assert safe_to_delete(report) is True


def test_guard_permits_an_empty_index():
    assert safe_to_delete(compare({}, {})) is True


def test_guard_threshold_is_configurable():
    src = {"a": rec("a", "v1")}
    idx = {"a": rec("a", "v1"), "b": rec("b", "v1"), "c": rec("c", "v1")}
    report = compare(src, idx)  # 2 of 3 orphaned
    assert safe_to_delete(report, max_fraction=0.25) is False
    assert safe_to_delete(report, max_fraction=0.9) is True

Troubleshooting

Troubleshooting — Day 325

No orphans are ever found

You are iterating source rather than set(source) | set(index). An orphan is by definition a document the source no longer mentions, so a source-driven loop can never visit it. This is the most common bug in this area and the reason deletions silently fail to propagate in real systems.

Check with:

python3 -c "
import sys; sys.path.insert(0,'examples')
from freshness import compare, Record, Drift
r = compare({}, {'gone': Record('gone','v1',0)})
print(r.count(Drift.ORPHANED))   # must be 1"

The p50 of [1, 2, 3, 4, 100] comes out as 2

You used round for the nearest-rank calculation. Python rounds halves to even, so round(2.5) is 2 and you land on the wrong observation. Nearest rank is math.ceil(p / 100 * N).

test_staleness_age_is_never_negative fails

A source document can carry an updated_at later than now — a clock skew, or a test constructing it deliberately. Floor the result at zero rather than returning a negative age.

Reconciliation is not idempotent

Either you are mutating the index while iterating a report built from the pre-reconcile state in a way that skips entries, or your version comparison is not symmetric with what you write. Recompute the report after reconciling and assert every document is fresh.

The guard blocks a run you expected to succeed

The threshold is a fraction of the indexed total — fresh plus stale plus orphaned — not of the source total. An index holding four documents with one orphan is at exactly 25 percent, which the default permits. Five documents with two orphans is 40 percent, which it does not.

ZeroDivisionError in safe_to_delete

An empty index is a normal state, especially on a first run. Return True when the indexed total is zero rather than dividing.

KeyError during reconcile

You are deleting from index while iterating over it, or looking up a MISSING document in index rather than in source. Iterate report.findings, which is a snapshot, and take the record from the side that actually has it.

NotImplementedError everywhere

Expected. The starter stubs all five functions, and every test depends on at least one — see expected-output/starter-run.txt.

Security notes

Security notes — Day 325

What this lab does and does not touch

  • No network. Source and index are in-memory dictionaries. No test opens a socket, and requires_network is false.
  • No credentials. No API key, token or account anywhere.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/.
  • No sudo, no global installs.

The delete guard is a security control, not just a safety net

safe_to_delete looks like defensive programming, and it is also a genuine containment measure. Consider what a failed source enumeration looks like:

  • an expired credential returning 200 OK with an empty result set;
  • a paginated API that stops at page one because a cursor was dropped;
  • a permissions change that silently narrows what the enumerating identity can see.

In every case the enumeration looks successful and reports far fewer documents than exist. Without a guard, reconciliation faithfully concludes that the missing documents were deleted and empties the index — while reporting complete success.

Refusing the whole run is the correct response rather than deleting a "safe" subset. If the enumeration is wrong, every classification derived from it is wrong, so partial action is still action on bad input.

Privacy: this is where erasure is actually honoured

Deletion propagation is the difference between a compliance story and a compliance failure. A record deleted from the primary system under a right-to-erasure request remains retrievable and quotable until the derived index also drops it.

Three consequences for real systems:

  • Deletion deserves a shorter tolerance than staleness. Being an hour behind on edits is usually fine; being an hour behind on deletions may not be.
  • It needs its own test and its own alert, not a line in a general freshness dashboard.
  • Caches count too. An index that dropped the document but a response cache that did not is still serving it.

Access

A freshness check reads the complete id list from both sides, so it needs broad read access — worth a least-privilege review, and worth keeping separate from any identity that can write.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.