AI Engineering: Agents and Applications › Production Retrieval and Pipelines › Day 325
Hands-on lab — Day 325: Keeping Indexes Fresh
- ← Back to the Day 325 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-325-keeping-indexes-fresh/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/freshness_demo.py Test
bash tests/run_tests.sh File tree
examples/freshness_demo.py examples/freshness.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/freshness.py tests/run_tests.sh tests/test_freshness.py troubleshooting.md
Lab README
Lab — Day 325: Keeping Indexes Fresh
Lesson
- Lesson title: Keeping Indexes Fresh
- Day number: 325 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-325-keeping-indexes-fresh
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-325-keeping-indexes-freshwhen the site is running.
Purpose
Build a freshness checker that measures how far an index has drifted from its source, rather than assuming it has not. You classify every document into one of four drift categories, measure staleness as a distribution, guard deletion against a failed source enumeration, and reconcile until the two sides agree.
Everything runs offline against in-memory state, using a logical clock so the results are deterministic.
Learning objectives
- Classify drift into fresh, missing, stale and orphaned, and explain why they differ in consequence.
- Compare over the union of both id sets, and explain why a source-driven scan cannot find an orphan.
- Report staleness as a percentile over per-document ages rather than a count.
- Refuse a reconciliation that would delete an implausible share of the index.
- Show that a correct reconciliation converges on a second run.
Prerequisites
- Day 323, "Data Ingestion Pipelines" — the content hash used as a version here comes from there.
- Day 324, "Document Processing at Scale".
- Comfortable with Python dictionaries, sets and enums.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.
Hardware requirements
Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.
Required software
- Python 3.10 or newer.
pytest7 or newer.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no hosted service. The lesson discusses Elasticsearch aliases, Debezium and Dagster freshness policies as real-world options; none is required here.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/freshness.py your work: compare, staleness_age, percentile,
reconcile, safe_to_delete
examples/freshness.py reference implementation
examples/freshness_demo.py compares a drifted pair and reconciles it
tests/test_freshness.py grouped by concern
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/freshness_demo.py # classify, measure, guard, reconcile
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/freshness.py, then copy it over the reference:
cp starter/freshness.py examples/freshness.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/freshness_demo.py builds a source and an index that disagree in every possible way, prints one line per document, reports staleness ages and a p95, checks the delete guard, reconciles, and compares again to show convergence.
bash tests/run_tests.sh runs pytest over seventeen tests grouped by concern — classification, staleness, reconciliation and the delete guard.
Expected output
doc-1 stale index at v1, source at v2
doc-2 fresh
doc-3 stale index at v4, source at v5
doc-4 missing at source, not indexed
doc-9 orphaned deleted at source, still indexed
fresh=1 missing=1 stale=2 orphaned=1
staleness ages: {'doc-1': 10, 'doc-3': 2}
p95 staleness: 10 ticks
safe to delete: True
indexed=1 updated=2 deleted=1
after reconcile: fresh=4 missing=0 stale=0 orphaned=0
doc-9 is the document to notice: it is not at the source at all, so any process starting from "for each document in the source" walks straight past it. expected-output/FIELDS.md explains each part.
Validation steps
bash tests/run_tests.shreports17 passed.- Reconcile twice. The second run must report
indexed=0 updated=0 deleted=0— a correct reconciliation converges. - Build a state with an empty source and twenty indexed documents.
safe_to_deletemust returnFalse; that shape is a broken enumeration, not a mass deletion.
Tests
Seventeen tests in four groups:
- classification — each category is identified; every document lands in exactly one; the comparison walks the union rather than the source.
- staleness — ages cover only stale documents, are never negative, and the percentile uses nearest rank (
ceil, notround). - reconcile — every category is fixed, the result is idempotent, and deletes can be withheld.
- delete guard — an implausible mass deletion is blocked, a normal one is allowed, an empty index is safe, and the threshold is configurable.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md for each failure this lab is designed to produce.
Security notes
See security.md. In short: no network, no credentials, no API key. The delete guard is itself a security control, and deletion propagation is where right-to-erasure is actually honoured.
Extension exercises
- Per-partition reconciliation. Reconcile one partition per run, track when each was last checked, and apply the delete guard per partition rather than globally.
- A freshness objective. State a target such as p95 staleness under ten ticks with zero orphans, and report an error budget against it over a simulated week of intermittent failures. Report honestly whether it was met.
- Adaptive scheduling. Track how often each document changes and check volatile documents more frequently, then measure whether p95 staleness actually improved for the same number of checks.
Navigation
- Lesson
- Previous: Day 324 — Document Processing at Scale
- Next: Day 326 — Scaling Retrieval
Expected output
FIELDS.md
# Reading the freshness output
## The per-document lines
doc-1 stale index at v1, source at v2
| Part | Meaning |
| --- | --- |
| id | The document. |
| drift | One of `fresh`, `missing`, `stale`, `orphaned` — every document lands in exactly one. |
| detail | For `stale`, both versions. For `missing` and `orphaned`, which side has it. `fresh` has none. |
## The summary line
fresh=1 missing=1 stale=2 orphaned=1
Counts per category. `orphaned` is the one to watch: it is the only category a source-driven scan cannot produce, and the only one that can mean deleted content is still being quoted.
## Staleness
staleness ages: {'doc-1': 10, 'doc-3': 2}
p95 staleness: 10 ticks
Ages, not counts. Both documents are stale, but `doc-1` has been stale five times longer — the count treats them identically and the ages do not. Ticks are a logical clock so the tests are deterministic; in production this is wall-clock time.
## The guard and the reconcile
safe to delete: True
indexed=1 updated=2 deleted=1
after reconcile: fresh=4 missing=0 stale=0 orphaned=0
One orphan out of four indexed documents is 25 percent, exactly at the default threshold, so deletion is permitted. Had the source enumeration returned nothing, all four would have looked orphaned, the fraction would have been 100 percent, and the guard would have refused the run outright.
The final line is the convergence check: everything fresh, nothing left in any other category.
examples-run.txt
doc-1 stale index at v1, source at v2
doc-2 fresh
doc-3 stale index at v4, source at v5
doc-4 missing at source, not indexed
doc-9 orphaned deleted at source, still indexed
fresh=1 missing=1 stale=2 orphaned=1
staleness ages: {'doc-1': 10, 'doc-3': 2}
p95 staleness: 10 ticks
safe to delete: True
indexed=1 updated=2 deleted=1
after reconcile: fresh=4 missing=0 stale=0 orphaned=0
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/freshness_demo.py -> 11 lines, exit 0
bash tests/run_tests.sh -> 17 passed in 0.01s
Values worth checking against your own implementation:
staleness ages {'doc-1': 10, 'doc-3': 2}
p95 staleness 10 (nearest rank over two observations)
safe to delete True (1 orphan of 4 indexed = 25%, exactly at the default
threshold, so it is permitted)
reconcile indexed=1 updated=2 deleted=1
after reconcile fresh=4 missing=0 stale=0 orphaned=0
A second reconcile against the same source reports indexed=0 updated=0
deleted=0. A reconciler that does not converge is disagreeing with its own
writer about what "same version" means.
run-output.txt
doc-1 stale index at v1, source at v2
doc-2 fresh
doc-3 stale index at v4, source at v5
doc-4 missing at source, not indexed
doc-9 orphaned deleted at source, still indexed
fresh=1 missing=1 stale=2 orphaned=1
staleness ages: {'doc-1': 10, 'doc-3': 2}
p95 staleness: 10 ticks
safe to delete: True
indexed=1 updated=2 deleted=1
after reconcile: fresh=4 missing=0 stale=0 orphaned=0
starter-run.txt
An untouched starter fails every test:
17 failed in 0.06s
All five stubbed functions raise NotImplementedError, and every test depends on
at least one of them.
test-output.txt
................. [100%]
17 passed in 0.01s
test-run.txt
................. [100%]
17 passed in 0.01s
Source files
examples/freshness_demo.py (1693 bytes)
#!/usr/bin/env python3
"""Detect drift between a source and an index, then reconcile it."""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from freshness import (
Drift,
Record,
compare,
percentile,
reconcile,
safe_to_delete,
staleness_age,
)
def build_state():
source = {
"doc-1": Record("doc-1", "v2", updated_at=10),
"doc-2": Record("doc-2", "v1", updated_at=2),
"doc-3": Record("doc-3", "v5", updated_at=18),
"doc-4": Record("doc-4", "v1", updated_at=4),
}
index = {
"doc-1": Record("doc-1", "v1", updated_at=3),
"doc-2": Record("doc-2", "v1", updated_at=2),
"doc-3": Record("doc-3", "v4", updated_at=11),
"doc-9": Record("doc-9", "v1", updated_at=1),
}
return source, index
def main() -> int:
source, index = build_state()
now = 20
report = compare(source, index, now=now)
for finding in report.findings:
line = f"{finding.doc_id:<8} {finding.drift.value}"
if finding.detail:
line = f"{finding.doc_id:<8} {finding.drift.value:<9} {finding.detail}"
print(line)
print(report.summary())
ages = staleness_age(source, index, now=now)
print(f"staleness ages: {ages}")
print(f"p95 staleness: {percentile(list(ages.values()), 95)} ticks")
print(f"safe to delete: {safe_to_delete(report)}")
result = reconcile(source, index, report)
print(result.summary())
after = compare(source, index, now=now)
print(f"after reconcile: {after.summary()}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/freshness.py (6426 bytes)
"""Index freshness: detecting drift between a source and an index, and fixing it.
Offline and standard-library only. The source and index are in-memory, but the
drift categories are the real ones, and the point of the lab is that they are
genuinely different problems with genuinely different fixes.
Four ways an index drifts from its source:
missing at the source, absent from the index -> a user cannot find it
stale in both, but the index holds an old copy -> answers cite old text
orphaned in the index, deleted at the source -> the serious one
fresh in both and identical -> nothing to do
Orphaned is the category that matters most and gets noticed least. A missing
document produces a complaint. An orphaned one produces a confident answer from
content that was deleted -- sometimes for legal reasons.
"""
from __future__ import annotations
import math
from dataclasses import dataclass, field
from enum import Enum
class Drift(str, Enum):
FRESH = "fresh"
MISSING = "missing"
STALE = "stale"
ORPHANED = "orphaned"
@dataclass(frozen=True)
class Record:
"""A document as either side holds it."""
doc_id: str
version: str # a content hash in production; a short token here
updated_at: int # logical clock, so tests are deterministic
@dataclass
class Finding:
doc_id: str
drift: Drift
detail: str = ""
@dataclass
class FreshnessReport:
findings: list[Finding] = field(default_factory=list)
checked_at: int = 0
def by_drift(self, drift: Drift) -> list[Finding]:
return [f for f in self.findings if f.drift is drift]
def count(self, drift: Drift) -> int:
return len(self.by_drift(drift))
def summary(self) -> str:
return (
f"fresh={self.count(Drift.FRESH)} missing={self.count(Drift.MISSING)} "
f"stale={self.count(Drift.STALE)} orphaned={self.count(Drift.ORPHANED)}"
)
def compare(source: dict[str, Record], index: dict[str, Record], *, now: int = 0) -> FreshnessReport:
"""Classify every document on either side into exactly one drift category.
The union of both key sets matters, not just the source: iterating the
source alone can never find an orphan, because an orphan is precisely a
document the source no longer mentions. That asymmetry is why deletion
propagation is so often missed.
"""
findings: list[Finding] = []
for doc_id in sorted(set(source) | set(index)):
src = source.get(doc_id)
idx = index.get(doc_id)
if src is not None and idx is None:
findings.append(Finding(doc_id, Drift.MISSING, "at source, not indexed"))
elif src is None and idx is not None:
findings.append(Finding(doc_id, Drift.ORPHANED, "deleted at source, still indexed"))
elif src.version != idx.version:
findings.append(
Finding(doc_id, Drift.STALE, f"index at {idx.version}, source at {src.version}")
)
else:
findings.append(Finding(doc_id, Drift.FRESH))
return FreshnessReport(findings=findings, checked_at=now)
def staleness_age(source: dict[str, Record], index: dict[str, Record], *, now: int) -> dict[str, int]:
"""How long each stale document has been stale, in logical ticks.
Freshness is a distribution, not a boolean. A p95 staleness age is what you
can actually put in an objective; "the index is fresh" is not measurable.
"""
ages: dict[str, int] = {}
for doc_id, src in source.items():
idx = index.get(doc_id)
if idx is not None and idx.version != src.version:
ages[doc_id] = max(0, now - src.updated_at)
return ages
def percentile(values: list[int], p: float) -> int:
"""Nearest-rank percentile. Returns 0 for an empty input.
The rank is `ceil(p/100 * N)`, which is the standard nearest-rank
definition. `round` is wrong here and wrong in a way that is easy to miss:
Python rounds halves to even, so `round(2.5)` is 2, and the p50 of
[1, 2, 3, 4, 100] comes out as 2 rather than 3.
"""
if not values:
return 0
ordered = sorted(values)
rank = max(1, min(len(ordered), math.ceil(p / 100 * len(ordered))))
return ordered[rank - 1]
@dataclass
class ReconcileResult:
indexed: list[str] = field(default_factory=list)
updated: list[str] = field(default_factory=list)
deleted: list[str] = field(default_factory=list)
def summary(self) -> str:
return (
f"indexed={len(self.indexed)} updated={len(self.updated)} "
f"deleted={len(self.deleted)}"
)
def reconcile(
source: dict[str, Record],
index: dict[str, Record],
report: FreshnessReport,
*,
allow_deletes: bool = True,
) -> ReconcileResult:
"""Bring the index into agreement with the source.
`allow_deletes` exists because deletion is the irreversible half. A
reconciliation run against a source that failed to enumerate correctly --
an auth error returning an empty list, say -- would otherwise delete the
entire index. Production systems gate deletes behind a sanity check; see
`safe_to_delete`.
"""
result = ReconcileResult()
for finding in report.findings:
if finding.drift is Drift.MISSING:
index[finding.doc_id] = source[finding.doc_id]
result.indexed.append(finding.doc_id)
elif finding.drift is Drift.STALE:
index[finding.doc_id] = source[finding.doc_id]
result.updated.append(finding.doc_id)
elif finding.drift is Drift.ORPHANED and allow_deletes:
del index[finding.doc_id]
result.deleted.append(finding.doc_id)
return result
def safe_to_delete(report: FreshnessReport, *, max_fraction: float = 0.25) -> bool:
"""Refuse a reconciliation that would delete an implausible share of the index.
If the source enumeration silently returned nothing, every indexed document
looks orphaned and a naive reconcile empties the index. The guard is crude
on purpose: a threshold a human can reason about beats a clever heuristic
nobody trusts at 3 a.m.
"""
indexed_total = report.count(Drift.FRESH) + report.count(Drift.STALE) + report.count(Drift.ORPHANED)
if indexed_total == 0:
return True
return (report.count(Drift.ORPHANED) / indexed_total) <= max_fraction
metadata.yml (730 bytes)
lesson_id: D325
day: 325
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/freshness_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-30'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3
examples/freshness_demo.py reproduced the documented output exactly; bash tests/run_tests.sh -> 17
passed in 0.01s. The untouched starter was verified to fail all 17. requires_network is false: source
and index are in-memory dictionaries and no test opens a socket.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/freshness.py (6573 bytes)
"""Index freshness: detecting drift between a source and an index, and fixing it.
Offline and standard-library only. The source and index are in-memory, but the
drift categories are the real ones, and the point of the lab is that they are
genuinely different problems with genuinely different fixes.
Four ways an index drifts from its source:
missing at the source, absent from the index -> a user cannot find it
stale in both, but the index holds an old copy -> answers cite old text
orphaned in the index, deleted at the source -> the serious one
fresh in both and identical -> nothing to do
Orphaned is the category that matters most and gets noticed least. A missing
document produces a complaint. An orphaned one produces a confident answer from
content that was deleted -- sometimes for legal reasons.
"""
from __future__ import annotations
import math
from dataclasses import dataclass, field
from enum import Enum
class Drift(str, Enum):
FRESH = "fresh"
MISSING = "missing"
STALE = "stale"
ORPHANED = "orphaned"
@dataclass(frozen=True)
class Record:
"""A document as either side holds it."""
doc_id: str
version: str # a content hash in production; a short token here
updated_at: int # logical clock, so tests are deterministic
@dataclass
class Finding:
doc_id: str
drift: Drift
detail: str = ""
@dataclass
class FreshnessReport:
findings: list[Finding] = field(default_factory=list)
checked_at: int = 0
def by_drift(self, drift: Drift) -> list[Finding]:
return [f for f in self.findings if f.drift is drift]
def count(self, drift: Drift) -> int:
return len(self.by_drift(drift))
def summary(self) -> str:
return (
f"fresh={self.count(Drift.FRESH)} missing={self.count(Drift.MISSING)} "
f"stale={self.count(Drift.STALE)} orphaned={self.count(Drift.ORPHANED)}"
)
def compare(source: dict[str, Record], index: dict[str, Record], *, now: int = 0) -> FreshnessReport:
"""Classify every document on either side into exactly one drift category.
The union of both key sets matters, not just the source: iterating the
source alone can never find an orphan, because an orphan is precisely a
document the source no longer mentions. That asymmetry is why deletion
propagation is so often missed.
"""
# TASK 1: classify every document into exactly one drift category.
# Iterate sorted(set(source) | set(index)) -- the UNION. Iterating the
# source alone can never find an orphan, because an orphan is precisely a
# document the source no longer mentions.
# at source, not indexed -> Drift.MISSING, "at source, not indexed"
# indexed, not at source -> Drift.ORPHANED, "deleted at source, still indexed"
# both, versions differ -> Drift.STALE, f"index at {i.version}, source at {s.version}"
# both, versions equal -> Drift.FRESH, no detail
raise NotImplementedError("implement compare")
def staleness_age(source: dict[str, Record], index: dict[str, Record], *, now: int) -> dict[str, int]:
"""How long each stale document has been stale, in logical ticks.
Freshness is a distribution, not a boolean. A p95 staleness age is what you
can actually put in an objective; "the index is fresh" is not measurable.
"""
# TASK 2: for each document present in BOTH sides with differing
# versions, return now - source.updated_at, floored at 0. Documents that
# are fresh, missing or orphaned do not appear in the result.
raise NotImplementedError("implement staleness_age")
def percentile(values: list[int], p: float) -> int:
"""Nearest-rank percentile. Returns 0 for an empty input.
The rank is `ceil(p/100 * N)`, which is the standard nearest-rank
definition. `round` is wrong here and wrong in a way that is easy to miss:
Python rounds halves to even, so `round(2.5)` is 2, and the p50 of
[1, 2, 3, 4, 100] comes out as 2 rather than 3.
"""
# TASK 3: nearest-rank percentile; return 0 for an empty list.
# rank = ceil(p/100 * N), clamped to [1, N], then return ordered[rank-1].
# Use math.ceil, NOT round: Python rounds halves to even, so round(2.5) is
# 2 and the p50 of [1,2,3,4,100] would come out as 2 rather than 3.
raise NotImplementedError("implement percentile")
@dataclass
class ReconcileResult:
indexed: list[str] = field(default_factory=list)
updated: list[str] = field(default_factory=list)
deleted: list[str] = field(default_factory=list)
def summary(self) -> str:
return (
f"indexed={len(self.indexed)} updated={len(self.updated)} "
f"deleted={len(self.deleted)}"
)
def reconcile(
source: dict[str, Record],
index: dict[str, Record],
report: FreshnessReport,
*,
allow_deletes: bool = True,
) -> ReconcileResult:
"""Bring the index into agreement with the source.
`allow_deletes` exists because deletion is the irreversible half. A
reconciliation run against a source that failed to enumerate correctly --
an auth error returning an empty list, say -- would otherwise delete the
entire index. Production systems gate deletes behind a sanity check; see
`safe_to_delete`.
"""
# TASK 4: apply the report.
# MISSING -> index[doc_id] = source[doc_id]; result.indexed.append(id)
# STALE -> index[doc_id] = source[doc_id]; result.updated.append(id)
# ORPHANED -> del index[doc_id] IF allow_deletes; result.deleted.append(id)
# Reconciling twice against an unchanged source must do nothing the second
# time.
raise NotImplementedError("implement reconcile")
def safe_to_delete(report: FreshnessReport, *, max_fraction: float = 0.25) -> bool:
"""Refuse a reconciliation that would delete an implausible share of the index.
If the source enumeration silently returned nothing, every indexed document
looks orphaned and a naive reconcile empties the index. The guard is crude
on purpose: a threshold a human can reason about beats a clever heuristic
nobody trusts at 3 a.m.
"""
# TASK 5: return False when the orphaned share of the INDEXED total
# (fresh + stale + orphaned) exceeds max_fraction. An empty index is safe.
# This is what stops a failed source enumeration -- one that returned
# nothing but looked successful -- from emptying the whole index.
raise NotImplementedError("implement safe_to_delete")
tests/run_tests.sh (205 bytes)
#!/usr/bin/env bash
# Run the day 325 index freshness test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_freshness.py (5273 bytes)
"""Grouped by drift category, so a failure names which one is misclassified.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from freshness import ( # noqa: E402
Drift,
Record,
compare,
percentile,
reconcile,
safe_to_delete,
staleness_age,
)
def rec(doc_id: str, version: str, updated_at: int = 0) -> Record:
return Record(doc_id, version, updated_at)
# ------------------------------------------------------------ classification
def test_identical_documents_are_fresh():
src = {"a": rec("a", "v1")}
idx = {"a": rec("a", "v1")}
report = compare(src, idx)
assert report.count(Drift.FRESH) == 1
assert report.summary() == "fresh=1 missing=0 stale=0 orphaned=0"
def test_document_only_at_source_is_missing():
report = compare({"a": rec("a", "v1")}, {})
assert report.by_drift(Drift.MISSING)[0].doc_id == "a"
def test_differing_version_is_stale():
report = compare({"a": rec("a", "v2")}, {"a": rec("a", "v1")})
finding = report.by_drift(Drift.STALE)[0]
assert finding.doc_id == "a"
assert "v1" in finding.detail and "v2" in finding.detail
def test_document_only_in_index_is_orphaned():
# The category a source-only scan can never find, because an orphan is
# exactly a document the source no longer mentions.
report = compare({}, {"gone": rec("gone", "v1")})
assert report.by_drift(Drift.ORPHANED)[0].doc_id == "gone"
def test_every_document_lands_in_exactly_one_category():
src = {"a": rec("a", "v1"), "b": rec("b", "v2")}
idx = {"b": rec("b", "v1"), "c": rec("c", "v1")}
report = compare(src, idx)
ids = [f.doc_id for f in report.findings]
assert sorted(ids) == ["a", "b", "c"]
assert len(ids) == len(set(ids))
def test_comparison_walks_the_union_not_just_the_source():
# Iterating the source alone would report a clean index here.
report = compare({}, {"x": rec("x", "v1"), "y": rec("y", "v1")})
assert report.count(Drift.ORPHANED) == 2
# ----------------------------------------------------------------- staleness
def test_staleness_age_measures_only_stale_documents():
src = {"a": rec("a", "v2", updated_at=10), "b": rec("b", "v1", updated_at=3)}
idx = {"a": rec("a", "v1", updated_at=4), "b": rec("b", "v1", updated_at=3)}
ages = staleness_age(src, idx, now=20)
assert set(ages) == {"a"}
assert ages["a"] == 10
def test_staleness_age_is_never_negative():
src = {"a": rec("a", "v2", updated_at=30)}
idx = {"a": rec("a", "v1", updated_at=1)}
assert staleness_age(src, idx, now=10)["a"] == 0
def test_percentile_of_empty_is_zero():
assert percentile([], 95) == 0
def test_percentile_picks_a_real_observation():
values = [1, 2, 3, 4, 100]
assert percentile(values, 95) == 100
assert percentile(values, 50) == 3
assert percentile(values, 100) == 100
# --------------------------------------------------------------- reconcile
def test_reconcile_fixes_every_category():
src = {"a": rec("a", "v1"), "b": rec("b", "v2")}
idx = {"b": rec("b", "v1"), "c": rec("c", "v1")}
report = compare(src, idx)
result = reconcile(src, idx, report)
assert result.indexed == ["a"]
assert result.updated == ["b"]
assert result.deleted == ["c"]
assert compare(src, idx).summary() == "fresh=2 missing=0 stale=0 orphaned=0"
def test_reconcile_is_idempotent():
src = {"a": rec("a", "v1"), "b": rec("b", "v2")}
idx = {"b": rec("b", "v1"), "c": rec("c", "v1")}
reconcile(src, idx, compare(src, idx))
snapshot = dict(idx)
second = reconcile(src, idx, compare(src, idx))
assert second.summary() == "indexed=0 updated=0 deleted=0"
assert idx == snapshot
def test_deletes_can_be_withheld():
src: dict[str, Record] = {}
idx = {"gone": rec("gone", "v1")}
result = reconcile(src, idx, compare(src, idx), allow_deletes=False)
assert result.deleted == []
assert "gone" in idx
# ------------------------------------------------------------- delete guard
def test_guard_blocks_an_implausible_mass_deletion():
# The source enumeration returned nothing -- an auth failure, say. Every
# indexed document now looks orphaned.
idx = {f"doc-{i}": rec(f"doc-{i}", "v1") for i in range(20)}
report = compare({}, idx)
assert report.count(Drift.ORPHANED) == 20
assert safe_to_delete(report) is False
def test_guard_allows_a_normal_amount_of_deletion():
src = {f"doc-{i}": rec(f"doc-{i}", "v1") for i in range(19)}
idx = dict(src)
idx["doc-gone"] = rec("doc-gone", "v1")
report = compare(src, idx)
assert report.count(Drift.ORPHANED) == 1
assert safe_to_delete(report) is True
def test_guard_permits_an_empty_index():
assert safe_to_delete(compare({}, {})) is True
def test_guard_threshold_is_configurable():
src = {"a": rec("a", "v1")}
idx = {"a": rec("a", "v1"), "b": rec("b", "v1"), "c": rec("c", "v1")}
report = compare(src, idx) # 2 of 3 orphaned
assert safe_to_delete(report, max_fraction=0.25) is False
assert safe_to_delete(report, max_fraction=0.9) is True
Troubleshooting
Troubleshooting — Day 325
No orphans are ever found
You are iterating source rather than set(source) | set(index). An orphan is by definition a document the source no longer mentions, so a source-driven loop can never visit it. This is the most common bug in this area and the reason deletions silently fail to propagate in real systems.
Check with:
python3 -c "
import sys; sys.path.insert(0,'examples')
from freshness import compare, Record, Drift
r = compare({}, {'gone': Record('gone','v1',0)})
print(r.count(Drift.ORPHANED)) # must be 1"
The p50 of [1, 2, 3, 4, 100] comes out as 2
You used round for the nearest-rank calculation. Python rounds halves to even, so round(2.5) is 2 and you land on the wrong observation. Nearest rank is math.ceil(p / 100 * N).
test_staleness_age_is_never_negative fails
A source document can carry an updated_at later than now — a clock skew, or a test constructing it deliberately. Floor the result at zero rather than returning a negative age.
Reconciliation is not idempotent
Either you are mutating the index while iterating a report built from the pre-reconcile state in a way that skips entries, or your version comparison is not symmetric with what you write. Recompute the report after reconciling and assert every document is fresh.
The guard blocks a run you expected to succeed
The threshold is a fraction of the indexed total — fresh plus stale plus orphaned — not of the source total. An index holding four documents with one orphan is at exactly 25 percent, which the default permits. Five documents with two orphans is 40 percent, which it does not.
ZeroDivisionError in safe_to_delete
An empty index is a normal state, especially on a first run. Return True when the indexed total is zero rather than dividing.
KeyError during reconcile
You are deleting from index while iterating over it, or looking up a MISSING document in index rather than in source. Iterate report.findings, which is a snapshot, and take the record from the side that actually has it.
NotImplementedError everywhere
Expected. The starter stubs all five functions, and every test depends on at least one — see expected-output/starter-run.txt.
Security notes
Security notes — Day 325
What this lab does and does not touch
- No network. Source and index are in-memory dictionaries. No test opens a socket, and
requires_networkisfalse. - No credentials. No API key, token or account anywhere.
- No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/. - No
sudo, no global installs.
The delete guard is a security control, not just a safety net
safe_to_delete looks like defensive programming, and it is also a genuine containment measure. Consider what a failed source enumeration looks like:
- an expired credential returning
200 OKwith an empty result set; - a paginated API that stops at page one because a cursor was dropped;
- a permissions change that silently narrows what the enumerating identity can see.
In every case the enumeration looks successful and reports far fewer documents than exist. Without a guard, reconciliation faithfully concludes that the missing documents were deleted and empties the index — while reporting complete success.
Refusing the whole run is the correct response rather than deleting a "safe" subset. If the enumeration is wrong, every classification derived from it is wrong, so partial action is still action on bad input.
Privacy: this is where erasure is actually honoured
Deletion propagation is the difference between a compliance story and a compliance failure. A record deleted from the primary system under a right-to-erasure request remains retrievable and quotable until the derived index also drops it.
Three consequences for real systems:
- Deletion deserves a shorter tolerance than staleness. Being an hour behind on edits is usually fine; being an hour behind on deletions may not be.
- It needs its own test and its own alert, not a line in a general freshness dashboard.
- Caches count too. An index that dropped the document but a response cache that did not is still serving it.
Access
A freshness check reads the complete id list from both sides, so it needs broad read access — worth a least-privilege review, and worth keeping separate from any identity that can write.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.