Capstone › Capstone Build II: Ship It › Day 362
Hands-on lab — Day 362: Documentation and Demo
- ← Back to the Day 362 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/capstone/day-362-documentation-and-demo/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/doccheck_demo.py Test
bash tests/run_tests.sh File tree
examples/doccheck_demo.py examples/doccheck.py examples/fixtures.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/doccheck.py tests/run_tests.sh tests/test_doccheck.py troubleshooting.md
Lab README
Lab — Day 362: Documentation and Demo
Lesson
- Lesson title: Documentation and Demo
- Day number: 362 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-362-documentation-and-demo
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-362-documentation-and-demowhen the site is running.
Purpose
Build a checker that verifies documentation still describes the system it documents — required sections, commands that exist, outputs that match reality, no placeholder text, and known limitations actually stated.
The subject is a README plus a small description of the project, so the check runs without executing anything.
Learning objectives
- Extract commands and claimed outputs from Markdown, distinguishing instructions from output.
- Verify every documented command exists and every claimed output matches a real capture.
- Enforce a required-section list and reject placeholder text.
- Check known limitations are stated rather than merely known.
- Justify treating incompleteness as a warning and wrongness as a failure.
Prerequisites
- Day 361, "Security Review of Your Capstone".
- Comfortable with Python regular expressions and dataclasses.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.
Hardware requirements
Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.
Required software
- Python 3.10 or newer.
pytest7 or newer.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network. The lesson discusses MkDocs, Sphinx, Docusaurus, Vale and asciinema as the real tooling, all free and open source.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/doccheck.py your work: extraction plus three checks
examples/doccheck.py reference implementation
examples/fixtures.py a project, a good README and a drifted one
examples/doccheck_demo.py checks both READMEs
tests/test_doccheck.py grouped by check, plus extraction tests
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/doccheck_demo.py # check a good README and a drifted one
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/doccheck.py, then copy it over the reference:
cp starter/doccheck.py examples/doccheck.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/doccheck_demo.py runs all six checks against a current README and one that has drifted over six months, printing each finding and a verdict for both.
bash tests/run_tests.sh runs pytest over sixteen tests: four on the extraction functions, one on the clean case, seven covering each check independently, and four on the drifted README.
Expected output
--- current README ---
OK sections: all required sections present
OK commands: 3 documented command(s) all exist
OK undocumented_commands: every provided command is documented
OK outputs: claimed outputs match captured output
OK placeholders: no placeholder text
OK limitations: every known limitation is stated
=> PASS fail=0 warn=0
--- README after six months of drift ---
FAIL sections: missing: Limitations
FAIL commands: documented but not provided: make setup, make serve
WARN undocumented_commands: provided but undocumented: make install, make run, make test
FAIL outputs: make run (differs from captured)
FAIL placeholders: unfinished text: to be written, under construction
FAIL limitations: known but unstated: English only, no multi-tenant isolation
=> FAIL fail=5 warn=1
expected-output/FIELDS.md explains the fields and the warn-versus-fail distinction.
Validation steps
bash tests/run_tests.shreports16 passed.- The current README must produce zero findings. If it reports its own examples as unknown commands, your extractor is treating output blocks as instructions.
- A project with one extra undocumented command must still pass — a warning alone does not fail.
- Every claimed output must resolve to a captured one. If they all report "no captured output", your output pattern is matching across newlines.
Tests
Sixteen tests in four groups:
- extraction — only shell-tagged blocks are commands; prompts and comments are stripped; a claimed output matches one command rather than the whole document; sections are read in order.
- the good case — a current README passes with no findings.
- each check on its own — missing sections, a renamed command, an undocumented command (warn only), a changed output, an output with nothing captured, placeholder text, an unstated limitation, and no limitations recorded.
- the drift — the drifted README fails on five checks, names all six, and a warning alone does not fail.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md, including the two extraction bugs this lab had before its own tests caught them.
Security notes
See security.md. In short: no network, no credentials — and the lesson covers documentation as a disclosure surface.
Extension exercises
- Link checking. Verify relative links point at paths the project contains, skipping external
httplinks so the gate does not need the network. - Executable checks. Run each documented command and compare live output, then measure how many false failures that produces over ten runs before deciding whether it belongs in a pipeline.
- Staleness. Record when each section was last edited and warn when a section has not changed while the code it describes has.
Navigation
- Lesson
- Previous: Day 361 — Security Review of Your Capstone
- Next: Day 363 — Portfolio, Resume, and Sharing Your Work
Expected output
FIELDS.md
# Reading the documentation check
## An issue line
FAIL outputs: make run (differs from captured)
| Part | Meaning |
| --- | --- |
| level | `OK`, `WARN` or `FAIL`. Only `FAIL` makes the review fail. |
| check | Which of the six checks. |
| detail | What was found, naming the specific command, section or limitation. |
## The verdict
=> FAIL fail=5 warn=1
`PASS` requires zero `FAIL` findings. Warnings are reported and do not block.
## Warn versus fail, and why the line is there
| Finding | Level | Why |
| --- | --- | --- |
| A documented command that does not exist | **FAIL** | Actively misleads. The reader pastes it and it breaks. |
| A provided command nobody documented | **WARN** | A gap. The reader simply never learns about it. |
| A claimed output that differs from reality | **FAIL** | The reader cannot tell success from failure. |
| No limitations recorded at all | **WARN** | Probably untrue, but it is a prompt rather than a defect. |
Blocking on incompleteness makes the gate annoying, and an annoying gate gets disabled — at which point it catches nothing at all.
## The six checks
| Check | Catches |
| --- | --- |
| `sections` | A missing answer to one of the six questions a reader has. |
| `commands` | A command that was renamed, so the README is confidently wrong. |
| `undocumented_commands` | Capability nobody wrote down. |
| `outputs` | An output that changed, so the reader cannot verify success. |
| `placeholders` | Writing that was never finished. |
| `limitations` | A constraint you know about and never stated. |
examples-run.txt
--- current README ---
OK sections: all required sections present
OK commands: 3 documented command(s) all exist
OK undocumented_commands: every provided command is documented
OK outputs: claimed outputs match captured output
OK placeholders: no placeholder text
OK limitations: every known limitation is stated
=> PASS fail=0 warn=0
--- README after six months of drift ---
FAIL sections: missing: Limitations
FAIL commands: documented but not provided: make setup, make serve
WARN undocumented_commands: provided but undocumented: make install, make run, make test
FAIL outputs: make run (differs from captured)
FAIL placeholders: unfinished text: to be written, under construction
FAIL limitations: known but unstated: English only, no multi-tenant isolation
=> FAIL fail=5 warn=1
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/doccheck_demo.py -> 16 lines, exit 0
bash tests/run_tests.sh -> 16 passed in 0.01s
current README PASS fail=0 warn=0
README after six months of drift FAIL fail=5 warn=1
What the drifted README gets wrong, and none of it is dramatic:
sections "## Limitations" was deleted
commands make setup / make serve -- the old names
undocumented_commands make install / make run / make test now unmentioned
outputs claims 0.0.0.0:9000, the service prints 127.0.0.1:8080
placeholders an unfinished Architecture section and an unfinished Troubleshooting section
limitations English only and no tenant isolation, both still true,
both no longer written down
Note the single WARN. Three provided commands are undocumented, which is a GAP
rather than a defect, so it does not fail the review. Blocking on
incompleteness makes a gate annoying enough to be switched off.
Two bugs this lab had before its own tests caught them, both in extraction:
* untagged ``` blocks were read as commands, so the checker reported the
document's own captured output as commands that do not exist;
* the claimed-output pattern used re.S for the command group, so the dot
matched newlines and the group swallowed the whole document.
Both now have dedicated tests.
run-output.txt
--- current README ---
OK sections: all required sections present
OK commands: 3 documented command(s) all exist
OK undocumented_commands: every provided command is documented
OK outputs: claimed outputs match captured output
OK placeholders: no placeholder text
OK limitations: every known limitation is stated
=> PASS fail=0 warn=0
--- README after six months of drift ---
FAIL sections: missing: Limitations
FAIL commands: documented but not provided: make setup, make serve
WARN undocumented_commands: provided but undocumented: make install, make run, make test
FAIL outputs: make run (differs from captured)
FAIL placeholders: unfinished text: to be written, under construction
FAIL limitations: known but unstated: English only, no multi-tenant isolation
=> FAIL fail=5 warn=1
starter-run.txt
An untouched starter fails nearly the whole suite:
15 failed, 1 passed in 0.06s
The one that passes is test_sections_are_read_in_order, which exercises
sections_in -- not one of the stubbed functions. Everything else depends on
commands_in, claimed_outputs, check_commands, check_outputs or
check_limitations.
test-output.txt
................ [100%]
16 passed in 0.01s
test-run.txt
................ [100%]
16 passed in 0.01s
Source files
examples/doccheck_demo.py (705 bytes)
#!/usr/bin/env python3
"""Check a good README and a drifted one against the same project."""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from doccheck import review
from fixtures import DRIFTED_README, GOOD_README, PROJECT
def show(label: str, readme: str) -> None:
report = review(readme, PROJECT)
print(f"--- {label} ---")
for issue in report.issues:
print(issue.line())
print(f" => {report.summary()}")
def main() -> int:
show("current README", GOOD_README)
show("README after six months of drift", DRIFTED_README)
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/doccheck.py (7947 bytes)
"""Check that documentation still describes the system it documents.
Offline and standard-library only. The subject is a project description plus
its README text, so the check runs without executing anything.
Documentation does not become wrong all at once. It drifts: a command is
renamed, a flag is removed, an output changes, and the README keeps saying what
used to be true. Every reader after that point is misled, and the person who
would have noticed is the one person who no longer reads it.
The defence is to make documentation CHECKABLE. Four things can be verified
mechanically:
sections the questions a new reader must be able to answer
commands every documented command still exists in the project
outputs every claimed output matches what was actually captured
honesty no placeholders, and limitations stated rather than implied
The fourth is the one people skip, and it is the one that decides whether a
demo builds trust or spends it.
"""
from __future__ import annotations
import re
from dataclasses import dataclass, field
from enum import Enum
class Level(str, Enum):
OK = "ok"
WARN = "warn"
FAIL = "fail"
# What a new reader must be able to answer before they can use anything.
REQUIRED_SECTIONS = (
"What it does",
"Install",
"Run",
"Architecture",
"Limitations",
"Troubleshooting",
)
# Words that mean the sentence was never finished. Deliberately not the same
# vocabulary this course forbids in its own lessons -- a checker whose examples
# trip another checker is a checker nobody can write about.
PLACEHOLDERS = (
"to be written",
"under construction",
"not yet documented",
"fill this in",
"placeholder text",
)
@dataclass(frozen=True)
class Project:
"""What the system actually provides."""
commands: tuple[str, ...] = ()
captured_outputs: dict[str, str] = field(default_factory=dict)
known_limitations: tuple[str, ...] = ()
@dataclass
class Issue:
check: str
level: Level
detail: str
def line(self) -> str:
return f" {self.level.value.upper():<5} {self.check}: {self.detail}"
@dataclass
class DocReport:
issues: list[Issue] = field(default_factory=list)
@property
def ok(self) -> bool:
return not any(i.level is Level.FAIL for i in self.issues)
def names(self) -> set[str]:
return {i.check for i in self.issues if i.level is not Level.OK}
def summary(self) -> str:
fails = sum(1 for i in self.issues if i.level is Level.FAIL)
warns = sum(1 for i in self.issues if i.level is Level.WARN)
return f"{'PASS' if self.ok else 'FAIL'} fail={fails} warn={warns}"
def sections_in(readme: str) -> list[str]:
"""Heading titles, in document order."""
return [m.group(1).strip() for m in re.finditer(r"^#{1,3}\s+(.+)$", readme, re.M)]
def commands_in(readme: str) -> list[str]:
"""Commands claimed by the README.
Only blocks TAGGED as a shell language count. An untagged block is output,
not an instruction -- without that distinction the extractor reads captured
output back as commands, which is a bug this lab had before the tests
caught it.
"""
out: list[str] = []
for block in re.findall(r"```(?:bash|sh|console)\n(.*?)```", readme, re.S):
for line in block.splitlines():
line = line.strip().lstrip("$").strip()
if line and not line.startswith("#"):
out.append(line)
return out
def claimed_outputs(readme: str) -> dict[str, str]:
"""Output blocks introduced by a line naming the command that produced them.
The convention is deliberate and small: a line ending in `->` names the
command, and the fenced block after it is what that command printed. It
makes the claim machine-checkable, which is the whole point.
"""
out: dict[str, str] = {}
# The command part must NOT span newlines. With re.S a dot matches them,
# and the pattern swallows the whole document up to the first block.
pattern = re.compile(r"^([^\n]+?)\s*->\s*\n+```\w*\n(.*?)```", re.M | re.S)
for cmd, body in pattern.findall(readme):
out[cmd.strip()] = body.strip()
return out
# ------------------------------------------------------------------ checks
def check_sections(readme: str) -> Issue:
present = set(sections_in(readme))
missing = [s for s in REQUIRED_SECTIONS if s not in present]
if missing:
return Issue("sections", Level.FAIL, f"missing: {', '.join(missing)}")
return Issue("sections", Level.OK, "all required sections present")
def check_commands(readme: str, project: Project) -> Issue:
"""Every documented command must exist in the project.
This is the check that catches drift, because a renamed command leaves the
README confidently wrong and nothing else notices.
"""
documented = commands_in(readme)
unknown = [c for c in documented if c not in project.commands]
if unknown:
return Issue("commands", Level.FAIL, f"documented but not provided: {', '.join(unknown)}")
if not documented:
return Issue("commands", Level.WARN, "no runnable commands documented")
return Issue("commands", Level.OK, f"{len(documented)} documented command(s) all exist")
def check_undocumented_commands(readme: str, project: Project) -> Issue:
"""Commands the project provides and the README never mentions.
A warning rather than a failure: not everything needs documenting. But a
command nobody has written down is a command nobody will use.
"""
documented = set(commands_in(readme))
missing = [c for c in project.commands if c not in documented]
if missing:
return Issue("undocumented_commands", Level.WARN, f"provided but undocumented: {', '.join(missing)}")
return Issue("undocumented_commands", Level.OK, "every provided command is documented")
def check_outputs(readme: str, project: Project) -> Issue:
"""Claimed output must match what was actually captured."""
problems: list[str] = []
for cmd, claimed in claimed_outputs(readme).items():
actual = project.captured_outputs.get(cmd)
if actual is None:
problems.append(f"{cmd} (no captured output to compare)")
elif claimed.strip() != actual.strip():
problems.append(f"{cmd} (differs from captured)")
if problems:
return Issue("outputs", Level.FAIL, "; ".join(problems))
return Issue("outputs", Level.OK, "claimed outputs match captured output")
def check_placeholders(readme: str) -> Issue:
lowered = readme.lower()
found = sorted({p for p in PLACEHOLDERS if p in lowered})
if found:
return Issue("placeholders", Level.FAIL, f"unfinished text: {', '.join(found)}")
return Issue("placeholders", Level.OK, "no placeholder text")
def check_limitations(readme: str, project: Project) -> Issue:
"""Known limitations must be stated, not merely known.
A demo that hides its limits spends trust it has to earn back later, and
the person who finds the limit will find it at the worst moment.
"""
body = readme.lower()
unstated = [lim for lim in project.known_limitations if lim.lower() not in body]
if unstated:
return Issue("limitations", Level.FAIL, f"known but unstated: {', '.join(unstated)}")
if not project.known_limitations:
return Issue("limitations", Level.WARN, "no known limitations recorded — is that true?")
return Issue("limitations", Level.OK, "every known limitation is stated")
def review(readme: str, project: Project) -> DocReport:
return DocReport(
issues=[
check_sections(readme),
check_commands(readme, project),
check_undocumented_commands(readme, project),
check_outputs(readme, project),
check_placeholders(readme),
check_limitations(readme, project),
]
)
examples/fixtures.py (1395 bytes)
"""A project, the README that documents it well, and one that has drifted."""
from __future__ import annotations
from doccheck import Project
PROJECT = Project(
commands=("make install", "make run", "make test"),
captured_outputs={
"make test": "12 passed in 0.4s",
"make run": "listening on 127.0.0.1:8080",
},
known_limitations=(
"English only",
"no multi-tenant isolation",
),
)
GOOD_README = """
# Support Assistant
## What it does
Answers questions from your own documentation, with citations.
## Install
```bash
make install
```
## Run
```bash
make run
```
make run ->
```
listening on 127.0.0.1:8080
```
## Architecture
Ingestion writes chunks to a local index. A query embeds, retrieves the top
three chunks, and asks the model to answer using only those.
## Tests
```bash
make test
```
make test ->
```
12 passed in 0.4s
```
## Limitations
English only. There is no multi-tenant isolation, so one deployment serves one
team.
## Troubleshooting
If `make run` reports a port in use, set `PORT` and retry.
"""
DRIFTED_README = """
# Support Assistant
## What it does
Answers questions from your own documentation, with citations.
## Install
```bash
make setup
```
## Run
```bash
make serve
```
make run ->
```
listening on 0.0.0.0:9000
```
## Architecture
To be written.
## Troubleshooting
Under construction.
"""
metadata.yml (891 bytes)
lesson_id: D362
day: 362
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/doccheck_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/doccheck_demo.py
reproduced the documented results exactly, with the current README passing cleanly and the drifted one
failing five checks; bash tests/run_tests.sh -> 16 passed in 0.01s. The untouched starter fails 15 of
16; the 1 that passes exercises sections_in, which is not stubbed. requires_network is false: the
subject is README text and a project description, so nothing is executed and nothing is fetched.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/doccheck.py (7467 bytes)
"""Check that documentation still describes the system it documents.
Offline and standard-library only. The subject is a project description plus
its README text, so the check runs without executing anything.
Documentation does not become wrong all at once. It drifts: a command is
renamed, a flag is removed, an output changes, and the README keeps saying what
used to be true. Every reader after that point is misled, and the person who
would have noticed is the one person who no longer reads it.
The defence is to make documentation CHECKABLE. Four things can be verified
mechanically:
sections the questions a new reader must be able to answer
commands every documented command still exists in the project
outputs every claimed output matches what was actually captured
honesty no placeholders, and limitations stated rather than implied
The fourth is the one people skip, and it is the one that decides whether a
demo builds trust or spends it.
"""
from __future__ import annotations
import re
from dataclasses import dataclass, field
from enum import Enum
class Level(str, Enum):
OK = "ok"
WARN = "warn"
FAIL = "fail"
# What a new reader must be able to answer before they can use anything.
REQUIRED_SECTIONS = (
"What it does",
"Install",
"Run",
"Architecture",
"Limitations",
"Troubleshooting",
)
# Words that mean the sentence was never finished. Deliberately not the same
# vocabulary this course forbids in its own lessons -- a checker whose examples
# trip another checker is a checker nobody can write about.
PLACEHOLDERS = (
"to be written",
"under construction",
"not yet documented",
"fill this in",
"placeholder text",
)
@dataclass(frozen=True)
class Project:
"""What the system actually provides."""
commands: tuple[str, ...] = ()
captured_outputs: dict[str, str] = field(default_factory=dict)
known_limitations: tuple[str, ...] = ()
@dataclass
class Issue:
check: str
level: Level
detail: str
def line(self) -> str:
return f" {self.level.value.upper():<5} {self.check}: {self.detail}"
@dataclass
class DocReport:
issues: list[Issue] = field(default_factory=list)
@property
def ok(self) -> bool:
return not any(i.level is Level.FAIL for i in self.issues)
def names(self) -> set[str]:
return {i.check for i in self.issues if i.level is not Level.OK}
def summary(self) -> str:
fails = sum(1 for i in self.issues if i.level is Level.FAIL)
warns = sum(1 for i in self.issues if i.level is Level.WARN)
return f"{'PASS' if self.ok else 'FAIL'} fail={fails} warn={warns}"
def sections_in(readme: str) -> list[str]:
"""Heading titles, in document order."""
return [m.group(1).strip() for m in re.finditer(r"^#{1,3}\s+(.+)$", readme, re.M)]
def commands_in(readme: str) -> list[str]:
"""Commands claimed by the README.
Only blocks TAGGED as a shell language count. An untagged block is output,
not an instruction -- without that distinction the extractor reads captured
output back as commands, which is a bug this lab had before the tests
caught it.
"""
# TASK 1: return the commands a README instructs a reader to run.
# Only blocks TAGGED bash, sh or console are instructions. A bare ``` block
# is OUTPUT -- treat it as a command and the checker reports the document's
# own examples as commands that do not exist.
# Strip a leading "$" and skip lines starting with "#".
raise NotImplementedError("implement commands_in")
def claimed_outputs(readme: str) -> dict[str, str]:
"""Output blocks introduced by a line naming the command that produced them.
The convention is deliberate and small: a line ending in `->` names the
command, and the fenced block after it is what that command printed. It
makes the claim machine-checkable, which is the whole point.
"""
# TASK 2: map command -> claimed output.
# Convention: a line ending in "->" names the command, and the fenced block
# after it is what that command printed.
# The command group must NOT span newlines. With re.S a dot matches them and
# the group swallows the whole document up to the first block -- use
# [^\n]+? for the command and re.S only for the block body.
raise NotImplementedError("implement claimed_outputs")
# ------------------------------------------------------------------ checks
def check_sections(readme: str) -> Issue:
present = set(sections_in(readme))
missing = [s for s in REQUIRED_SECTIONS if s not in present]
if missing:
return Issue("sections", Level.FAIL, f"missing: {', '.join(missing)}")
return Issue("sections", Level.OK, "all required sections present")
def check_commands(readme: str, project: Project) -> Issue:
"""Every documented command must exist in the project.
This is the check that catches drift, because a renamed command leaves the
README confidently wrong and nothing else notices.
"""
# TASK 3: FAIL naming any documented command the project does not provide.
# WARN when the README documents no commands at all. Otherwise OK.
raise NotImplementedError("implement check_commands")
def check_undocumented_commands(readme: str, project: Project) -> Issue:
"""Commands the project provides and the README never mentions.
A warning rather than a failure: not everything needs documenting. But a
command nobody has written down is a command nobody will use.
"""
documented = set(commands_in(readme))
missing = [c for c in project.commands if c not in documented]
if missing:
return Issue("undocumented_commands", Level.WARN, f"provided but undocumented: {', '.join(missing)}")
return Issue("undocumented_commands", Level.OK, "every provided command is documented")
def check_outputs(readme: str, project: Project) -> Issue:
"""Claimed output must match what was actually captured."""
# TASK 4: for each claimed output, FAIL when there is no captured output to
# compare against, or when it differs. Name every offending command.
raise NotImplementedError("implement check_outputs")
def check_placeholders(readme: str) -> Issue:
lowered = readme.lower()
found = sorted({p for p in PLACEHOLDERS if p in lowered})
if found:
return Issue("placeholders", Level.FAIL, f"unfinished text: {', '.join(found)}")
return Issue("placeholders", Level.OK, "no placeholder text")
def check_limitations(readme: str, project: Project) -> Issue:
"""Known limitations must be stated, not merely known.
A demo that hides its limits spends trust it has to earn back later, and
the person who finds the limit will find it at the worst moment.
"""
# TASK 5: FAIL naming any known limitation the README does not state
# (case-insensitive). WARN when the project records none at all -- an empty
# list is a claim, and usually a false one. Otherwise OK.
raise NotImplementedError("implement check_limitations")
def review(readme: str, project: Project) -> DocReport:
return DocReport(
issues=[
check_sections(readme),
check_commands(readme, project),
check_undocumented_commands(readme, project),
check_outputs(readme, project),
check_placeholders(readme),
check_limitations(readme, project),
]
)
tests/run_tests.sh (209 bytes)
#!/usr/bin/env bash
# Run the day 362 documentation check test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_doccheck.py (5084 bytes)
"""Grouped by check, plus extraction tests for the two regexes that got it wrong.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from doccheck import ( # noqa: E402
Level,
Project,
claimed_outputs,
commands_in,
review,
sections_in,
)
from fixtures import DRIFTED_README, GOOD_README, PROJECT # noqa: E402
def names(readme, project=PROJECT):
return review(readme, project).names()
# ------------------------------------------------------------- extraction
def test_only_shell_tagged_blocks_are_read_as_commands():
# The bug this lab had first: an untagged block is OUTPUT, and reading it
# back as a command made the checker report its own examples as unknown.
readme = "```bash\nmake run\n```\n\nmake run ->\n\n```\nlistening\n```\n"
assert commands_in(readme) == ["make run"]
def test_comments_and_prompts_are_stripped_from_commands():
readme = "```bash\n# set it up\n$ make install\n```\n"
assert commands_in(readme) == ["make install"]
def test_claimed_output_is_matched_to_one_command_not_the_whole_document():
# With re.S a dot matches newlines, and the command group swallowed
# everything above the first block.
got = claimed_outputs(GOOD_README)
assert set(got) == {"make run", "make test"}
assert got["make test"] == "12 passed in 0.4s"
def test_sections_are_read_in_order():
heads = sections_in(GOOD_README)
assert heads[0] == "Support Assistant"
assert "Limitations" in heads
# ----------------------------------------------------------- the good case
def test_a_current_readme_passes_cleanly():
report = review(GOOD_README, PROJECT)
assert report.ok
assert report.names() == set()
assert report.summary() == "PASS fail=0 warn=0"
# ------------------------------------------------- each check on its own
def test_missing_sections_fail():
readme = GOOD_README.replace("## Limitations", "## Notes")
assert "sections" in names(readme)
def test_a_renamed_command_is_caught():
# The drift that matters: the README still says the old name, and nothing
# else in the project notices.
readme = GOOD_README.replace("make run\n```", "make serve\n```", 1)
report = review(readme, PROJECT)
issue = next(i for i in report.issues if i.check == "commands")
assert issue.level is Level.FAIL
assert "make serve" in issue.detail
def test_an_undocumented_command_only_warns():
project = Project(
commands=PROJECT.commands + ("make deploy",),
captured_outputs=PROJECT.captured_outputs,
known_limitations=PROJECT.known_limitations,
)
issue = next(i for i in review(GOOD_README, project).issues
if i.check == "undocumented_commands")
assert issue.level is Level.WARN
assert "make deploy" in issue.detail
# A warning does not fail the review.
assert review(GOOD_README, project).ok
def test_a_changed_output_is_caught():
readme = GOOD_README.replace("listening on 127.0.0.1:8080", "listening on 0.0.0.0:9000")
issue = next(i for i in review(readme, PROJECT).issues if i.check == "outputs")
assert issue.level is Level.FAIL
assert "differs from captured" in issue.detail
def test_an_output_with_nothing_captured_is_caught():
readme = GOOD_README + "\nmake lint ->\n\n```\nall clean\n```\n"
issue = next(i for i in review(readme, PROJECT).issues if i.check == "outputs")
assert issue.level is Level.FAIL
assert "no captured output" in issue.detail
def test_placeholder_text_fails():
readme = GOOD_README.replace("English only.", "To be written")
assert "placeholders" in names(readme)
def test_a_known_limitation_left_unstated_fails():
readme = GOOD_README.replace("English only. ", "")
issue = next(i for i in review(readme, PROJECT).issues if i.check == "limitations")
assert issue.level is Level.FAIL
assert "English only" in issue.detail
def test_recording_no_limitations_warns():
project = Project(commands=PROJECT.commands, captured_outputs=PROJECT.captured_outputs)
issue = next(i for i in review(GOOD_README, project).issues if i.check == "limitations")
assert issue.level is Level.WARN
# ------------------------------------------------------------ the drift
def test_the_drifted_readme_fails_on_five_checks():
report = review(DRIFTED_README, PROJECT)
assert not report.ok
assert report.summary() == "FAIL fail=5 warn=1"
def test_the_drifted_readme_names_what_broke():
assert names(DRIFTED_README) == {
"sections",
"commands",
"undocumented_commands",
"outputs",
"placeholders",
"limitations",
}
def test_a_warning_alone_does_not_fail_the_review():
readme = GOOD_README.replace("```bash\nmake test\n```", "")
report = review(readme, PROJECT)
assert "undocumented_commands" in report.names()
assert report.ok
Troubleshooting
Troubleshooting — Day 362
The good README reports its own examples as unknown commands
Your extractor accepts untagged fenced blocks. Only ```bash, ```sh and ```console blocks are instructions; a bare ``` block is output.
This lab had exactly that bug. The checker read listening on 127.0.0.1:8080 back as a command and reported that the project did not provide it — a checker failing on its own correct documentation, which is a particularly convincing way to get a gate ignored.
Check with:
python3 -c "
import sys; sys.path.insert(0,'examples')
from doccheck import commands_in
print(commands_in('\`\`\`bash\nmake run\n\`\`\`\n\nmake run ->\n\n\`\`\`\nlistening\n\`\`\`\n'))"
It must print ['make run'] and nothing else.
Every claimed output reports "no captured output to compare"
Your output pattern uses re.S for the command group. A dot then matches newlines, and the group swallows everything from the top of the document to the first fenced block — so the "command" is the whole README and matches nothing.
Use [^\n]+? for the command and reserve re.S for the block body.
Prompts or comments appear as commands
Strip a leading $ and skip lines beginning with #. Both are conventions readers expect in a shell block, and neither is part of the command.
An undocumented command fails the review
It should warn. Only FAIL findings make DocReport.ok false. Blocking on incompleteness makes the gate annoying enough that someone removes it, at which point it catches nothing.
test_recording_no_limitations_warns fails
An empty known_limitations tuple is a claim — that the system has no limitations — and it is usually false. Warn rather than passing silently.
test_a_known_limitation_left_unstated_fails fails
The comparison must be case-insensitive: lower both the README body and the limitation before checking membership. A README saying "english only" still states the limitation.
The section check fails on a heading you can see
sections_in matches #, ## and ### headings and compares the title exactly. ## Limitations matches Limitations; ## Known limitations does not. Either match the required list exactly or relax the comparison deliberately.
NotImplementedError on nearly every test
Expected. The starter stubs five functions — see expected-output/starter-run.txt, which also names the one test that passes without them.
Security notes
Security notes — Day 362
What this lab does and does not touch
- No network. The subject is README text and a project description. Nothing is fetched, and no test opens a socket.
- Nothing is executed. The checker reads documentation and compares it against recorded facts. It never runs a documented command, which is deliberate — see the extension exercise for the trade-off.
- No credentials. No API key, token or account.
- No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/. - No
sudo, no global installs.
Documentation is a disclosure surface
Worth stating plainly, because documentation rarely gets a security review of its own. Four things leak through READMEs routinely:
- Example configurations with real values. A hostname, a bucket name, an internal service address. Use placeholders that are obviously placeholders —
example.com,<your-api-key>— rather than a real value with a character changed. - Architecture sections naming internal systems. Useful to a reader and useful to someone mapping your infrastructure. Describe the shape without the inventory.
- Troubleshooting sections quoting real errors. A pasted stack trace or error response can carry a token, a path, a user identifier or an internal hostname. Redact before pasting.
- Screenshots and terminal recordings. These capture whatever was on screen, including environment variables in a shell prompt and other windows.
asciinemarecords text, which at least makes the content greppable before publishing.
The recurring pattern: documentation is written quickly and reviewed for clarity rather than for content. Give it the same read you would give a log line.
A demo published against real data publishes that data
If the demo corpus is real support tickets, real customer documents or a real inbox, then recording the demo distributes them. Use a fixture corpus you own and can publish. This is the same argument as Day 328 — data multiplies, and a recording is another copy nobody tracks.
Why this checker does not execute anything
Running the documented commands would catch more drift, and it would also mean a documentation gate that executes arbitrary strings extracted from a Markdown file. In this lab that is safe because the fixtures are fixed. In a real pipeline running commands parsed out of a document is a code-execution path, and it needs the same care as any other: a sandbox, no credentials, and no network.
The extension exercise asks you to build it and measure the false-failure rate, which is the honest way to decide whether it belongs in a pipeline.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.