Capstone › Capstone Build II: Ship It › Day 364

Hands-on lab — Day 364: Capstone Retrospective

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/retro_demo.py

Test

bash tests/run_tests.sh

File tree

examples/retro_demo.py
examples/retro.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/retro.py
tests/run_tests.sh
tests/test_retro.py
troubleshooting.md

Lab README

Lab — Day 364: Capstone Retrospective

Lesson

  • Lesson title: Capstone Retrospective
  • Day number: 364 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-364-capstone-retrospective
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-364-capstone-retrospective when the site is running.

Purpose

Turn a project's record into findings that transfer to the next project: how wrong the estimates were and in which direction, whether that error is uniform or concentrated in one kind of work, and which incidents reached a user.

The tool does not judge whether the project was good. It measures what was predictable and was not predicted, which is the only part that carries forward.

Learning objectives

  • Compute a calibration ratio, and explain why it is a median rather than a mean.
  • Split the error by area and decide whether a single multiplier is a valid correction.
  • Rank incidents by detection stage and identify which escaped to production.
  • Separate a preventable escape, which names a gate, from a genuine unknown.
  • Produce sentences a reader can act on rather than scores.

Prerequisites

  • Day 363, "Portfolio, Resume, and Sharing Your Work".
  • Comfortable with Python dataclasses, enums and dictionaries.

Supported operating systems

macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.

Hardware requirements

Any machine running Python 3.10 or newer. No GPU. The suite completes in hundredths of a second.

Required software

  • Python 3.10 or newer.
  • pytest 7 or newer.

Free and open-source options

Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network — the input is a recorded list of tasks and incidents, so there is nothing to fetch.

Installation

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Removing .venv fully reverses the installation.

File structure

starter/retro.py        your work: eight stubbed tasks
examples/retro.py       reference implementation
examples/retro_demo.py  a capstone's record, turned into findings
tests/test_retro.py     grouped by what the retrospective computes
tests/run_tests.sh      suite entry point
expected-output/        real captured output and measured values
requirements/           pinned dependency

How to run

python3 examples/retro_demo.py   # analyse a capstone's record
bash tests/run_tests.sh          # run the suite

To work on the exercise, edit starter/retro.py, then copy it over the reference:

cp starter/retro.py examples/retro.py
bash tests/run_tests.sh

Keep a copy of the reference first if you want to compare.

What the commands do

python3 examples/retro_demo.py analyses seven tasks and five incidents from a capstone: it prints the calibration line, the median ratio per area, what a future estimate becomes under your own history, where each incident was caught, and the findings as sentences.

bash tests/run_tests.sh runs pytest over twenty-two tests grouped by what the retrospective computes: median, calibration, spread, multiplier, detection and findings.

Expected output

--- calibration ---
  median ratio 1.20x  under 6 / over 0  worst unfamiliar 3.33x  best familiar 1.15x
  uniform across areas: False
  median ratio by area:
    familiar     1.15x
    unfamiliar   3.33x
--- detection ---
  review=1 tests=1 staging=1 monitoring=1 user=1  escaped=2 (40%)
--- findings ---
  - Estimates ran 1.20x long at the median. Multiply the next one by 1.20 before
    committing to it.
  - The error is concentrated, not uniform: unfamiliar ran 3.33x while familiar
    ran 1.15x. A single multiplier will not fix this.

The second finding contradicts the first, deliberately. Both are true, and the second is the one that changes what you do — there is no task in this record for which 1.20x was the right multiplier.

expected-output/FIELDS.md explains every field, and expected-output/measured-values.txt records the threshold sensitivity.

Validation steps

  1. bash tests/run_tests.sh reports 22 passed.
  2. The demo must report uniform across areas: False. If it reports True, is_uniform is comparing areas against the overall median instead of against each other.
  3. The median of [1.0, 1.1, 1.2, 1.3, 8.0] must be 1.2. If you get a value above 2.0, you are computing a mean.
  4. Five incidents, one per stage, must yield escaped=2. Staging is deployed and reached nobody, so it is not an escape.
  5. A clean record — accurate estimates, nothing escaped — must produce the "no systematic pattern" line rather than a manufactured finding.

Tests

Twenty-two tests in six groups:

  • median — one extreme task does not move it; even counts average the middle two; an empty list is zero rather than an error.
  • calibration — the ratio is actual over estimated; a zero estimate does not divide by zero; direction is counted separately from size; the worst and best areas are named; empty input returns an empty result.
  • spread — a uniform error and a concentrated one are distinguished; a single area is trivially uniform; each area gets its own median.
  • multiplier — your own history applied to a new estimate.
  • detection — incidents are counted by stage; only production incidents escape; preventable escapes are separated from unknowns; no incidents is a zero rate rather than an error.
  • findings — the multiplier is stated when estimates ran long and padding when they ran short; a concentrated error is called out specifically; each preventable escape names its gate; a clean record says so.

Cleanup

rm -rf __pycache__ .pytest_cache
rm -rf .venv

Both are safe to remove and fully reversible.

Troubleshooting

See troubleshooting.md.

Security notes

See security.md. In short: no network, no credentials — and a retrospective is a document that describes, in order, what your system got wrong and which controls are still missing.

Extension exercises

  1. Group by estimate size. Under two hours, two to eight, over eight. If large estimates are worse, break work down; if small ones are worse, you are underestimating fixed overheads that do not scale down. The two findings have opposite fixes.
  2. Run it on your own capstone, with the real numbers.
  3. Split the escapes honestly. For each incident that reached production, decide whether a gate would genuinely have caught it. If everything looks preventable in hindsight, you are hindsight-biasing — the test is what you would have needed to know at the time to justify building the gate before the incident.
  • Lesson
  • Previous: Day 363 — Portfolio, Resume, and Sharing Your Work
  • Next: Day 365 — Graduation: Your AI Roadmap Going Forward

Expected output

FIELDS.md

# Reading the retrospective output

## The calibration line

    median ratio 1.20x  under 6 / over 0  worst unfamiliar 3.33x  best familiar 1.15x

| Part | Meaning |
| --- | --- |
| median ratio | Actual over estimated across every task, at the median. 1.0 would be exact. |
| under / over | How many tasks ran long and how many ran short. A task at exactly 1.0 counts as neither. |
| worst | The area with the highest median ratio, and that ratio. |
| best | The area with the lowest. |

Read the median first and the last two second. If the worst and best are far
apart, the median is an average of two populations and describes neither.

## The uniformity flag

    uniform across areas: False

| Value | Meaning | What to do |
| --- | --- | --- |
| `True` | The highest area is within 1.5x of the lowest. | Apply the median as a multiplier to the next estimate. |
| `False` | The error lives in one kind of work. | Do not apply a single multiplier. Estimate the bad area differently, or split it until the pieces resemble work you have done. |

The 1.5x threshold is a judgement, not a law. It is the point at which two
areas stop being noisy versions of the same number.

## The future-estimate table

          4h estimated  ->    4.8h expected

Your own history applied to a new estimate. Note what this table is on this
record: computed from the overall median of 1.20x, and therefore **wrong for
the unfamiliar work**, where the record says 3.33x. The tool computes it; you
decide whether the uniformity flag permits you to use it.

## The detection summary

    review=1 tests=1 staging=1 monitoring=1 user=1  escaped=2 (40%)

| Stage | Meaning | Escaped |
| --- | --- | --- |
| `review` | Caught before it ran anywhere. | no |
| `tests` | Caught before it deployed. | no |
| `staging` | Deployed, no user affected. | no |
| `monitoring` | In production, a signal found it. | yes |
| `user` | In production, a person found it. | yes |

`escaped` counts the last two. Staging is deliberately not an escape: the code
was deployed, and it reached nobody.

## The findings

Sentences rather than scores, because a score is not an instruction. Four kinds
appear:

| Finding | Trigger |
| --- | --- |
| the multiplier | Median ratio above 1.0. |
| padding | Median ratio below 1.0 — estimates were conservative, which costs opportunities rather than deadlines. |
| concentrated error | The uniformity check failed. This one contradicts the multiplier finding, deliberately. |
| a named gate | One line per escaped incident that has a `preventable_by` set. |

If none of them fire, the report says the record is clean rather than
manufacturing a finding — which is the behaviour a retrospective tool most
often gets wrong.

examples-run.txt

--- calibration ---
  median ratio 1.20x  under 6 / over 0  worst unfamiliar 3.33x  best familiar 1.15x
  uniform across areas: False
  median ratio by area:
    familiar     1.15x
    unfamiliar   3.33x
--- what a future estimate becomes ---
      4h estimated  ->    4.8h expected
     10h estimated  ->   12.0h expected
     40h estimated  ->   48.0h expected
--- detection ---
  review=1 tests=1 staging=1 monitoring=1 user=1  escaped=2 (40%)
--- findings ---
  - Estimates ran 1.20x long at the median. Multiply the next one by 1.20 before committing to it.
  - The error is concentrated, not uniform: unfamiliar ran 3.33x while familiar ran 1.15x. A single multiplier will not fix this — estimate unfamiliar work differently.
  - 2 of 5 incidents reached production (40%).
  - 'spend spike from a retry loop' reached production and a per-request cap would have caught it.
  - 'answers citing nothing after an index rebuild' reached production and a groundedness signal would have caught it.

measured-values.txt

Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.

  python3 examples/retro_demo.py -> 18 lines, exit 0
  bash tests/run_tests.sh        -> 22 passed in 0.01s
  the untouched starter          -> 22 failed in 0.06s

Seven tasks and five incidents.

The number this lab exists for:

  median ratio          1.20x
  familiar work         1.15x
  unfamiliar work       3.33x
  uniform across areas  False

1.20x is a real measurement of the project and a poor description of any task
in it. There is no task for which 1.20x was the right multiplier -- it is the
average of two populations that behave differently. The finding worth carrying
is not "I underestimate", it is "I estimate familiar work well and unfamiliar
work by a factor of three", which suggests a different practice.

The future-estimate table is computed from that same 1.20x, and so it is wrong
for exactly the tasks you most need it for: it predicts 4.8h, and the one
unfamiliar 4h task in the record ("deployment and rollback") actually ran 15h.
The tool prints the table; the uniformity flag is what tells you not to use it
here.

Detection, five incidents, one per stage:

  review=1 tests=1 staging=1 monitoring=1 user=1  escaped=2 (40%)

Both escapes have a named gate:

  'spend spike from a retry loop'                  a per-request cap
  'answers citing nothing after an index rebuild'  a groundedness signal

Those two sentences are the next project's backlog. "Improve monitoring" would
have produced neither, and both map onto checks introduced on day 360 and day
361 -- which is what it looks like when a review's findings are confirmed by
events rather than by argument.

One note on the threshold. is_uniform compares the highest area median against
the lowest, and calls the error uniform when the ratio between them is at most
1.5x. On this record that ratio is 2.909, so the check reports False -- but it
would report True at any threshold of 3.0 or above. Measured:

  spread=1.5  ->  False
  spread=2.0  ->  False
  spread=2.9  ->  False
  spread=3.0  ->  True

The threshold is a judgement rather than a law, and it is worth knowing how
close your own record sits to it. What is not a judgement is WHAT gets
compared: the areas must be compared to each other. Comparing them to the
overall median answers a different and less useful question, because the
overall median is itself pulled by whichever area contributed more tasks.

run-output.txt

--- calibration ---
  median ratio 1.20x  under 6 / over 0  worst unfamiliar 3.33x  best familiar 1.15x
  uniform across areas: False
  median ratio by area:
    familiar     1.15x
    unfamiliar   3.33x
--- what a future estimate becomes ---
      4h estimated  ->    4.8h expected
     10h estimated  ->   12.0h expected
     40h estimated  ->   48.0h expected
--- detection ---
  review=1 tests=1 staging=1 monitoring=1 user=1  escaped=2 (40%)
--- findings ---
  - Estimates ran 1.20x long at the median. Multiply the next one by 1.20 before committing to it.
  - The error is concentrated, not uniform: unfamiliar ran 3.33x while familiar ran 1.15x. A single multiplier will not fix this — estimate unfamiliar work differently.
  - 2 of 5 incidents reached production (40%).
  - 'spend spike from a retry loop' reached production and a per-request cap would have caught it.
  - 'answers citing nothing after an index rebuild' reached production and a groundedness signal would have caught it.

starter-run.txt

An untouched starter fails the entire suite:

    22 failed in 0.06s

Every test reaches one of the eight NotImplementedError stubs. Unlike some
earlier labs there is no test that passes by accident, because the dataclasses
alone compute nothing -- Task.ratio is itself task 1.

The first failure to fix is TASK 2, the median. Nothing else can be checked
until it returns numbers, and the whole point of the lab is that it is a median
rather than a mean.

test-output.txt

......................                                                   [100%]
22 passed in 0.01s

test-run.txt

......................                                                   [100%]
22 passed in 0.01s

Source files

examples/retro_demo.py (1921 bytes)
#!/usr/bin/env python3
"""A capstone's record, turned into findings."""

from __future__ import annotations

import os
import sys

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from retro import (
    Caught,
    Incident,
    Task,
    apply_multiplier,
    by_area,
    calibration,
    detection,
    findings,
    is_uniform,
)

TASKS = [
    Task("ingestion pipeline", "familiar", 8, 9),
    Task("chunking and index", "familiar", 6, 7),
    Task("retrieval endpoint", "familiar", 4, 4),
    Task("eval harness", "familiar", 5, 6),
    Task("agent tool calling", "unfamiliar", 6, 20),
    Task("streaming frontend", "unfamiliar", 8, 26),
    Task("deployment and rollback", "unfamiliar", 4, 15),
]

INCIDENTS = [
    Incident("chunk ids not stable, index doubled", Caught.TESTS),
    Incident("secret committed to the repository", Caught.REVIEW),
    Incident("readiness probe missing, served errors", Caught.STAGING),
    Incident("spend spike from a retry loop", Caught.MONITORING, preventable_by="a per-request cap"),
    Incident("answers citing nothing after an index rebuild", Caught.USER,
             preventable_by="a groundedness signal"),
]


def main() -> int:
    cal = calibration(TASKS)
    print("--- calibration ---")
    print(f"  {cal.line()}")
    print(f"  uniform across areas: {is_uniform(TASKS)}")
    print("  median ratio by area:")
    for area, ratio in sorted(by_area(TASKS).items()):
        print(f"    {area:<12} {ratio:.2f}x")

    print("--- what a future estimate becomes ---")
    for hours in (4, 10, 40):
        print(f"    {hours:>3}h estimated  ->  {apply_multiplier(hours, cal):>5}h expected")

    print("--- detection ---")
    print(f"  {detection(INCIDENTS).summary()}")

    print("--- findings ---")
    for finding in findings(TASKS, INCIDENTS):
        print(f"  - {finding}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
examples/retro.py (7710 bytes)
"""Turn a project's record into findings you can act on.

Offline and standard-library only. The input is a list of tasks with estimates
and actuals, plus the incidents that occurred, so nothing is fetched.

A retrospective usually produces feelings: "the API work took longer than
expected", "we should test more". Feelings do not transfer to the next project.
Numbers do, and three of them are computable from a record you already have:

  calibration    how wrong your estimates were, and IN WHICH DIRECTION
  bias by area   whether the error is uniform or concentrated somewhere
  detection      which incidents a gate caught, and which reached a user

The direction matters more than the magnitude. Being uniformly 2x over is a
multiplier you can apply. Being accurate on familiar work and 5x over on
unfamiliar work is a different problem with a different fix -- and the average
of the two hides both.

Nothing here judges whether the project was good. It measures what was
predictable and was not predicted, which is the only part that transfers.
"""

from __future__ import annotations

import math
from dataclasses import dataclass, field
from enum import Enum


class Caught(str, Enum):
    """Where an incident was detected. Ordered by how much it cost."""

    REVIEW = "review"  # before it ran anywhere
    TESTS = "tests"  # before it deployed
    STAGING = "staging"  # deployed, no user affected
    MONITORING = "monitoring"  # in production, found by a signal
    USER = "user"  # in production, found by a person


# How much later each stage is than the one before it. Not a cost in money --
# a rank, so "further right" is comparable across projects.
STAGE_RANK = {
    Caught.REVIEW: 0,
    Caught.TESTS: 1,
    Caught.STAGING: 2,
    Caught.MONITORING: 3,
    Caught.USER: 4,
}


@dataclass(frozen=True)
class Task:
    name: str
    area: str
    estimated_hours: float
    actual_hours: float

    @property
    def ratio(self) -> float:
        """Actual over estimated. 1.0 is perfect, 2.0 is twice as long."""
        return self.actual_hours / self.estimated_hours if self.estimated_hours else 0.0


@dataclass(frozen=True)
class Incident:
    what: str
    caught_at: Caught
    preventable_by: str = ""  # the gate that SHOULD have caught it, if any


@dataclass
class Calibration:
    median_ratio: float
    worst_area: str
    worst_area_ratio: float
    best_area: str
    best_area_ratio: float
    underestimated: int
    overestimated: int

    def line(self) -> str:
        return (
            f"median ratio {self.median_ratio:.2f}x  "
            f"under {self.underestimated} / over {self.overestimated}  "
            f"worst {self.worst_area} {self.worst_area_ratio:.2f}x  "
            f"best {self.best_area} {self.best_area_ratio:.2f}x"
        )


def median(values: list[float]) -> float:
    """Median rather than mean, because one task that ran 8x over would
    dominate an average and make the whole estimate look worse than it was."""
    if not values:
        return 0.0
    ordered = sorted(values)
    mid = len(ordered) // 2
    if len(ordered) % 2:
        return ordered[mid]
    return (ordered[mid - 1] + ordered[mid]) / 2


def by_area(tasks: list[Task]) -> dict[str, float]:
    """Median ratio per area of work."""
    areas: dict[str, list[float]] = {}
    for task in tasks:
        areas.setdefault(task.area, []).append(task.ratio)
    return {area: median(ratios) for area, ratios in areas.items()}


def calibration(tasks: list[Task]) -> Calibration:
    """How wrong the estimates were, and where the error concentrates."""
    if not tasks:
        return Calibration(0.0, "", 0.0, "", 0.0, 0, 0)
    ratios = [t.ratio for t in tasks]
    areas = by_area(tasks)
    worst = max(areas, key=lambda a: (areas[a], a))
    best = min(areas, key=lambda a: (areas[a], a))
    return Calibration(
        median_ratio=median(ratios),
        worst_area=worst,
        worst_area_ratio=areas[worst],
        best_area=best,
        best_area_ratio=areas[best],
        underestimated=sum(1 for r in ratios if r > 1.0),
        overestimated=sum(1 for r in ratios if r < 1.0),
    )


def is_uniform(tasks: list[Task], *, spread: float = 1.5) -> bool:
    """Whether the estimation error is spread evenly across areas.

    A uniform error is a multiplier you can apply next time. A concentrated one
    means you estimate familiar work well and unfamiliar work badly, which
    needs a different fix -- and the overall median hides which you have.
    """
    areas = by_area(tasks)
    if len(areas) < 2:
        return True
    values = list(areas.values())
    lo, hi = min(values), max(values)
    return hi <= lo * spread if lo > 0 else False


def apply_multiplier(estimate: float, cal: Calibration) -> float:
    """What this estimate becomes once your own history is applied."""
    return round(estimate * cal.median_ratio, 1)


@dataclass
class DetectionReport:
    counts: dict[Caught, int] = field(default_factory=dict)
    escaped: list[Incident] = field(default_factory=list)
    preventable: list[Incident] = field(default_factory=list)

    @property
    def escape_rate(self) -> float:
        total = sum(self.counts.values())
        return len(self.escaped) / total if total else 0.0

    def summary(self) -> str:
        order = " ".join(f"{s.value}={self.counts.get(s, 0)}" for s in Caught)
        return f"{order}  escaped={len(self.escaped)} ({self.escape_rate:.0%})"


def detection(incidents: list[Incident]) -> DetectionReport:
    """Where problems were caught, and which ones reached a user.

    "Escaped" means monitoring or a person found it in production. Those are
    the incidents worth spending the next project's effort on, because they are
    the ones your gates did not see.
    """
    counts: dict[Caught, int] = {}
    for incident in incidents:
        counts[incident.caught_at] = counts.get(incident.caught_at, 0) + 1
    escaped = [i for i in incidents if STAGE_RANK[i.caught_at] >= STAGE_RANK[Caught.MONITORING]]
    preventable = [i for i in escaped if i.preventable_by]
    return DetectionReport(counts=counts, escaped=escaped, preventable=preventable)


def findings(tasks: list[Task], incidents: list[Incident]) -> list[str]:
    """The retrospective, as sentences that transfer to the next project."""
    out: list[str] = []
    cal = calibration(tasks)

    if cal.median_ratio > 1.0:
        out.append(
            f"Estimates ran {cal.median_ratio:.2f}x long at the median. "
            f"Multiply the next one by {cal.median_ratio:.2f} before committing to it."
        )
    elif cal.median_ratio and cal.median_ratio < 1.0:
        out.append(
            f"Estimates were {1 / cal.median_ratio:.2f}x conservative at the median — "
            "padding, which costs opportunities rather than deadlines."
        )

    if tasks and not is_uniform(tasks):
        out.append(
            f"The error is concentrated, not uniform: {cal.worst_area} ran "
            f"{cal.worst_area_ratio:.2f}x while {cal.best_area} ran {cal.best_area_ratio:.2f}x. "
            "A single multiplier will not fix this — estimate unfamiliar work differently."
        )

    report = detection(incidents)
    if report.escaped:
        out.append(
            f"{len(report.escaped)} of {sum(report.counts.values())} incidents reached production "
            f"({report.escape_rate:.0%})."
        )
    for incident in report.preventable:
        out.append(f"'{incident.what}' reached production and {incident.preventable_by} would have caught it.")

    if not out:
        out.append("No systematic pattern in the record. Keep the estimates and the gates as they are.")
    return out
metadata.yml (862 bytes)
lesson_id: D364
day: 364
kind: python
languages:
  - python
setup_commands:
  - pip install -r requirements/requirements.txt
run_commands:
  - python examples/retro_demo.py
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/retro_demo.py
  reproduced the documented figures exactly (median 1.20x, familiar 1.15x, unfamiliar 3.33x, escaped 2
  of 5); bash tests/run_tests.sh -> 22 passed in 0.01s. The untouched starter fails all 22, since
  Task.ratio is itself one of the eight stubs and nothing computes without it. requires_network is
  false: the input is a recorded list of tasks and incidents, so nothing is fetched.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/retro.py (6389 bytes)
"""STARTER -- implement the eight tasks below, then run the tests.

Turn a project's record into findings you can act on.

Offline and standard-library only. The input is a list of tasks with estimates
and actuals, plus the incidents that occurred, so nothing is fetched.

A retrospective usually produces feelings: "the API work took longer than
expected", "we should test more". Feelings do not transfer to the next project.
Numbers do, and three of them are computable from a record you already have:

  calibration    how wrong your estimates were, and IN WHICH DIRECTION
  bias by area   whether the error is uniform or concentrated somewhere
  detection      which incidents a gate caught, and which reached a user

The direction matters more than the magnitude. Being uniformly 2x over is a
multiplier you can apply. Being accurate on familiar work and 5x over on
unfamiliar work is a different problem with a different fix -- and the average
of the two hides both.

Nothing here judges whether the project was good. It measures what was
predictable and was not predicted, which is the only part that transfers.
"""

from __future__ import annotations

import math
from dataclasses import dataclass, field
from enum import Enum


class Caught(str, Enum):
    """Where an incident was detected. Ordered by how much it cost."""

    REVIEW = "review"  # before it ran anywhere
    TESTS = "tests"  # before it deployed
    STAGING = "staging"  # deployed, no user affected
    MONITORING = "monitoring"  # in production, found by a signal
    USER = "user"  # in production, found by a person


# How much later each stage is than the one before it. Not a cost in money --
# a rank, so "further right" is comparable across projects.
STAGE_RANK = {
    Caught.REVIEW: 0,
    Caught.TESTS: 1,
    Caught.STAGING: 2,
    Caught.MONITORING: 3,
    Caught.USER: 4,
}


@dataclass(frozen=True)
class Task:
    name: str
    area: str
    estimated_hours: float
    actual_hours: float

    @property
    def ratio(self) -> float:
        """Actual over estimated. 1.0 is perfect, 2.0 is twice as long.

        TASK 1. Guard the zero estimate -- a task nobody estimated is a real
        case in a real record, and it must not raise.
        """
        raise NotImplementedError("TASK 1: return actual over estimated, 0.0 if no estimate")


@dataclass(frozen=True)
class Incident:
    what: str
    caught_at: Caught
    preventable_by: str = ""  # the gate that SHOULD have caught it, if any


@dataclass
class Calibration:
    median_ratio: float
    worst_area: str
    worst_area_ratio: float
    best_area: str
    best_area_ratio: float
    underestimated: int
    overestimated: int

    def line(self) -> str:
        return (
            f"median ratio {self.median_ratio:.2f}x  "
            f"under {self.underestimated} / over {self.overestimated}  "
            f"worst {self.worst_area} {self.worst_area_ratio:.2f}x  "
            f"best {self.best_area} {self.best_area_ratio:.2f}x"
        )


def median(values: list[float]) -> float:
    """Median rather than mean, because one task that ran 8x over would
    dominate an average and make the whole estimate look worse than it was."""
    raise NotImplementedError(
        "TASK 2: return the median. Even counts average the middle two; "
        "an empty list is 0.0. Do NOT use the mean."
    )


def by_area(tasks: list[Task]) -> dict[str, float]:
    """Median ratio per area of work."""
    raise NotImplementedError("TASK 3: group ratios by task.area, then median each group")


def calibration(tasks: list[Task]) -> Calibration:
    """How wrong the estimates were, and where the error concentrates."""
    raise NotImplementedError(
        "TASK 4: build a Calibration. Empty input returns an empty Calibration "
        "rather than raising. Count DIRECTION separately -- a task at exactly "
        "1.0 is neither under nor over."
    )


def is_uniform(tasks: list[Task], *, spread: float = 1.5) -> bool:
    """Whether the estimation error is spread evenly across areas.

    A uniform error is a multiplier you can apply next time. A concentrated one
    means you estimate familiar work well and unfamiliar work badly, which
    needs a different fix -- and the overall median hides which you have.
    """
    raise NotImplementedError(
        "TASK 5: compare the HIGHEST area against the LOWEST, not against the "
        "overall median. Fewer than two areas is trivially uniform."
    )


def apply_multiplier(estimate: float, cal: Calibration) -> float:
    """What this estimate becomes once your own history is applied."""
    raise NotImplementedError("TASK 6: apply the median ratio, rounded to one decimal")


@dataclass
class DetectionReport:
    counts: dict[Caught, int] = field(default_factory=dict)
    escaped: list[Incident] = field(default_factory=list)
    preventable: list[Incident] = field(default_factory=list)

    @property
    def escape_rate(self) -> float:
        total = sum(self.counts.values())
        return len(self.escaped) / total if total else 0.0

    def summary(self) -> str:
        order = " ".join(f"{s.value}={self.counts.get(s, 0)}" for s in Caught)
        return f"{order}  escaped={len(self.escaped)} ({self.escape_rate:.0%})"


def detection(incidents: list[Incident]) -> DetectionReport:
    """Where problems were caught, and which ones reached a user.

    "Escaped" means monitoring or a person found it in production. Those are
    the incidents worth spending the next project's effort on, because they are
    the ones your gates did not see.
    """
    raise NotImplementedError(
        "TASK 7: count by stage, and collect the escapes. Use STAGE_RANK -- an "
        "incident caught in STAGING was deployed but reached nobody, so it is "
        "NOT an escape."
    )


def findings(tasks: list[Task], incidents: list[Incident]) -> list[str]:
    """The retrospective, as sentences that transfer to the next project."""
    raise NotImplementedError(
        "TASK 8: produce sentences, not scores. Cover (a) the multiplier when "
        "estimates ran long, or the padding when they ran short; (b) a "
        "concentrated error, naming both areas; (c) the escape count; (d) one "
        "line per preventable escape naming the gate. If none of those apply, "
        "say the record is clean rather than inventing a finding."
    )

tests/run_tests.sh (203 bytes)
#!/usr/bin/env bash
# Run the day 364 retrospective test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_retro.py (5933 bytes)
"""Grouped by what the retrospective computes.

Run with: bash tests/run_tests.sh
"""

from __future__ import annotations

import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))

from retro import (  # noqa: E402
    Caught,
    Incident,
    Task,
    apply_multiplier,
    by_area,
    calibration,
    detection,
    findings,
    is_uniform,
    median,
)


def tasks_uniform():
    return [
        Task("a", "backend", 4, 8),
        Task("b", "backend", 6, 12),
        Task("c", "frontend", 5, 10),
        Task("d", "frontend", 3, 6),
    ]


def tasks_concentrated():
    return [
        Task("a", "familiar", 8, 9),
        Task("b", "familiar", 6, 7),
        Task("c", "unfamiliar", 6, 20),
        Task("d", "unfamiliar", 8, 26),
    ]


# ------------------------------------------------------------------- median


def test_median_ignores_one_extreme_task():
    # A single 8x task would drag a mean far above what most tasks did.
    values = [1.0, 1.1, 1.2, 1.3, 8.0]
    assert median(values) == 1.2
    assert sum(values) / len(values) > 2.0


def test_median_of_an_even_count_averages_the_middle_two():
    assert median([1.0, 2.0, 3.0, 4.0]) == 2.5


def test_median_of_nothing_is_zero():
    assert median([]) == 0.0


# -------------------------------------------------------------- calibration


def test_ratio_is_actual_over_estimated():
    assert Task("t", "a", 4, 8).ratio == 2.0
    assert Task("t", "a", 8, 4).ratio == 0.5


def test_a_zero_estimate_does_not_divide_by_zero():
    assert Task("t", "a", 0, 5).ratio == 0.0


def test_calibration_counts_direction_not_just_size():
    # Direction is the part that transfers. Uniformly over is a multiplier;
    # a mix is a different problem.
    cal = calibration([Task("a", "x", 4, 8), Task("b", "x", 8, 4), Task("c", "x", 4, 4)])
    assert cal.underestimated == 1
    assert cal.overestimated == 1


def test_calibration_names_the_worst_and_best_areas():
    cal = calibration(tasks_concentrated())
    assert cal.worst_area == "unfamiliar"
    assert cal.best_area == "familiar"
    assert cal.worst_area_ratio > cal.best_area_ratio


def test_calibration_of_no_tasks_is_empty_rather_than_an_error():
    cal = calibration([])
    assert cal.median_ratio == 0.0 and cal.worst_area == ""


# ----------------------------------------------------------------- spread


def test_uniform_error_is_recognised():
    assert is_uniform(tasks_uniform())


def test_concentrated_error_is_recognised():
    # THE finding of this lab: a median of 1.20x across the project hides
    # familiar work at 1.15x and unfamiliar work at 3.33x.
    assert not is_uniform(tasks_concentrated())


def test_a_single_area_is_trivially_uniform():
    assert is_uniform([Task("a", "only", 4, 12)])


def test_by_area_reports_a_median_per_area():
    areas = by_area(tasks_concentrated())
    assert set(areas) == {"familiar", "unfamiliar"}
    assert areas["unfamiliar"] > 3.0
    assert areas["familiar"] < 1.3


# ------------------------------------------------------------- multiplier


def test_the_multiplier_applies_your_own_history():
    cal = calibration(tasks_uniform())  # every task ran exactly 2x
    assert cal.median_ratio == 2.0
    assert apply_multiplier(10, cal) == 20.0


# ------------------------------------------------------------- detection


def test_incidents_are_counted_by_where_they_were_caught():
    report = detection(
        [
            Incident("a", Caught.REVIEW),
            Incident("b", Caught.TESTS),
            Incident("c", Caught.USER),
        ]
    )
    assert report.counts[Caught.REVIEW] == 1
    assert report.counts[Caught.USER] == 1


def test_only_production_incidents_count_as_escaped():
    report = detection(
        [
            Incident("caught early", Caught.TESTS),
            Incident("caught in staging", Caught.STAGING),
            Incident("found by a metric", Caught.MONITORING),
            Incident("found by a person", Caught.USER),
        ]
    )
    assert len(report.escaped) == 2
    assert report.escape_rate == 0.5


def test_preventable_escapes_are_separated_out():
    report = detection(
        [
            Incident("spend spike", Caught.MONITORING, preventable_by="a per-request cap"),
            Incident("unknown unknown", Caught.USER),
        ]
    )
    assert len(report.escaped) == 2
    assert len(report.preventable) == 1
    assert report.preventable[0].preventable_by == "a per-request cap"


def test_no_incidents_is_a_zero_escape_rate_not_an_error():
    assert detection([]).escape_rate == 0.0


# -------------------------------------------------------------- findings


def test_findings_state_the_multiplier_when_estimates_ran_long():
    out = findings(tasks_uniform(), [])
    assert any("2.00x" in f and "Multiply" in f for f in out)


def test_findings_call_out_concentrated_error_specifically():
    out = findings(tasks_concentrated(), [])
    assert any("concentrated" in f and "unfamiliar" in f for f in out)
    assert any("single multiplier will not fix this" in f for f in out)


def test_findings_report_padding_when_estimates_were_conservative():
    generous = [Task("a", "x", 10, 5), Task("b", "x", 8, 4)]
    out = findings(generous, [])
    assert any("conservative" in f for f in out)


def test_findings_name_each_preventable_escape():
    out = findings(
        tasks_uniform(),
        [Incident("spend spike", Caught.MONITORING, preventable_by="a per-request cap")],
    )
    assert any("a per-request cap would have caught it" in f for f in out)


def test_a_clean_record_says_so_rather_than_inventing_a_finding():
    clean = [Task("a", "x", 4, 4), Task("b", "x", 6, 6)]
    out = findings(clean, [Incident("caught early", Caught.TESTS)])
    assert out == ["No systematic pattern in the record. Keep the estimates and the gates as they are."]

Troubleshooting

Troubleshooting — Day 364

NotImplementedError on every test

Expected. The starter stubs eight functions and Task.ratio is one of them, so nothing computes until task 1 is done. See expected-output/starter-run.txt.

Fix them in order. Task 2, the median, is the one everything else depends on.

One bad task dominates the calibration

You are computing a mean. A single task that ran eight times over will pull an average far above the typical task and make the whole project look worse than it was, which is exactly the distortion the median exists to avoid.

test_median_of_an_even_count_averages_the_middle_two fails

Your median returns ordered[len(ordered) // 2] unconditionally. For an even count there is no single middle element, so average the two either side.

is_uniform returns True for the concentrated fixture

You are comparing each area against the overall median. Compare the highest area against the lowest. The overall median is itself pulled by whichever area contributed more tasks, so measuring against it answers a weaker question.

Measured on the demo record, the highest-to-lowest ratio is 2.909, so a 1.5x threshold reports False. If yours reports True, check the threshold as well as the comparison — anything at 3.0 or above calls this record uniform.

Every incident counts as escaped

STAGING is not an escape. The code was deployed and reached nobody, which is the whole reason a staging environment exists. Only MONITORING and USER are escapes, and using STAGE_RANK to compare against MONITORING keeps that decision in one place.

A task with a zero estimate raises ZeroDivisionError

A task nobody estimated is a real entry in a real record, so ratio must return 0.0 rather than raising. This is task 1 for a reason.

test_calibration_counts_direction_not_just_size fails

A task at exactly 1.0 is neither under-estimated nor over-estimated. If your counts add up to the number of tasks on a record containing an exact estimate, you are using >= or <= somewhere.

worst_area and best_area come back the same

With a single area that is correct — the highest and lowest of one value are the same value. On multi-area input, check that you are taking max and min over the area medians rather than over the raw task ratios.

apply_multiplier returns a long float

Round to one decimal. 4 * 1.2 is 4.800000000000001 in binary floating point, and an estimate printed to fifteen decimal places suggests a precision the number does not have.

The findings list is empty on a project that clearly went badly

Check that findings calls calibration and detection rather than reimplementing them, and that the concentrated-error branch is not nested inside the multiplier branch. On a record where estimates were accurate but incidents escaped, the multiplier finding correctly does not fire and the escape findings still must.

The report invents a finding on a clean record

The opposite failure, and the more damaging one. If nothing fired, say so. A retrospective tool that always produces a finding trains people to ignore its findings.

Security notes

Security notes — Day 364

What this lab does and does not touch

  • No network. The input is a recorded list of tasks and incidents. Nothing is fetched.
  • No credentials. No API key, token or account.
  • No writes outside this directory. Only __pycache__/, .pytest_cache/ and, if you create it, .venv/.
  • No sudo, no global installs.
  • The example record is fictional. The task names and incidents are representative of a capstone rather than drawn from any real project.

A retrospective is a document about incidents

That is the security point of the day, and it is easy to miss because a retrospective feels like process rather than infrastructure.

The findings you are producing describe, in order: what your system got wrong, which controls were missing, and how long the gap stayed open. That is a useful document for you and a useful document for somebody attacking the system. Three consequences worth holding:

  • Describe the incident, do not reproduce it. "Answers cited nothing after an index rebuild" is a finding. Pasting the offending answer, the prompt that produced it, or the document it should have cited puts user content into a file that may be published.
  • A recorded incident may contain personal data. If a user reported it, their report may quote their own query. Summarise it; do not carry it forward verbatim into a public write-up.
  • An unfixed finding is a disclosed weakness. "A per-request cap would have caught it" says plainly that there is currently no per-request cap. If the retrospective is going to be public — in a portfolio, say — either fix the gap first or describe it at a level that does not amount to instructions.

Blamelessness is a data-quality control, not only a kindness

A review that names individuals produces defensive accounts, and defensive accounts are inaccurate ones. If the record you are analysing was gathered in an environment where an honest answer carries a cost, the estimates and incident reports in it are already distorted and no amount of arithmetic recovers them. This matters for a team retrospective more than a solo one, and it is worth stating before the numbers are collected rather than after.

Reversing everything this lab did

rm -rf __pycache__ .pytest_cache .venv

Nothing else was created, and nothing outside this directory was modified.