Capstone › Capstone Build II: Ship It › Day 359
Hands-on lab — Day 359: Deploying Your Capstone
- ← Back to the Day 359 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/capstone/day-359-deploying-your-capstone/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/release_demo.py Test
bash tests/run_tests.sh File tree
examples/release_demo.py examples/release.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/release.py tests/run_tests.sh tests/test_release.py troubleshooting.md
Lab README
Lab — Day 359: Deploying Your Capstone
Lesson
- Lesson title: Deploying Your Capstone
- Day number: 359 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-359-deploying-your-capstone
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-359-deploying-your-capstonewhen the site is running.
Purpose
Implement the release path — preflight, health, canary, promotion and rollback — and watch the same defect cost a different amount depending on which gate catches it.
There is no container runtime and no cloud here. What is modelled is the decision structure of a deployment, which is where deployments actually go wrong.
Learning objectives
- Build a preflight gate that reports every blocker at once and deploys nothing when it fails.
- Separate liveness from readiness, and check dependencies.
- Decide a canary on an error budget rather than on elapsed time.
- Implement rollback as a reversible pointer swap.
- Explain why an irreversible migration blocks the whole path.
Prerequisites
- Day 358, "Frontend and User Experience".
- Comfortable with Python dataclasses and enums.
- Familiar with the idea of a container image and a health endpoint, though neither is used directly.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.
Hardware requirements
Any machine running Python 3.10 or newer. No GPU, no container runtime. The suite completes in hundredths of a second.
Required software
- Python 3.10 or newer.
pytest7 or newer.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. No account, no API key, no network. The lesson discusses Docker Compose, Kubernetes, Nomad, Argo Rollouts and Flagger as the real open-source options, and Fly.io, Railway, Render and Cloud Run as managed ones with free tiers.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/release.py your work: preflight, deploy, rollback
examples/release.py reference implementation
examples/release_demo.py five deployments and a rollback
tests/test_release.py grouped by gate
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/release_demo.py # five releases, each stopping differently
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/release.py, then copy it over the reference:
cp starter/release.py examples/release.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/release_demo.py attempts the same version five ways — clean, failing tests, an irreversible migration, a process that starts but cannot serve, and a canary over its error budget — printing each event and the resulting state, then demonstrates a rollback after a successful promotion.
bash tests/run_tests.sh runs pytest over seventeen tests grouped by gate: preflight, health, canary and rollback.
Expected output
--- clean release ---
deploying rolling out v1.1.0
canary 10% of traffic
promoted v1.1.0 at 100%
=> promoted live=v1.1.0 traffic_to_new=100%
--- irreversible migration ---
blocked migration is not reversible
=> blocked live=v1.0.0 traffic_to_new=0%
--- starts but cannot serve ---
deploying rolling out v1.1.0
rolled_back health check failed: readiness
=> rolled_back live=v1.0.0 traffic_to_new=0%
--- canary exceeds error budget ---
deploying rolling out v1.1.0
canary 10% of traffic
rolled_back error rate 9.0% exceeds budget 2.0%
=> rolled_back live=v1.0.0 traffic_to_new=0%
expected-output/FIELDS.md explains each field and the cost gradient across the gates.
Validation steps
bash tests/run_tests.shreports17 passed.- A blocked build must leave
liveunchanged andtraffic_to_newat zero — preflight deploys nothing at all. - Roll back twice and confirm you return to the newer version. If the second rollback does nothing, you are discarding rather than swapping.
- Confirm the readiness failure never reaches a
canaryevent. A failed health check must not route traffic.
Tests
Seventeen tests in four groups:
- preflight — a clean build passes; failing tests, an irreversible migration and an unpinned image each block; blockers accumulate; a blocked build deploys nothing.
- health — a process that starts but cannot serve is rolled back, an unreachable dependency is rolled back, every failing signal is named, and a failed check never reaches canary.
- canary — a healthy release is promoted through canary, one over budget is rolled back, one inside budget is promoted, and only a fraction of traffic is exposed before the decision.
- rollback — returns to the previous version, is itself reversible, and reports rather than crashes when there is nothing to return to.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md.
Security notes
See security.md. In short: no network, no credentials — and the lesson covers why pinning an image digest is a supply-chain control rather than a style preference.
Extension exercises
- A cost gate. Block a release whose estimated cost per thousand requests has risen more than a stated percentage. An AI release can be entirely correct and still double the bill.
- Canary analysis over a window. Decide on a sequence of per-minute samples with a minimum sample count, then report the request volume below which the analysis cannot distinguish a 2 percent error rate from a 9 percent one.
- Feature flags. Separate deploy from release, so a version can ship dark and be exposed independently.
Navigation
- Lesson
- Previous: Day 358 — Frontend and User Experience
- Next: Day 360 — Monitoring and Cost Controls
Expected output
FIELDS.md
# Reading the deployment output
## An event line
rolled_back error rate 9.0% exceeds budget 2.0%
| Part | Meaning |
| --- | --- |
| stage | `preflight`, `deploying`, `canary`, `promoted`, `rolled_back` or `blocked`. |
| detail | Why. On a block, every reason at once; on a rollback, which signal failed and by how much. |
## A summary line
=> rolled_back live=v1.0.0 traffic_to_new=0%
| Field | Meaning |
| --- | --- |
| stage | The last event's stage — where the release path stopped. |
| `live` | Which version is serving now. On any failure this must be the *old* one. |
| `traffic_to_new` | Percentage routed to the new version. Zero after any failure, 100 after promotion. |
## The six stages
| Stage | Meaning |
| --- | --- |
| `blocked` | Preflight refused. **Nothing was deployed** — `live` is unchanged and traffic is zero. |
| `deploying` | The new version is rolling out. |
| `canary` | A fraction of real traffic is on the new version, pending a decision. |
| `promoted` | All traffic moved, and the outgoing version recorded so rollback is possible. |
| `rolled_back` | Returned to the previous version. Reachable from health, from canary, or manually. |
## What to compare
Run the demo and read how far each failure travelled before something stopped it:
- **preflight** — nothing deployed, no user affected;
- **health** — deployed, no user affected;
- **canary** — 10 percent of users saw errors;
- **after promotion** — everyone did.
Same defect, four very different costs. That gradient is why the gates exist.
examples-run.txt
--- clean release ---
deploying rolling out v1.1.0
canary 10% of traffic
promoted v1.1.0 at 100%
=> promoted live=v1.1.0 traffic_to_new=100%
--- tests failed ---
blocked tests did not pass
=> blocked live=v1.0.0 traffic_to_new=0%
--- irreversible migration ---
blocked migration is not reversible
=> blocked live=v1.0.0 traffic_to_new=0%
--- starts but cannot serve ---
deploying rolling out v1.1.0
rolled_back health check failed: readiness
=> rolled_back live=v1.0.0 traffic_to_new=0%
--- canary exceeds error budget ---
deploying rolling out v1.1.0
canary 10% of traffic
rolled_back error rate 9.0% exceeds budget 2.0%
=> rolled_back live=v1.0.0 traffic_to_new=0%
--- rollback after a bad promotion ---
promoted: promoted live=v1.1.0 traffic_to_new=100%
rolled_back back to v1.0.0
=> rolled_back live=v1.0.0 traffic_to_new=0%
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/release_demo.py -> 24 lines, exit 0
bash tests/run_tests.sh -> 17 passed in 0.01s
Five deployments of the same version, each stopping at a different gate:
scenario stopped at live after traffic who saw it
clean release promoted v1.1.0 100% everyone
tests failed preflight v1.0.0 0% nobody
irreversible migration preflight v1.0.0 0% nobody
starts but cannot serve health v1.0.0 0% nobody
canary over error budget canary v1.0.0 0% 10% of users
Read the last column downwards. It is the whole argument for having several
gates rather than one good one: the same defect costs a different amount
depending on where it is caught, and the difference is users.
Note the health case in particular. The process STARTED -- liveness was fine.
Readiness was not, so it could not serve. A deployment system that only checks
liveness reports that release as healthy and leaves it serving errors.
Rollback:
promoted: live=v1.1.0 traffic_to_new=100%
rolled_back: back to v1.0.0
One operation, no rebuild. Because it swaps rather than discards, rolling back
again returns to v1.1.0.
run-output.txt
--- clean release ---
deploying rolling out v1.1.0
canary 10% of traffic
promoted v1.1.0 at 100%
=> promoted live=v1.1.0 traffic_to_new=100%
--- tests failed ---
blocked tests did not pass
=> blocked live=v1.0.0 traffic_to_new=0%
--- irreversible migration ---
blocked migration is not reversible
=> blocked live=v1.0.0 traffic_to_new=0%
--- starts but cannot serve ---
deploying rolling out v1.1.0
rolled_back health check failed: readiness
=> rolled_back live=v1.0.0 traffic_to_new=0%
--- canary exceeds error budget ---
deploying rolling out v1.1.0
canary 10% of traffic
rolled_back error rate 9.0% exceeds budget 2.0%
=> rolled_back live=v1.0.0 traffic_to_new=0%
--- rollback after a bad promotion ---
promoted: promoted live=v1.1.0 traffic_to_new=100%
rolled_back back to v1.0.0
=> rolled_back live=v1.0.0 traffic_to_new=0%
starter-run.txt
An untouched starter fails every test:
17 failed in 0.05s
All three stubbed functions raise NotImplementedError, and every test depends
on at least one. Unlike some earlier labs there is no test that passes by
accident here.
test-output.txt
................. [100%]
17 passed in 0.01s
test-run.txt
................. [100%]
17 passed in 0.01s
Source files
examples/release_demo.py (1306 bytes)
#!/usr/bin/env python3
"""Five deployments: one clean, four that stop at different gates."""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from release import Build, DeployBlocked, Deployment, Health, deploy, rollback
def run(label: str, build: Build, **kwargs) -> None:
dep = Deployment(live_version="v1.0.0")
print(f"--- {label} ---")
try:
deploy(dep, build, **kwargs)
except DeployBlocked:
pass
for event in dep.events:
print(event.line())
print(f" => {dep.summary()}")
def main() -> int:
run("clean release", Build("v1.1.0"))
run("tests failed", Build("v1.1.0", tests_passed=False))
run("irreversible migration", Build("v1.1.0", migrations_reversible=False))
run("starts but cannot serve", Build("v1.1.0"), health=Health(readiness=False))
run("canary exceeds error budget", Build("v1.1.0"), error_rate=0.09)
print("--- rollback after a bad promotion ---")
dep = Deployment(live_version="v1.0.0")
deploy(dep, Build("v1.1.0"))
print(f" promoted: {dep.summary()}")
rollback(dep)
print(f" {dep.events[-1].line().strip()}")
print(f" => {dep.summary()}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/release.py (5968 bytes)
"""Release gates, health checks and rollback for a capstone deployment.
Offline and standard-library only. There is no container runtime and no cloud
here; what is modelled is the decision structure of a deployment, which is
where deployments actually go wrong.
Four ideas, in the order they matter:
preflight refuse to deploy a build that fails a stated condition
health distinguish "the process started" from "the system works"
progressive send a fraction of traffic first, and decide on evidence
rollback get back to the last known-good state without a rebuild
The fourth is the one that decides how bad a bad day is. A deployment you
cannot reverse is not a deployment, it is a commitment.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class Stage(str, Enum):
PREFLIGHT = "preflight"
DEPLOYING = "deploying"
CANARY = "canary"
PROMOTED = "promoted"
ROLLED_BACK = "rolled_back"
BLOCKED = "blocked"
class DeployBlocked(RuntimeError):
"""A preflight condition failed, so nothing was deployed."""
@dataclass(frozen=True)
class Build:
version: str
tests_passed: bool = True
migrations_reversible: bool = True
secrets_present: bool = True
image_digest: str = "sha256:deadbeef"
@dataclass
class Health:
"""What a health endpoint reports.
`liveness` means the process is running. `readiness` means it can actually
serve -- dependencies reachable, model loaded, index warm. Conflating them
is why a deployment can go green while every request fails.
"""
liveness: bool = True
readiness: bool = True
dependency_ok: bool = True
@property
def serving(self) -> bool:
return self.liveness and self.readiness and self.dependency_ok
@dataclass
class Event:
stage: Stage
detail: str = ""
def line(self) -> str:
return f" {self.stage.value:<12} {self.detail}" if self.detail else f" {self.stage.value}"
@dataclass
class Deployment:
live_version: str
previous_version: str | None = None
events: list[Event] = field(default_factory=list)
traffic_to_new: int = 0 # percent
@property
def stage(self) -> Stage:
return self.events[-1].stage if self.events else Stage.PREFLIGHT
def summary(self) -> str:
return (
f"{self.stage.value} live={self.live_version} "
f"traffic_to_new={self.traffic_to_new}%"
)
def preflight(build: Build) -> list[str]:
"""Reasons this build must not be deployed. Empty means it may proceed.
Every condition here is one that is cheap to check now and expensive to
discover in production. An irreversible migration is the worst of them,
because it removes the rollback that every other failure depends on.
"""
problems: list[str] = []
if not build.tests_passed:
problems.append("tests did not pass")
if not build.migrations_reversible:
problems.append("migration is not reversible")
if not build.secrets_present:
problems.append("required secrets are missing")
if not build.image_digest.startswith("sha256:"):
problems.append("image is not pinned to a digest")
return problems
def deploy(
current: Deployment,
build: Build,
*,
health: Health | None = None,
canary_percent: int = 10,
error_rate: float = 0.0,
error_budget: float = 0.02,
) -> Deployment:
"""Run the full release path, stopping at the first thing that says no."""
health = health or Health()
blockers = preflight(build)
if blockers:
current.events.append(Event(Stage.BLOCKED, "; ".join(blockers)))
raise DeployBlocked("; ".join(blockers))
current.events.append(Event(Stage.DEPLOYING, f"rolling out {build.version}"))
# A process that started is not a system that works. Readiness is the
# question worth asking, and it is the one a naive check skips.
if not health.serving:
failed = [
name
for name, ok in (
("liveness", health.liveness),
("readiness", health.readiness),
("dependency", health.dependency_ok),
)
if not ok
]
current.events.append(
Event(Stage.ROLLED_BACK, f"health check failed: {', '.join(failed)}")
)
current.traffic_to_new = 0
return current
current.traffic_to_new = canary_percent
current.events.append(Event(Stage.CANARY, f"{canary_percent}% of traffic"))
# Decide on evidence rather than on elapsed time. "It has been up for ten
# minutes" is not the same claim as "it is serving correctly".
if error_rate > error_budget:
current.events.append(
Event(
Stage.ROLLED_BACK,
f"error rate {error_rate:.1%} exceeds budget {error_budget:.1%}",
)
)
current.traffic_to_new = 0
return current
current.previous_version = current.live_version
current.live_version = build.version
current.traffic_to_new = 100
current.events.append(Event(Stage.PROMOTED, f"{build.version} at 100%"))
return current
def rollback(current: Deployment) -> Deployment:
"""Return to the previous version without rebuilding anything.
Rollback has to be a data operation -- swap which artefact is live -- not a
build. A rollback that requires a rebuild takes as long as the deployment
did, during which the incident continues.
"""
if current.previous_version is None:
current.events.append(Event(Stage.ROLLED_BACK, "no previous version to return to"))
return current
current.live_version, current.previous_version = (
current.previous_version,
current.live_version,
)
current.traffic_to_new = 0
current.events.append(Event(Stage.ROLLED_BACK, f"back to {current.live_version}"))
return current
metadata.yml (786 bytes)
lesson_id: D359
day: 359
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/release_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/release_demo.py
reproduced the documented five deployments and the rollback exactly; bash tests/run_tests.sh -> 17
passed in 0.01s. The untouched starter fails all 17. requires_network is false: there is no container
runtime and no cloud, only the decision structure of a deployment, so nothing leaves the machine.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/release.py (5112 bytes)
"""Release gates, health checks and rollback for a capstone deployment.
Offline and standard-library only. There is no container runtime and no cloud
here; what is modelled is the decision structure of a deployment, which is
where deployments actually go wrong.
Four ideas, in the order they matter:
preflight refuse to deploy a build that fails a stated condition
health distinguish "the process started" from "the system works"
progressive send a fraction of traffic first, and decide on evidence
rollback get back to the last known-good state without a rebuild
The fourth is the one that decides how bad a bad day is. A deployment you
cannot reverse is not a deployment, it is a commitment.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class Stage(str, Enum):
PREFLIGHT = "preflight"
DEPLOYING = "deploying"
CANARY = "canary"
PROMOTED = "promoted"
ROLLED_BACK = "rolled_back"
BLOCKED = "blocked"
class DeployBlocked(RuntimeError):
"""A preflight condition failed, so nothing was deployed."""
@dataclass(frozen=True)
class Build:
version: str
tests_passed: bool = True
migrations_reversible: bool = True
secrets_present: bool = True
image_digest: str = "sha256:deadbeef"
@dataclass
class Health:
"""What a health endpoint reports.
`liveness` means the process is running. `readiness` means it can actually
serve -- dependencies reachable, model loaded, index warm. Conflating them
is why a deployment can go green while every request fails.
"""
liveness: bool = True
readiness: bool = True
dependency_ok: bool = True
@property
def serving(self) -> bool:
return self.liveness and self.readiness and self.dependency_ok
@dataclass
class Event:
stage: Stage
detail: str = ""
def line(self) -> str:
return f" {self.stage.value:<12} {self.detail}" if self.detail else f" {self.stage.value}"
@dataclass
class Deployment:
live_version: str
previous_version: str | None = None
events: list[Event] = field(default_factory=list)
traffic_to_new: int = 0 # percent
@property
def stage(self) -> Stage:
return self.events[-1].stage if self.events else Stage.PREFLIGHT
def summary(self) -> str:
return (
f"{self.stage.value} live={self.live_version} "
f"traffic_to_new={self.traffic_to_new}%"
)
def preflight(build: Build) -> list[str]:
"""Reasons this build must not be deployed. Empty means it may proceed.
Every condition here is one that is cheap to check now and expensive to
discover in production. An irreversible migration is the worst of them,
because it removes the rollback that every other failure depends on.
"""
# TASK 1: return EVERY reason not to deploy, not just the first.
# tests_passed False -> "tests did not pass"
# migrations_reversible False -> "migration is not reversible"
# secrets_present False -> "required secrets are missing"
# image_digest not sha256:... -> "image is not pinned to a digest"
# A build with three problems should tell you three things once, rather
# than sending you round the loop three times.
raise NotImplementedError("implement preflight")
def deploy(
current: Deployment,
build: Build,
*,
health: Health | None = None,
canary_percent: int = 10,
error_rate: float = 0.0,
error_budget: float = 0.02,
) -> Deployment:
"""Run the full release path, stopping at the first thing that says no."""
# TASK 2: run the release path, stopping at the first gate that says no.
# - preflight blockers -> append a BLOCKED event and raise DeployBlocked,
# WITHOUT changing live_version or traffic_to_new. Nothing deployed.
# - append DEPLOYING, then check health BEFORE routing any traffic. If it
# is not serving, append ROLLED_BACK naming every failing signal.
# - set traffic_to_new to canary_percent and append CANARY.
# - if error_rate > error_budget, append ROLLED_BACK and return traffic to
# zero. Decide on the rate, never on elapsed time.
# - otherwise record the outgoing version as previous_version, promote,
# and set traffic to 100.
raise NotImplementedError("implement deploy")
def rollback(current: Deployment) -> Deployment:
"""Return to the previous version without rebuilding anything.
Rollback has to be a data operation -- swap which artefact is live -- not a
build. A rollback that requires a rebuild takes as long as the deployment
did, during which the incident continues.
"""
# TASK 3: return to the previous version without rebuilding.
# SWAP live_version and previous_version rather than discarding, so the
# rollback is itself reversible -- which matters on the day the rollback
# was the mistake. With no previous version, append a ROLLED_BACK event
# saying so rather than crashing.
raise NotImplementedError("implement rollback")
tests/run_tests.sh (200 bytes)
#!/usr/bin/env bash
# Run the day 359 deployment test suite.
# Offline, standard library plus pytest. No network, no API key.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_release.py (4810 bytes)
"""Grouped by gate, so a failure names which one let a bad build through.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
import pytest
HERE = os.path.dirname(os.path.abspath(__file__))
LAB = os.path.dirname(HERE)
sys.path.insert(0, os.path.join(LAB, "examples"))
from release import ( # noqa: E402
Build,
DeployBlocked,
Deployment,
Health,
Stage,
deploy,
preflight,
rollback,
)
def fresh() -> Deployment:
return Deployment(live_version="v1.0.0")
# ---------------------------------------------------------------- preflight
def test_a_clean_build_has_no_blockers():
assert preflight(Build("v1.1.0")) == []
def test_failing_tests_block_the_deploy():
assert "tests did not pass" in preflight(Build("v1", tests_passed=False))
def test_an_irreversible_migration_blocks_the_deploy():
# The worst blocker, because it removes the rollback every other failure
# depends on.
assert "migration is not reversible" in preflight(Build("v1", migrations_reversible=False))
def test_an_unpinned_image_blocks_the_deploy():
assert "not pinned" in " ".join(preflight(Build("v1", image_digest="latest")))
def test_blockers_accumulate_rather_than_stopping_at_the_first():
problems = preflight(Build("v1", tests_passed=False, secrets_present=False))
assert len(problems) == 2
def test_a_blocked_build_deploys_nothing():
dep = fresh()
with pytest.raises(DeployBlocked):
deploy(dep, Build("v1.1.0", tests_passed=False))
assert dep.live_version == "v1.0.0"
assert dep.traffic_to_new == 0
assert dep.stage is Stage.BLOCKED
# ------------------------------------------------------------------- health
def test_a_process_that_starts_but_cannot_serve_is_rolled_back():
# Liveness is true, readiness is not. Conflating them is why a deployment
# goes green while every request fails.
dep = deploy(fresh(), Build("v1.1.0"), health=Health(readiness=False))
assert dep.stage is Stage.ROLLED_BACK
assert dep.live_version == "v1.0.0"
assert "readiness" in dep.events[-1].detail
def test_an_unreachable_dependency_is_rolled_back():
dep = deploy(fresh(), Build("v1.1.0"), health=Health(dependency_ok=False))
assert dep.stage is Stage.ROLLED_BACK
assert "dependency" in dep.events[-1].detail
def test_health_names_every_failing_signal():
dep = deploy(fresh(), Build("v1.1.0"), health=Health(readiness=False, dependency_ok=False))
detail = dep.events[-1].detail
assert "readiness" in detail and "dependency" in detail
def test_a_failed_health_check_never_reaches_canary():
dep = deploy(fresh(), Build("v1.1.0"), health=Health(readiness=False))
assert Stage.CANARY not in [e.stage for e in dep.events]
# ------------------------------------------------------------------ canary
def test_a_healthy_release_is_promoted_through_canary():
dep = deploy(fresh(), Build("v1.1.0"))
stages = [e.stage for e in dep.events]
assert stages == [Stage.DEPLOYING, Stage.CANARY, Stage.PROMOTED]
assert dep.live_version == "v1.1.0"
assert dep.traffic_to_new == 100
def test_a_canary_over_the_error_budget_is_rolled_back():
dep = deploy(fresh(), Build("v1.1.0"), error_rate=0.09, error_budget=0.02)
assert dep.stage is Stage.ROLLED_BACK
assert dep.live_version == "v1.0.0"
assert "9.0%" in dep.events[-1].detail
def test_a_canary_inside_the_error_budget_is_promoted():
dep = deploy(fresh(), Build("v1.1.0"), error_rate=0.01, error_budget=0.02)
assert dep.stage is Stage.PROMOTED
def test_canary_takes_only_a_fraction_of_traffic_before_the_decision():
dep = fresh()
deploy(dep, Build("v1.1.0"), canary_percent=5, error_rate=0.09)
canary = next(e for e in dep.events if e.stage is Stage.CANARY)
assert "5%" in canary.detail
# And the failed canary leaves no traffic on the new version.
assert dep.traffic_to_new == 0
# ---------------------------------------------------------------- rollback
def test_rollback_returns_to_the_previous_version():
dep = deploy(fresh(), Build("v1.1.0"))
assert dep.live_version == "v1.1.0"
rollback(dep)
assert dep.live_version == "v1.0.0"
assert dep.traffic_to_new == 0
def test_rollback_is_reversible():
# Swapping rather than discarding means a rollback can itself be undone,
# which matters when the rollback was the mistake.
dep = deploy(fresh(), Build("v1.1.0"))
rollback(dep)
rollback(dep)
assert dep.live_version == "v1.1.0"
def test_rollback_with_no_previous_version_is_reported_not_crashed():
dep = fresh()
rollback(dep)
assert dep.stage is Stage.ROLLED_BACK
assert "no previous version" in dep.events[-1].detail
assert dep.live_version == "v1.0.0"
Troubleshooting
Troubleshooting — Day 359
A blocked build still changes the live version
You are mutating state before raising. Collect the blockers first, and if there are any, append the BLOCKED event and raise without touching live_version or traffic_to_new. A preflight failure must deploy nothing at all.
preflight reports only one problem
You are returning at the first failure. Accumulate all of them into a list and return it. A build with three problems should tell you three things once, rather than sending you round the loop three times.
A failed health check still produces a canary event
You are checking health after routing traffic. Check before, and return immediately on failure — test_a_failed_health_check_never_reaches_canary asserts that no CANARY event exists in that path.
test_health_names_every_failing_signal fails
You are reporting the first failing signal instead of all of them. Build the detail from every one of liveness, readiness and dependency that is false, for the same reason preflight accumulates.
Rolling back twice does nothing the second time
You are setting previous_version to None, or overwriting it, instead of swapping. Exchange the two values so the operation is symmetric — a rollback you cannot undo is another one-way door pressed under pressure.
The canary promotes when it should roll back
Check the comparison direction: roll back when error_rate > error_budget. A rate exactly equal to the budget is within it.
test_canary_takes_only_a_fraction_of_traffic_before_the_decision fails
Two things must both hold: the canary event names the configured percentage, and a failed canary leaves traffic_to_new at zero rather than at the canary value. Reset it when you roll back.
The demo output ordering differs from the reference
Events are appended in the order the gates run, and the reference order is preflight, deploying, health, canary, promotion. If your deploying event comes after the health check, the transcript no longer shows that the version was rolled out before it was found unable to serve.
NotImplementedError on every test
Expected. The starter stubs all three functions, and every test depends on at least one — see expected-output/starter-run.txt.
Security notes
Security notes — Day 359
What this lab does and does not touch
- No network. There is no container runtime, no registry and no cloud. Only the decision structure is modelled.
- No credentials. No API key, token or account.
secrets_presentis a boolean flag on a dataclass, not a real secret. - No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/. - No
sudo, no global installs.
Why pinning an image digest is a security control
preflight blocks a build whose image is not pinned to a sha256: digest, and it is worth being explicit that this is not tidiness.
A tag such as latest, or even v1.1.0, is a mutable pointer. Whoever can push to the registry can change what that tag resolves to, at any time, including after your tests passed against it. A digest is the content hash: it either matches the bytes you tested or it does not exist.
The attack this prevents is a supply-chain one. An attacker who compromises a build account does not need to touch your repository — repointing a tag is enough, and everything downstream continues to look correct. Pinning means the artefact you deploy is provably the artefact you tested.
The same argument applies to base images in a Dockerfile and to dependencies in a lockfile.
Secrets belong in the environment, never in the image
A container image is a distributable artefact. Anything baked into it travels with it — into the registry, into every cache, into anyone's docker pull, and into the layer history where a later RUN rm does not remove it.
preflight checks that required secrets are present rather than checking their values, which is the right shape: the deployment needs to know they were supplied, and should never handle them itself.
Deployment is a privacy checkpoint
A new version may log new fields. Because a deployment is the moment that starts happening, it is the natural place to ask what the logs will now contain — before the data accumulates rather than after. That connects directly to Day 328: a store you did not know you created is a store your erasure path does not cover.
Rollback is an availability control
The blunt version: an incident lasts exactly as long as the rollback takes. That makes rollback speed a security property, not just an operational convenience, and it is why the lab insists rollback be a pointer swap rather than a rebuild.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.