Deployment, MLOps, and Security › Containers and Cloud › Day 333
Hands-on lab — Day 333: Kubernetes Concepts
- ← Back to the Day 333 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deployment-mlops-and-security/day-333-kubernetes-concepts/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/reconcile_demo.py Test
bash tests/run_tests.sh File tree
examples/reconcile_demo.py examples/reconcile.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/reconcile.py tests/run_tests.sh tests/test_reconcile.py troubleshooting.md
Lab README
Day 333 Lab: Kubernetes Concepts
Lesson
- Lesson title: Kubernetes Concepts
- Day number: 333 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-333-kubernetes-concepts
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-333-kubernetes-conceptswhen the site is running.
Purpose
Build the control loop that is Kubernetes' one idea, and watch four different situations turn out to be the same situation.
You declare three replicas. The loop sees what exists, compares, and takes one step. It never asks how the gap appeared — which is why a node failure and kubectl delete pod produce identical behaviour, and why a rolling update is not a special operation but the same loop with a changed spec.
No cluster required. The loop is about forty lines, and the forty lines are the part worth understanding.
Learning objectives
- Implement a reconciliation step: compare desired against actual, then act.
- Distinguish a pod that is Running from one that is serving, and explain what a Service routes to.
- Show that node failure, manual deletion and scaling are one mechanism.
- Enforce
maxUnavailableandmaxSurgeduring a rolling update. - Explain why "no actions this step" does not mean "the rollout is finished".
Prerequisites
- Day 332, "Docker Compose for Multi-Service Apps" — the readiness distinction, made continuous here.
- Comfortable with Python dataclasses and enums.
Supported operating systems
macOS, Linux and Windows (via WSL or PowerShell). Nothing is platform-specific.
Hardware requirements
Any machine running Python 3.10 or newer. No cluster, no container runtime, no cloud account. The suite completes in hundredths of a second.
Required software
- Python 3.10 or newer.
pytest7 or newer.
No Kubernetes cluster is required and none is contacted. Installing one is a large detour, and the idea you need is not in the YAML — it is in the loop.
Free and open-source options
Every dependency is free and open source: Python is PSF-licensed, pytest is MIT-licensed. If you later want a real cluster locally, kind, k3s and minikube are all Apache-2.0 and run on a laptop.
Installation
python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt
Removing .venv fully reverses the installation.
File structure
starter/reconcile.py your work: eleven stubbed tasks
examples/reconcile.py reference implementation
examples/reconcile_demo.py four situations, one loop
tests/test_reconcile.py grouped by what the loop decides
tests/run_tests.sh suite entry point
expected-output/ real captured output and measured values
requirements/ pinned dependency
How to run
python3 examples/reconcile_demo.py # deploy, lose a node, delete a pod, roll out
bash tests/run_tests.sh # run the suite
To work on the exercise, edit starter/reconcile.py, then copy it over the reference:
cp starter/reconcile.py examples/reconcile.py
bash tests/run_tests.sh
Keep a copy of the reference first if you want to compare.
What the commands do
python3 examples/reconcile_demo.py runs one deployment through four situations: a first deploy from nothing, a node dying and taking a pod with it, somebody deleting a pod by hand, and a rolling update to a new image. It prints each action the loop takes and the reason, then the serving count throughout the rollout.
bash tests/run_tests.sh runs pytest over nineteen tests grouped by what the loop decides: serving, first deploy, healing, scaling, rollout and completion.
Expected output
--- 4. a rolling update to app:1.5 ---
step 1: 3 action(s) -> 4 alive, 2 serving api-004(1.4) api-005(1.4) api-006(1.5) api-007(1.5)
step 2: 0 action(s) -> 4 alive, 4 serving (no action — waiting for the new pods to become ready)
step 3: 3 action(s) -> 3 alive, 2 serving api-006(1.5) api-007(1.5) api-008(1.5)
step 5: rollout complete — every pod on app:1.5
serving during rollout: [2, 4, 2, 3] minimum 2
Step 2 is the one to read: zero actions, and the rollout is not finished. The controller is waiting for the new pods to become ready before it may retire any more.
Validation steps
bash tests/run_tests.shreports19 passed.- A first deploy must converge in 2 steps, not 1 — a pod needs one tick to run and another to become ready.
- Losing a node and deleting a pod by hand must produce the same kind of action. If they differ, the loop is reasoning about history.
- During the rollout,
servingmust never fall belowreplicas - max_unavailable, andalivemust never exceedreplicas + max_surge. - Replacements must never be scheduled onto a dead node. If they are,
convergewill raise after churning forever.
Tests
Nineteen tests in six groups:
- serving — a pod must be both Running and ready; a failed pod never serves.
- first deploy — an empty cluster creates every replica; a satisfied deployment does nothing; convergence reaches the declared count.
- healing — failed pods and pods on dead nodes are reaped; a manual delete is simply refilled; replacements avoid dead nodes; a cluster with no live node raises.
- scaling — scaling up creates only the difference; zero replicas is a valid desired state.
- rollout — a changed image starts one; the surge ceiling and the unavailability floor both hold; it finishes with every pod on the new image; no actions does not mean finished.
- completion —
rollout_donerequires readiness and the full replica count.
Cleanup
rm -rf __pycache__ .pytest_cache
rm -rf .venv
Both are safe to remove and fully reversible.
Troubleshooting
See troubleshooting.md.
Security notes
See security.md. In short: the loop erases manual changes to running pods, which makes the manifest repository the only durable configuration — and therefore production infrastructure.
Extension exercises
- Add a liveness probe. A pod that fails liveness is restarted rather than deleted. Model the difference and answer the question it raises: what happens when a liveness probe checks a downstream dependency and that dependency is briefly slow?
- Add resource requests and node capacity. Give each node a CPU budget and each pod a request, then make
schedulerefuse to place a pod that does not fit. Watch a Deployment stay Pending forever, which is the most common "why is nothing happening" in a real cluster. - Add a PodDisruptionBudget. Model voluntary disruption — a node drain — separately from involuntary, and enforce a minimum available across both.
Navigation
- Lesson
- Previous: Day 332 — Docker Compose for Multi-Service Apps
- Next: Day 334 — Cloud Options and Free Tiers
Expected output
FIELDS.md
# Reading the reconciliation output
## A step
the loop compares and acts:
delete api-001 node node-a is gone
create api-004 2 of 3 replicas present
-> 3 alive, 2 serving api-002(1.4) api-003(1.4) api-004(1.4)
| Part | Meaning |
| --- | --- |
| `delete` / `create` | One action the loop decided to take this step. |
| the reason | Why. Always a statement about the **current gap**, never about history. |
| `N alive` | Pods that exist, are not failed, and are not on a dead node. |
| `N serving` | Pods a Service would actually route to — running **and** ready. |
`alive` and `serving` differ whenever a pod is starting. That gap is the whole
of readiness.
## Alive versus serving
3 alive, 2 serving
Three pods exist and two can take a request. The third is Running and has not
yet passed its readiness probe. A Service routes to two of them.
This is the same distinction as yesterday's `service_healthy`, made continuous:
Compose asks it once at startup, Kubernetes asks it every few seconds forever.
## The four scenarios
| Scenario | What the loop does |
| --- | --- |
| first deploy | Nothing exists; create every replica. Converges in **2** steps, because a pod needs one tick to run and another to become ready. |
| a node dies | Reap the stranded pod, create a replacement — **on a live node**. |
| a manual delete | Exactly the same as a node death. The loop never asks how the gap appeared. |
| a rolling update | Replace stale pods within the budget, wait for readiness, repeat. |
Scenarios 2 and 3 producing the same behaviour is the point. `kubectl delete
pod` is not destructive because deletion is just another gap.
## The rollout
step 1: 3 action(s) -> 4 alive, 2 serving api-004(1.4) api-005(1.4) api-006(1.5) api-007(1.5)
step 2: 0 action(s) -> 4 alive, 4 serving (no action — waiting for the new pods to become ready)
step 3: 3 action(s) -> 3 alive, 2 serving api-006(1.5) api-007(1.5) api-008(1.5)
step 4: 0 action(s) -> 3 alive, 3 serving (no action — waiting for the new pods to become ready)
step 5: rollout complete — every pod on app:1.5
**Step 2 is the one to read.** Zero actions, and the rollout is not finished.
The controller has already replaced as many pods as `maxUnavailable` permits
and may not retire another until the new ones are ready. That pause is the
budget working.
It is also why `rollout_done` exists separately from "reconcile returned no
actions" — they answer different questions, and conflating them makes a
half-finished rollout report success.
## The guarantee
serving during rollout: [2, 4, 2, 3] minimum 2
2 required, 2 observed
With `replicas=3` and `maxUnavailable=1`, at least two pods must be serving at
every moment. The observed minimum is exactly two — the budget is fully used
and never exceeded.
Note the 4 in the middle: `maxSurge=1` permits one extra pod temporarily, which
is how the rollout makes progress without dipping below the floor.
examples-run.txt
--- 1. first deploy: nothing exists yet ---
converged in 2 steps -> 3 alive, 3 serving api-001(1.4) api-002(1.4) api-003(1.4)
--- 2. a node dies, taking one pod with it ---
the loop compares and acts:
delete api-001 node node-a is gone
create api-004 2 of 3 replicas present
-> 3 alive, 2 serving api-002(1.4) api-003(1.4) api-004(1.4)
converged in 1 more steps -> 3 alive, 3 serving api-002(1.4) api-003(1.4) api-004(1.4)
--- 3. somebody deletes a pod by hand ---
deleted api-002 manually
the loop does not care how it happened:
create api-005 2 of 3 replicas present
-> 3 alive, 2 serving api-003(1.4) api-004(1.4) api-005(1.4)
converged in 1 more steps -> 3 alive, 3 serving api-003(1.4) api-004(1.4) api-005(1.4)
--- 4. a rolling update to app:1.5 ---
step 1: 3 action(s) -> 4 alive, 2 serving api-004(1.4) api-005(1.4) api-006(1.5) api-007(1.5)
step 2: 0 action(s) -> 4 alive, 4 serving api-004(1.4) api-005(1.4) api-006(1.5) api-007(1.5) (no action — waiting for the new pods to become ready)
step 3: 3 action(s) -> 3 alive, 2 serving api-006(1.5) api-007(1.5) api-008(1.5)
step 4: 0 action(s) -> 3 alive, 3 serving api-006(1.5) api-007(1.5) api-008(1.5) (no action — waiting for the new pods to become ready)
step 5: rollout complete — every pod on app:1.5
serving during rollout: [2, 4, 2, 3] minimum 2
never dropped below max_unavailable allows: 2 required, 2 observed
measured-values.txt
Measured on macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1.
python3 examples/reconcile_demo.py -> 22 lines, exit 0
bash tests/run_tests.sh -> 19 passed in 0.01s
the untouched starter -> 18 failed, 1 passed in 0.06s
One deployment, one loop, four situations.
1. first deploy converged in 2 steps -> 3 alive, 3 serving
2. a node dies delete api-001, create api-004 -> converged in 1 more step
3. a manual delete create api-005 -> converged in 1 more step
4. a rolling update completed in 5 steps, every pod on app:1.5
The thing worth noticing is that scenarios 2 and 3 produce the SAME behaviour.
A node failure and somebody running `kubectl delete pod` are indistinguishable
to the loop, because the loop does not ask how the gap appeared. It compares
what was declared against what exists and takes one step.
That is why `kubectl delete pod` is not a destructive operation, and it is the
single idea the day is about.
The first deploy takes 2 steps rather than 1 because a pod needs one tick to
move from Pending to Running and another to pass its readiness probe. After
step 1 there are 3 alive and 0 serving. The spec is satisfied and a Service
would route to nothing.
The rollout, step by step:
step 1: 3 actions 4 alive, 2 serving two 1.4 pods, two 1.5 pods
step 2: 0 actions 4 alive, 4 serving waiting for the new pods to be ready
step 3: 3 actions 3 alive, 2 serving
step 4: 0 actions 3 alive, 3 serving waiting again
step 5: complete
Step 2 is the instructive one. Zero actions, and the rollout is NOT finished.
The controller has replaced as many pods as maxUnavailable allows and may not
retire another until the replacements are serving. An earlier version of the
demo treated "no actions" as "rollout complete" and reported success while two
pods were still on the old image -- which is why rollout_done exists as a
separate question.
The guarantee, measured:
serving during rollout: [2, 4, 2, 3] minimum 2
replicas 3, max_unavailable 1 -> floor of 2
The observed minimum is exactly the floor. The budget is fully used and never
exceeded, and the 4 in the middle is maxSurge permitting one extra pod so the
rollout can make progress without dipping below it.
One implementation note. New pods must be scheduled onto a LIVE node. An
earlier version placed replacements on the default node, which was the node
that had just died, so the loop reaped them and recreated them forever and
converge raised. That is a real failure mode when a scheduler's node filter is
wrong, and test_replacements_are_not_scheduled_onto_a_dead_node pins it.
run-output.txt
--- 1. first deploy: nothing exists yet ---
converged in 2 steps -> 3 alive, 3 serving api-001(1.4) api-002(1.4) api-003(1.4)
--- 2. a node dies, taking one pod with it ---
the loop compares and acts:
delete api-001 node node-a is gone
create api-004 2 of 3 replicas present
-> 3 alive, 2 serving api-002(1.4) api-003(1.4) api-004(1.4)
converged in 1 more steps -> 3 alive, 3 serving api-002(1.4) api-003(1.4) api-004(1.4)
--- 3. somebody deletes a pod by hand ---
deleted api-002 manually
the loop does not care how it happened:
create api-005 2 of 3 replicas present
-> 3 alive, 2 serving api-003(1.4) api-004(1.4) api-005(1.4)
converged in 1 more steps -> 3 alive, 3 serving api-003(1.4) api-004(1.4) api-005(1.4)
--- 4. a rolling update to app:1.5 ---
step 1: 3 action(s) -> 4 alive, 2 serving api-004(1.4) api-005(1.4) api-006(1.5) api-007(1.5)
step 2: 0 action(s) -> 4 alive, 4 serving api-004(1.4) api-005(1.4) api-006(1.5) api-007(1.5) (no action — waiting for the new pods to become ready)
step 3: 3 action(s) -> 3 alive, 2 serving api-006(1.5) api-007(1.5) api-008(1.5)
step 4: 0 action(s) -> 3 alive, 3 serving api-006(1.5) api-007(1.5) api-008(1.5) (no action — waiting for the new pods to become ready)
step 5: rollout complete — every pod on app:1.5
serving during rollout: [2, 4, 2, 3] minimum 2
never dropped below max_unavailable allows: 2 required, 2 observed
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
FAILED tests/test_reconcile.py::test_a_manual_delete_is_simply_refilled - Not...
FAILED tests/test_reconcile.py::test_replacements_are_not_scheduled_onto_a_dead_node
FAILED tests/test_reconcile.py::test_scaling_up_creates_the_difference_only
FAILED tests/test_reconcile.py::test_scaling_to_zero_is_a_valid_desired_state
FAILED tests/test_reconcile.py::test_a_changed_image_starts_a_rollout - NotIm...
FAILED tests/test_reconcile.py::test_a_rollout_never_exceeds_replicas_plus_surge
FAILED tests/test_reconcile.py::test_a_rollout_never_drops_below_the_unavailability_budget
FAILED tests/test_reconcile.py::test_a_rollout_finishes_with_every_pod_on_the_new_image
FAILED tests/test_reconcile.py::test_no_actions_does_not_mean_the_rollout_is_finished
FAILED tests/test_reconcile.py::test_rollout_done_requires_readiness_not_just_running
FAILED tests/test_reconcile.py::test_rollout_done_requires_the_full_replica_count
18 failed, 1 passed in 0.06s
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
................... [100%]
19 passed in 0.01s
test-run.txt
................... [100%]
19 passed in 0.01s
Source files
examples/reconcile_demo.py (2819 bytes)
#!/usr/bin/env python3
"""The same loop handles a first deploy, a lost node, a manual delete, and a rollout."""
from __future__ import annotations
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from reconcile import (
Cluster, Deployment, Phase, Pod, apply, converge, reconcile, rollout_done, tick,
)
def line(cluster: Cluster, dep: Deployment) -> str:
alive = cluster.alive(dep)
return (f"{len(alive)} alive, {len(cluster.serving(dep))} serving "
+ " ".join(f"{p.name}({p.image.split(':')[-1]})" for p in alive))
def step(cluster: Cluster, dep: Deployment, label: str) -> None:
actions = reconcile(cluster, dep)
print(f" {label}")
for a in actions:
print(f" {a.verb:<6} {a.pod:<12} {a.reason}")
if not actions:
print(" (nothing to do — actual state matches desired)")
apply(cluster, actions, dep)
tick(cluster)
print(f" -> {line(cluster, dep)}")
def main() -> int:
dep = Deployment("api", "app:1.4", replicas=3)
cluster = Cluster()
print("--- 1. first deploy: nothing exists yet ---")
steps = converge(cluster, dep)
print(f" converged in {steps} steps -> {line(cluster, dep)}")
print("--- 2. a node dies, taking one pod with it ---")
cluster.dead_nodes.add("node-a")
cluster.pods[0].node = "node-a"
for p in cluster.pods[1:]:
p.node = "node-b"
step(cluster, dep, "the loop compares and acts:")
print(f" converged in {converge(cluster, dep)} more steps -> {line(cluster, dep)}")
print("--- 3. somebody deletes a pod by hand ---")
removed = cluster.pods[0].name
cluster.pods = cluster.pods[1:]
print(f" deleted {removed} manually")
step(cluster, dep, "the loop does not care how it happened:")
print(f" converged in {converge(cluster, dep)} more steps -> {line(cluster, dep)}")
print("--- 4. a rolling update to app:1.5 ---")
dep.image = "app:1.5"
served = []
for i in range(1, 10):
if rollout_done(cluster, dep):
print(f" step {i}: rollout complete — every pod on {dep.image}")
break
actions = reconcile(cluster, dep)
apply(cluster, actions, dep)
tick(cluster)
n = len(cluster.serving(dep))
served.append(n)
note = "" if actions else " (no action — waiting for the new pods to become ready)"
print(f" step {i}: {len(actions)} action(s) -> {line(cluster, dep)}{note}")
print(f" serving during rollout: {served} minimum {min(served) if served else 0}")
print(f" never dropped below max_unavailable allows: "
f"{dep.replicas - dep.max_unavailable} required, {min(served) if served else 0} observed")
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/reconcile.py (7764 bytes)
"""Kubernetes as one idea: a loop that moves actual state toward desired state.
Offline and standard-library only. No cluster is required, because the thing
worth understanding is not the YAML -- it is the control loop, and the loop is
about forty lines.
Everything else follows from it:
you declare 3 replicas of image app:1.4
the loop sees 2 running, 1 of them on a dead node
the loop acts delete the dead one, create two
It never asks how it got there. It compares what you asked for against what
exists and takes one step. That is why `kubectl delete pod` is not destructive
-- the loop simply notices the gap again and refills it -- and why a rolling
update is not a special operation but the same loop with a changed spec.
Readiness is the second idea. A pod that is Running is not necessarily able to
serve, and a Service routes only to pods that are READY. Yesterday's Compose
lesson had the same distinction; Kubernetes makes it explicit and continuous.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class Phase(str, Enum):
PENDING = "Pending" # accepted, not yet running
RUNNING = "Running" # the container is up
FAILED = "Failed" # exited non-zero, or its node went away
@dataclass
class Pod:
name: str
image: str
phase: Phase = Phase.PENDING
ready: bool = False # the readiness probe passes
node: str = "node-a"
@property
def serving(self) -> bool:
"""A Service routes here only if the pod is BOTH running and ready."""
return self.phase is Phase.RUNNING and self.ready
@dataclass
class Deployment:
"""The desired state. Note it says nothing about how to get there."""
name: str
image: str
replicas: int
max_unavailable: int = 1
max_surge: int = 1
@dataclass
class Action:
verb: str # create | delete
pod: str
reason: str
@dataclass
class Cluster:
pods: list[Pod] = field(default_factory=list)
nodes: list[str] = field(default_factory=lambda: ["node-a", "node-b", "node-c"])
dead_nodes: set[str] = field(default_factory=set)
_seq: int = 0
def schedule(self) -> str:
"""Pick a node for a new pod. Live nodes only.
Without this the loop places replacements on the node that just died,
reaps them next step, and churns forever -- which is a real failure
mode when a scheduler's node filter is wrong, not an artefact.
"""
live = [n for n in self.nodes if n not in self.dead_nodes]
if not live:
raise RuntimeError("no live node to schedule onto")
counts = {n: sum(1 for p in self.pods if p.node == n) for n in live}
return min(live, key=lambda n: (counts[n], n))
def alive(self, deployment: Deployment) -> list[Pod]:
"""Pods belonging to this deployment that are not doomed."""
return [
p for p in self.pods
if p.name.startswith(deployment.name + "-")
and p.phase is not Phase.FAILED
and p.node not in self.dead_nodes
]
def owned(self, deployment: Deployment) -> list[Pod]:
return [p for p in self.pods if p.name.startswith(deployment.name + "-")]
def serving(self, deployment: Deployment) -> list[Pod]:
"""What a Service would actually route traffic to."""
return [p for p in self.alive(deployment) if p.serving]
def next_name(self, deployment: Deployment) -> str:
self._seq += 1
return f"{deployment.name}-{self._seq:03d}"
def reconcile(cluster: Cluster, deployment: Deployment) -> list[Action]:
"""One step of the loop. Compare, then act. Never ask how we got here.
Deliberately returns a SINGLE step's worth of actions rather than looping
to convergence -- the loop runs forever in a real cluster, and modelling one
step is what makes the "it just notices again" behaviour visible.
"""
actions: list[Action] = []
# 1. Reap anything failed or stranded on a dead node.
for pod in cluster.owned(deployment):
if pod.phase is Phase.FAILED:
actions.append(Action("delete", pod.name, "pod failed"))
elif pod.node in cluster.dead_nodes:
actions.append(Action("delete", pod.name, f"node {pod.node} is gone"))
# 2. Anything running the wrong image is replaced, respecting the surge
# and unavailability budget so the service stays up during a rollout.
healthy = [p for p in cluster.alive(deployment) if p.image == deployment.image]
stale = [p for p in cluster.alive(deployment) if p.image != deployment.image]
serving_now = len([p for p in cluster.alive(deployment) if p.serving])
budget = max(0, serving_now - (deployment.replicas - deployment.max_unavailable))
for pod in stale[:budget]:
actions.append(Action("delete", pod.name, f"image {pod.image} != {deployment.image}"))
# 3. Create what is missing, without exceeding the surge ceiling.
# Two limits apply at once and both matter: we need `replicas` pods on
# the new image eventually, and we may not hold more than
# replicas + max_surge pods at any moment during a rollout.
kept = len(healthy) + max(0, len(stale) - budget)
ceiling = deployment.replicas + (deployment.max_surge if stale else 0)
wanted = min(deployment.replicas - len(healthy), ceiling - kept)
for _ in range(max(0, wanted)):
actions.append(
Action("create", cluster.next_name(deployment),
f"{kept} of {deployment.replicas} replicas present")
)
kept += 1
return actions
def apply(cluster: Cluster, actions: list[Action], deployment: Deployment) -> None:
"""Carry out one step's actions. New pods start Pending and not ready."""
for act in actions:
if act.verb == "delete":
cluster.pods = [p for p in cluster.pods if p.name != act.pod]
elif act.verb == "create":
cluster.pods.append(
Pod(act.pod, deployment.image, Phase.PENDING, False, cluster.schedule())
)
def tick(cluster: Cluster) -> None:
"""Time passes: pending pods start, running pods pass their probe."""
for pod in cluster.pods:
if pod.phase is Phase.PENDING:
pod.phase = Phase.RUNNING
elif pod.phase is Phase.RUNNING and not pod.ready:
pod.ready = True
def rollout_done(cluster: Cluster, deployment: Deployment) -> bool:
"""Whether every serving pod runs the desired image, at full replica count.
Distinct from "reconcile returned no actions". A step can legitimately have
nothing to do while old pods are still running -- the controller is waiting
for the new ones to pass their readiness probe before it is allowed to
retire any more. That pause is maxUnavailable doing its job.
"""
pods = cluster.alive(deployment)
return (
len(pods) == deployment.replicas
and all(p.image == deployment.image and p.serving for p in pods)
)
def converge(cluster: Cluster, deployment: Deployment, *, max_steps: int = 20) -> int:
"""Run until the spec is satisfied AND every pod is serving.
Two conditions, not one. A cluster with the right number of Running pods
has satisfied the spec and may still be routing traffic to none of them,
because readiness is a separate signal that arrives later.
"""
for step in range(1, max_steps + 1):
actions = reconcile(cluster, deployment)
if not actions and rollout_done(cluster, deployment):
return step - 1
apply(cluster, actions, deployment)
tick(cluster)
raise RuntimeError("did not converge; the desired state may be unreachable")
metadata.yml (1083 bytes)
lesson_id: D333
day: 333
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python3 examples/reconcile_demo.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 55
last_executed: '2026-08-31'
executed_on: 'macOS 26.5.1 (Apple Silicon), Python 3.14.0, pytest 9.1.1 — python3 examples/reconcile_demo.py
reproduced the documented behaviour exactly (a first deploy converging in 2 steps, a dead node and a
manual delete both healed by the same loop, and a rolling update to app:1.5 completing in 5 steps
with serving never dropping below the 2 that max_unavailable permits); bash tests/run_tests.sh -> 19
passed in 0.01s. The untouched starter fails 18 of 19; the one that passes exercises Pod.serving on a
FAILED pod, which is decided by the phase alone. requires_network is false and no cluster is
required: the idea worth learning is the control loop, and the loop is about forty lines.'
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/reconcile.py (5630 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
Kubernetes as one idea: a loop that moves actual state toward desired state.
Offline and standard-library only. No cluster is required, because the thing
worth understanding is not the YAML -- it is the control loop, and the loop is
about forty lines.
Everything else follows from it:
you declare 3 replicas of image app:1.4
the loop sees 2 running, 1 of them on a dead node
the loop acts delete the dead one, create two
It never asks how it got there. It compares what you asked for against what
exists and takes one step. That is why `kubectl delete pod` is not destructive
-- the loop simply notices the gap again and refills it -- and why a rolling
update is not a special operation but the same loop with a changed spec.
Readiness is the second idea. A pod that is Running is not necessarily able to
serve, and a Service routes only to pods that are READY. Yesterday's Compose
lesson had the same distinction; Kubernetes makes it explicit and continuous.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
class Phase(str, Enum):
PENDING = 'Pending'
RUNNING = 'Running'
FAILED = 'Failed'
@dataclass
class Pod:
name: str
image: str
phase: Phase = Phase.PENDING
ready: bool = False
node: str = 'node-a'
@property
def serving(self) -> bool:
"""A Service routes here only if the pod is BOTH running and ready."""
raise NotImplementedError('TASK 1: implement serving. A Service routes here only if the pod is BOTH running and ready.')
@dataclass
class Deployment:
"""The desired state. Note it says nothing about how to get there."""
name: str
image: str
replicas: int
max_unavailable: int = 1
max_surge: int = 1
@dataclass
class Action:
verb: str
pod: str
reason: str
@dataclass
class Cluster:
pods: list[Pod] = field(default_factory=list)
nodes: list[str] = field(default_factory=lambda: ['node-a', 'node-b', 'node-c'])
dead_nodes: set[str] = field(default_factory=set)
_seq: int = 0
def schedule(self) -> str:
"""Pick a node for a new pod. Live nodes only.
Without this the loop places replacements on the node that just died,
reaps them next step, and churns forever -- which is a real failure
mode when a scheduler's node filter is wrong, not an artefact.
"""
raise NotImplementedError('TASK 2: implement schedule. Pick a node for a new pod. Live nodes only.')
def alive(self, deployment: Deployment) -> list[Pod]:
"""Pods belonging to this deployment that are not doomed."""
raise NotImplementedError('TASK 3: implement alive. Pods belonging to this deployment that are not doomed.')
def owned(self, deployment: Deployment) -> list[Pod]:
raise NotImplementedError('TASK 4: implement owned.')
def serving(self, deployment: Deployment) -> list[Pod]:
"""What a Service would actually route traffic to."""
raise NotImplementedError('TASK 5: implement serving. What a Service would actually route traffic to.')
def next_name(self, deployment: Deployment) -> str:
raise NotImplementedError('TASK 6: implement next_name.')
def reconcile(cluster: Cluster, deployment: Deployment) -> list[Action]:
"""One step of the loop. Compare, then act. Never ask how we got here.
Deliberately returns a SINGLE step's worth of actions rather than looping
to convergence -- the loop runs forever in a real cluster, and modelling one
step is what makes the "it just notices again" behaviour visible.
"""
raise NotImplementedError('TASK 7: implement reconcile. One step of the loop. Compare, then act. Never ask how we got here.')
def apply(cluster: Cluster, actions: list[Action], deployment: Deployment) -> None:
"""Carry out one step's actions. New pods start Pending and not ready."""
raise NotImplementedError("TASK 8: implement apply. Carry out one step's actions. New pods start Pending and not ready.")
def tick(cluster: Cluster) -> None:
"""Time passes: pending pods start, running pods pass their probe."""
raise NotImplementedError('TASK 9: implement tick. Time passes: pending pods start, running pods pass their probe.')
def rollout_done(cluster: Cluster, deployment: Deployment) -> bool:
"""Whether every serving pod runs the desired image, at full replica count.
Distinct from "reconcile returned no actions". A step can legitimately have
nothing to do while old pods are still running -- the controller is waiting
for the new ones to pass their readiness probe before it is allowed to
retire any more. That pause is maxUnavailable doing its job.
"""
raise NotImplementedError('TASK 10: implement rollout_done. Whether every serving pod runs the desired image, at full replica count.')
def converge(cluster: Cluster, deployment: Deployment, *, max_steps: int=20) -> int:
"""Run until the spec is satisfied AND every pod is serving.
Two conditions, not one. A cluster with the right number of Running pods
has satisfied the spec and may still be routing traffic to none of them,
because readiness is a separate signal that arrives later.
"""
raise NotImplementedError('TASK 11: implement converge. Run until the spec is satisfied AND every pod is serving.')
tests/run_tests.sh (204 bytes)
#!/usr/bin/env bash
# Run the day 333 reconciliation test suite.
# Offline, standard library plus pytest. No cluster is required.
set -euo pipefail
cd "$(dirname "$0")/.."
python3 -m pytest tests -q "$@"
tests/test_reconcile.py (6015 bytes)
"""Grouped by what the control loop decides.
Run with: bash tests/run_tests.sh
"""
from __future__ import annotations
import os
import sys
import pytest
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(os.path.dirname(HERE), "examples"))
from reconcile import ( # noqa: E402
Cluster,
Deployment,
Phase,
Pod,
apply,
converge,
reconcile,
rollout_done,
tick,
)
def dep(replicas=3, image="app:1.4", **kw):
return Deployment("api", image, replicas, **kw)
def running(name, image="app:1.4", ready=True, node="node-b"):
return Pod(name, image, Phase.RUNNING, ready, node)
# ------------------------------------------------------------------ serving
def test_a_pod_must_be_both_running_and_ready_to_serve():
assert running("api-1").serving
assert not Pod("api-2", "app:1.4", Phase.RUNNING, False).serving
assert not Pod("api-3", "app:1.4", Phase.PENDING, True).serving
def test_a_failed_pod_never_serves():
assert not Pod("api-4", "app:1.4", Phase.FAILED, True).serving
# ------------------------------------------------------------ first deploy
def test_an_empty_cluster_creates_every_replica():
actions = reconcile(Cluster(), dep(replicas=3))
assert [a.verb for a in actions] == ["create"] * 3
def test_a_satisfied_deployment_produces_no_actions():
c = Cluster(pods=[running(f"api-{i}") for i in range(3)])
assert reconcile(c, dep(replicas=3)) == []
def test_convergence_reaches_the_declared_replica_count():
c = Cluster()
d = dep(replicas=4)
converge(c, d)
assert len(c.serving(d)) == 4
# ----------------------------------------------------------------- healing
def test_a_failed_pod_is_reaped_and_replaced():
c = Cluster(pods=[running("api-1"), Pod("api-2", "app:1.4", Phase.FAILED)])
actions = reconcile(c, dep(replicas=2))
assert any(a.verb == "delete" and a.pod == "api-2" for a in actions)
assert any(a.verb == "create" for a in actions)
def test_a_pod_on_a_dead_node_is_reaped():
c = Cluster(pods=[running("api-1", node="node-a")], dead_nodes={"node-a"})
actions = reconcile(c, dep(replicas=1))
assert any(a.verb == "delete" and "node-a" in a.reason for a in actions)
def test_a_manual_delete_is_simply_refilled():
# The loop never asks how the gap appeared.
c = Cluster(pods=[running("api-1"), running("api-2")])
d = dep(replicas=3)
actions = reconcile(c, d)
assert [a.verb for a in actions] == ["create"]
def test_replacements_are_not_scheduled_onto_a_dead_node():
# Without this the loop places a pod on the node that just died, reaps it
# next step, and churns forever.
c = Cluster(pods=[running("api-1", node="node-a")], dead_nodes={"node-a"})
d = dep(replicas=2)
converge(c, d)
assert all(p.node not in c.dead_nodes for p in c.alive(d))
def test_a_cluster_with_no_live_node_cannot_schedule():
c = Cluster(nodes=["node-a"], dead_nodes={"node-a"})
with pytest.raises(RuntimeError):
converge(c, dep(replicas=1))
# ----------------------------------------------------------------- scaling
def test_scaling_up_creates_the_difference_only():
c = Cluster(pods=[running(f"api-{i}") for i in range(2)])
assert len(reconcile(c, dep(replicas=5))) == 3
def test_scaling_to_zero_is_a_valid_desired_state():
c = Cluster(pods=[running("api-1")])
d = dep(replicas=0)
assert reconcile(c, d) == [] # nothing to create
assert rollout_done(c, d) is False # and one pod too many still exists
# ----------------------------------------------------------------- rollout
def test_a_changed_image_starts_a_rollout():
c = Cluster(pods=[running(f"api-{i}") for i in range(3)])
actions = reconcile(c, dep(replicas=3, image="app:1.5"))
assert any(a.verb == "delete" and "1.4" in a.reason for a in actions)
def test_a_rollout_never_exceeds_replicas_plus_surge():
c = Cluster(pods=[running(f"api-{i}") for i in range(3)])
d = dep(replicas=3, image="app:1.5", max_surge=1)
for _ in range(8):
if rollout_done(c, d):
break
apply(c, reconcile(c, d), d)
tick(c)
assert len(c.alive(d)) <= d.replicas + d.max_surge
def test_a_rollout_never_drops_below_the_unavailability_budget():
# THE guarantee: max_unavailable is what keeps the service up.
c = Cluster(pods=[running(f"api-{i}") for i in range(3)])
d = dep(replicas=3, image="app:1.5", max_unavailable=1)
floor = d.replicas - d.max_unavailable
seen = []
for _ in range(8):
if rollout_done(c, d):
break
apply(c, reconcile(c, d), d)
tick(c)
seen.append(len(c.serving(d)))
assert seen and min(seen) >= floor
def test_a_rollout_finishes_with_every_pod_on_the_new_image():
c = Cluster(pods=[running(f"api-{i}") for i in range(3)])
d = dep(replicas=3, image="app:1.5")
converge(c, d)
assert rollout_done(c, d)
assert {p.image for p in c.alive(d)} == {"app:1.5"}
def test_no_actions_does_not_mean_the_rollout_is_finished():
# A step can legitimately have nothing to do while old pods still run --
# the controller is waiting for the new ones to pass their probe.
c = Cluster(pods=[running(f"api-{i}") for i in range(3)])
d = dep(replicas=3, image="app:1.5")
apply(c, reconcile(c, d), d)
tick(c)
assert reconcile(c, d) == []
assert not rollout_done(c, d)
# -------------------------------------------------------------- rollout_done
def test_rollout_done_requires_readiness_not_just_running():
c = Cluster(pods=[Pod(f"api-{i}", "app:1.4", Phase.RUNNING, False) for i in range(3)])
d = dep(replicas=3)
assert reconcile(c, d) == [] # the spec is satisfied
assert not rollout_done(c, d) # and nothing is serving
def test_rollout_done_requires_the_full_replica_count():
c = Cluster(pods=[running("api-1"), running("api-2")])
assert not rollout_done(c, dep(replicas=3))
Troubleshooting
Troubleshooting — Day 333
NotImplementedError on nearly every test
Expected. The starter stubs eleven functions — see expected-output/starter-run.txt, which names the one test that passes without them.
Start with Cluster.alive and Pod.serving; reconcile is meaningless until they are right.
converge raises "did not converge"
The loop is oscillating. Two causes, and the first is far more common:
Replacements scheduled onto a dead node. The loop creates a pod on the node that just died, reaps it next step, creates another, forever. Cluster.schedule must filter dead_nodes. This is a real failure mode when a scheduler's node filter is wrong, not an artefact of the model.
A ceiling below the replica count. If wanted can never be positive, the loop can never reach the desired state.
test_a_rollout_never_exceeds_replicas_plus_surge fails
Your create step is not accounting for the stale pods it is keeping. Two limits apply at once: you need replicas pods on the new image eventually, and you may hold at most replicas + max_surge pods at any moment. Take the minimum of both.
test_a_rollout_never_drops_below_the_unavailability_budget fails
The delete budget is serving_now - (replicas - max_unavailable), and it must be clamped at zero. If you delete stale pods without checking how many are currently serving, the rollout takes the service down — which is exactly what maxUnavailable exists to prevent.
test_no_actions_does_not_mean_the_rollout_is_finished fails
This is the subtle one and it is worth understanding rather than working around. A reconcile step can legitimately return no actions while old pods are still running: the controller has already replaced as many as the budget allows and is now waiting for the new pods to pass their readiness probe before it may retire any more.
"Nothing to do right now" and "the rollout is complete" are different questions. That is what rollout_done is for.
test_rollout_done_requires_readiness_not_just_running fails
rollout_done must check serving, not phase. A cluster with the right number of Running pods has satisfied the spec and may still be routing traffic to none of them.
Pods are created but never serve
tick advances one state per call: Pending becomes Running, then Running becomes ready. A pod therefore needs two ticks before it serves, which is why a first deploy converges in two steps rather than one.
test_scaling_to_zero_is_a_valid_desired_state fails
Zero replicas is a legitimate thing to declare — it is how you take something down without deleting it. Nothing should be created, and rollout_done should be False while a pod still exists.
Security notes
Security notes — Day 333
What this lab does and does not touch
- No Kubernetes cluster required and none is contacted. The cluster is a Python dataclass. Nothing is scheduled, no kubeconfig is read, no API server is reached.
- No network, no credentials, no cloud account.
- No writes outside this directory. Only
__pycache__/,.pytest_cache/and, if you create it,.venv/. - No
sudo, no global installs. Installing a real cluster is neither, and this lab deliberately does not ask you to.
The control loop is a security property
A loop that continuously drives actual state toward declared state has a consequence people meet by accident: manual changes do not persist.
kubectl edit a Deployment during an incident and the change survives. kubectl edit the Pod it created and the change is erased the moment the controller notices. That is the intended behaviour, and it is why the declared state — the thing in Git — is the only durable configuration. An attacker who modifies a running pod has made a change with a short half-life; an attacker who modifies your manifests has made a permanent one.
This is the argument for treating the manifest repository as production infrastructure with the review requirements that implies.
Readiness probes gate traffic, and that makes them a control
A Service routes only to pods that are ready. That means a readiness probe is not diagnostics — it decides whether requests reach a process.
Two failure modes, in opposite directions:
- A probe that always passes routes traffic to a pod that cannot serve it. This is the Compose failure from yesterday, made continuous.
- A probe that is too strict — one that checks a downstream dependency — removes every pod from the Service the moment that dependency wobbles, converting a partial outage into a total one. Probe what this pod can do, not what the system can do.
What this model deliberately omits
It is a teaching model of the loop, not of Kubernetes' security surface. Absent here and important in a real cluster:
- RBAC. Who may change the declared state is the whole question, and this model has no notion of an actor at all.
- Namespaces and NetworkPolicy. Every pod here can reach every other; a real cluster's default is the same, which is why NetworkPolicy exists.
- Pod security context.
runAsNonRoot, dropped capabilities, a read-only root filesystem. Day 330'sUSERfinding is the same idea one layer down. - Secrets. Kubernetes Secrets are base64, not encrypted, unless encryption at rest is configured. Base64 is an encoding.
Day 344 threat-models these properly. The point today is that the loop's behaviour — declared state wins, always, eventually — is the foundation the rest sits on.
Reversing everything this lab did
rm -rf __pycache__ .pytest_cache .venv
Nothing else was created, and nothing outside this directory was modified.