Machine Learning › Machine Learning in Practice › Day 190

Hands-on lab — Day 190: The ML Project Lifecycle

← Back to the Day 190 lesson
Open the hands-on files on GitHub — clone or download them from the public labs repository
Local path in your clone: labs/sections/machine-learning/day-190-the-ml-project-lifecycle/

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/the_ml_project_lifecycle_lib.py

Test

./tests/run_tests.sh

File tree

examples/test_the_ml_project_lifecycle_lib.py
examples/the_ml_project_lifecycle_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_the_ml_project_lifecycle_lib.py
starter/the_ml_project_lifecycle_lib.py
tests/run_tests.sh
tests/test_the_ml_project_lifecycle_lib.py
troubleshooting.md

Lab README

Lab: Day 190 -- The ML Project Lifecycle

Lesson

Day number: 190 of 365. Course: Course04-SS03 (Beyond Supervised Learning). Topic: Machine Learning Project Lifecycle and Deployment Quality Gates.

Purpose

Build a complete, automated deployment quality gate engine in pure Python. You will implement statistical superiority checks, cohort slice regression guards, operational latency/memory SLA validations, and safety infrastructure checks before promoting candidate models to production traffic.

Learning objectives

Model the 6 phases of the production ML lifecycle.
Formulate quantitative deployment readiness criteria.
Enforce strict subgroup slicing checks to prevent demographic or cohort regression.
Validate operational SLA bounds (p99 latency, container RAM) and circuit breaker readiness.

Prerequisites

Python 3.11+ data structures (dataclasses, dictionaries).
Core understanding of classification metrics (PR-AUC, precision, recall).

Supported operating systems

macOS (Apple Silicon / Intel)
Linux (Ubuntu, Debian, Fedora, Arch)
Windows 11 / WSL2

Hardware requirements

1+ CPU cores.
512 MB RAM.
50 MB disk space.

Required software

Python 3.11 or newer.
pip package manager.
virtualenv or venv module.

Free and open-source options

All tools used in this lab (Python, pytest) are free and open-source under BSD/MIT licenses.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

starter/the_ml_project_lifecycle_lib.py: Student scaffold file.
examples/the_ml_project_lifecycle_lib.py: Complete reference implementation.
tests/test_the_ml_project_lifecycle_lib.py: Pytest automated validation suite.
expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/the_ml_project_lifecycle_lib.py

What the commands do

Evaluates candidate model report v1.1.0 against production champion v1.0.0.
Executes the 4-quadrant quality gate checklist.
Logs final canary promotion decision.

Expected output

ML Lifecycle Demo: Candidate Passed All Gates = True

Validation steps

Verify that overall PR-AUC improvement is >= 0.015 (+1.5%).
Verify that no subgroup slice regresses by more than 0.02 (-2.0%).
Verify that p99 latency is within 20ms.
Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

Missing Slice Key: Ensure candidate dictionary includes all cohorts evaluated in champion report.

Security notes

All computations execute locally without external network transmission.

Extension exercises

Integrate Expected Value Cost Matrix calculation into decision rules.
Build an automated markdown Model Card audit generator.

Lesson title: The ML Project Lifecycle
Day number: 190 of 365
Lesson article: https://ai-roadmap-365.github.io/day-190-the-ml-project-lifecycle
Lab files: everything you need is in this directory — follow “How to run” below.
Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-190-the-ml-project-lifecycle when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 190

- `Candidate Passed All Gates`: Boolean decision (True/False).
- `Metric Improvement`: Delta between candidate and champion PR-AUC.
- `Latency p99`: Measured 99th percentile inference latency in milliseconds.

examples-run.txt

ML Lifecycle Demo: Candidate Passed All Gates = True

measured-values.txt

Candidate Passed All Gates: True
Metric Improvement: 0.0230
Latency p99: 11.2000

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_the_ml_project_lifecycle_lib.py::test_quality_gate_passes_superior_candidate PASSED [ 50%]
tests/test_the_ml_project_lifecycle_lib.py::test_quality_gate_blocks_slice_regression PASSED   [100%]

============================== 2 passed in 0.08s ===============================

Source files

examples/test_the_ml_project_lifecycle_lib.py (1876 bytes)

import pytest
from examples.the_ml_project_lifecycle_lib import ModelEvaluationReport, DeploymentQualityGateEngine

def test_quality_gate_passes_superior_candidate():
    champ = ModelEvaluationReport(
        model_name="fraud_model", version="v1.0",
        overall_pr_auc=0.800,
        slice_pr_auc={"tier1": 0.820, "tier2": 0.780},
        p99_latency_ms=10.0, memory_mb=500.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    cand = ModelEvaluationReport(
        model_name="fraud_model", version="v1.1",
        overall_pr_auc=0.825, # +0.025 improvement
        slice_pr_auc={"tier1": 0.830, "tier2": 0.810},
        p99_latency_ms=12.0, memory_mb=600.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    engine = DeploymentQualityGateEngine(min_pr_auc_improvement=0.015)
    res = engine.evaluate_gates(cand, champ)
    assert res["passed_all"] is True
    assert res["checks"]["metric_superiority"]["passed"] is True

def test_quality_gate_blocks_slice_regression():
    champ = ModelEvaluationReport(
        model_name="fraud_model", version="v1.0",
        overall_pr_auc=0.800,
        slice_pr_auc={"tier1": 0.820, "tier2": 0.780},
        p99_latency_ms=10.0, memory_mb=500.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    cand = ModelEvaluationReport(
        model_name="fraud_model", version="v1.1",
        overall_pr_auc=0.830, # +0.030 improvement
        slice_pr_auc={"tier1": 0.890, "tier2": 0.730}, # -0.050 drop on tier2!
        p99_latency_ms=12.0, memory_mb=600.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    engine = DeploymentQualityGateEngine(max_slice_drop=0.02)
    res = engine.evaluate_gates(cand, champ)
    assert res["passed_all"] is False
    assert res["checks"]["slice_regression"]["passed"] is False

examples/the_ml_project_lifecycle_lib.py (3754 bytes)

from dataclasses import dataclass
from typing import Dict, Any

@dataclass
class ModelEvaluationReport:
    model_name: str
    version: str
    overall_pr_auc: float
    slice_pr_auc: Dict[str, float]
    p99_latency_ms: float
    memory_mb: float
    has_schema_validation: bool
    has_fallback_circuit_breaker: bool

class DeploymentQualityGateEngine:
    def __init__(self, min_pr_auc_improvement: float = 0.015, max_slice_drop: float = 0.02,
                 max_p99_latency_ms: float = 20.0, max_memory_mb: float = 2000.0):
        self.min_pr_auc_improvement = min_pr_auc_improvement
        self.max_slice_drop = max_slice_drop
        self.max_p99_latency_ms = max_p99_latency_ms
        self.max_memory_mb = max_memory_mb

    def evaluate_gates(self, candidate: ModelEvaluationReport, champion: ModelEvaluationReport) -> Dict[str, Any]:
        results = {"passed_all": True, "checks": {}}

        # Gate 1: Overall Metric Superiority
        improvement = candidate.overall_pr_auc - champion.overall_pr_auc
        gate1_passed = improvement >= self.min_pr_auc_improvement
        results["checks"]["metric_superiority"] = {
            "passed": gate1_passed,
            "improvement": round(improvement, 4),
            "required": self.min_pr_auc_improvement
        }

        # Gate 2: Subgroup Slice Regression
        slice_passed = True
        slice_details = {}
        for s_name, champ_score in champion.slice_pr_auc.items():
            cand_score = candidate.slice_pr_auc.get(s_name, 0.0)
            diff = cand_score - champ_score
            passed = diff >= -self.max_slice_drop
            slice_details[s_name] = {"diff": round(diff, 4), "passed": passed}
            if not passed:
                slice_passed = False

        results["checks"]["slice_regression"] = {
            "passed": slice_passed,
            "details": slice_details
        }

        # Gate 3: Operational Latency and Memory SLA
        lat_passed = candidate.p99_latency_ms <= self.max_p99_latency_ms
        mem_passed = candidate.memory_mb <= self.max_memory_mb
        results["checks"]["operational_sla"] = {
            "passed": lat_passed and mem_passed,
            "latency_p99_ms": candidate.p99_latency_ms,
            "memory_mb": candidate.memory_mb
        }

        # Gate 4: Safety, Schema, and Circuit Breakers
        safety_passed = candidate.has_schema_validation and candidate.has_fallback_circuit_breaker
        results["checks"]["safety_infrastructure"] = {
            "passed": safety_passed,
            "schema_validated": candidate.has_schema_validation,
            "circuit_breaker": candidate.has_fallback_circuit_breaker
        }

        results["passed_all"] = gate1_passed and slice_passed and lat_passed and mem_passed and safety_passed
        return results

def run_lifecycle_demo():
    champ = ModelEvaluationReport(
        model_name="churn_classifier", version="v1.0.0",
        overall_pr_auc=0.8420,
        slice_pr_auc={"mobile": 0.835, "desktop": 0.850, "international": 0.810},
        p99_latency_ms=8.5, memory_mb=450.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    cand = ModelEvaluationReport(
        model_name="churn_classifier", version="v1.1.0",
        overall_pr_auc=0.8650,
        slice_pr_auc={"mobile": 0.860, "desktop": 0.871, "international": 0.805},
        p99_latency_ms=11.2, memory_mb=620.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    engine = DeploymentQualityGateEngine()
    eval_res = engine.evaluate_gates(cand, champ)
    print(f"ML Lifecycle Demo: Candidate Passed All Gates = {eval_res['passed_all']}")
    return engine, eval_res

if __name__ == "__main__":
    run_lifecycle_demo()

metadata.yml (437 bytes)

lesson_id: D190
day: 190
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/the_ml_project_lifecycle_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'

requirements/requirements.txt (14 bytes)

pytest>=8.0.0

starter/test_the_ml_project_lifecycle_lib.py (1876 bytes)

import pytest
from examples.the_ml_project_lifecycle_lib import ModelEvaluationReport, DeploymentQualityGateEngine

def test_quality_gate_passes_superior_candidate():
    champ = ModelEvaluationReport(
        model_name="fraud_model", version="v1.0",
        overall_pr_auc=0.800,
        slice_pr_auc={"tier1": 0.820, "tier2": 0.780},
        p99_latency_ms=10.0, memory_mb=500.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    cand = ModelEvaluationReport(
        model_name="fraud_model", version="v1.1",
        overall_pr_auc=0.825, # +0.025 improvement
        slice_pr_auc={"tier1": 0.830, "tier2": 0.810},
        p99_latency_ms=12.0, memory_mb=600.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    engine = DeploymentQualityGateEngine(min_pr_auc_improvement=0.015)
    res = engine.evaluate_gates(cand, champ)
    assert res["passed_all"] is True
    assert res["checks"]["metric_superiority"]["passed"] is True

def test_quality_gate_blocks_slice_regression():
    champ = ModelEvaluationReport(
        model_name="fraud_model", version="v1.0",
        overall_pr_auc=0.800,
        slice_pr_auc={"tier1": 0.820, "tier2": 0.780},
        p99_latency_ms=10.0, memory_mb=500.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    cand = ModelEvaluationReport(
        model_name="fraud_model", version="v1.1",
        overall_pr_auc=0.830, # +0.030 improvement
        slice_pr_auc={"tier1": 0.890, "tier2": 0.730}, # -0.050 drop on tier2!
        p99_latency_ms=12.0, memory_mb=600.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    engine = DeploymentQualityGateEngine(max_slice_drop=0.02)
    res = engine.evaluate_gates(cand, champ)
    assert res["passed_all"] is False
    assert res["checks"]["slice_regression"]["passed"] is False

starter/the_ml_project_lifecycle_lib.py (928 bytes)

from dataclasses import dataclass
from typing import Dict, Any

@dataclass
class ModelEvaluationReport:
    model_name: str
    version: str
    overall_pr_auc: float
    slice_pr_auc: Dict[str, float]
    p99_latency_ms: float
    memory_mb: float
    has_schema_validation: bool
    has_fallback_circuit_breaker: bool

class DeploymentQualityGateEngine:
    def __init__(self, min_pr_auc_improvement: float = 0.015, max_slice_drop: float = 0.02,
                 max_p99_latency_ms: float = 20.0, max_memory_mb: float = 2000.0):
        self.min_pr_auc_improvement = min_pr_auc_improvement
        self.max_slice_drop = max_slice_drop
        self.max_p99_latency_ms = max_p99_latency_ms
        self.max_memory_mb = max_memory_mb

    def evaluate_gates(self, candidate: ModelEvaluationReport, champion: ModelEvaluationReport) -> Dict[str, Any]:
        # TODO: Implement 4-stage quality gate verification logic
        pass

tests/run_tests.sh (227 bytes)

#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 190 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."

tests/test_the_ml_project_lifecycle_lib.py (1876 bytes)

import pytest
from examples.the_ml_project_lifecycle_lib import ModelEvaluationReport, DeploymentQualityGateEngine

def test_quality_gate_passes_superior_candidate():
    champ = ModelEvaluationReport(
        model_name="fraud_model", version="v1.0",
        overall_pr_auc=0.800,
        slice_pr_auc={"tier1": 0.820, "tier2": 0.780},
        p99_latency_ms=10.0, memory_mb=500.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    cand = ModelEvaluationReport(
        model_name="fraud_model", version="v1.1",
        overall_pr_auc=0.825, # +0.025 improvement
        slice_pr_auc={"tier1": 0.830, "tier2": 0.810},
        p99_latency_ms=12.0, memory_mb=600.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    engine = DeploymentQualityGateEngine(min_pr_auc_improvement=0.015)
    res = engine.evaluate_gates(cand, champ)
    assert res["passed_all"] is True
    assert res["checks"]["metric_superiority"]["passed"] is True

def test_quality_gate_blocks_slice_regression():
    champ = ModelEvaluationReport(
        model_name="fraud_model", version="v1.0",
        overall_pr_auc=0.800,
        slice_pr_auc={"tier1": 0.820, "tier2": 0.780},
        p99_latency_ms=10.0, memory_mb=500.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    cand = ModelEvaluationReport(
        model_name="fraud_model", version="v1.1",
        overall_pr_auc=0.830, # +0.030 improvement
        slice_pr_auc={"tier1": 0.890, "tier2": 0.730}, # -0.050 drop on tier2!
        p99_latency_ms=12.0, memory_mb=600.0,
        has_schema_validation=True, has_fallback_circuit_breaker=True
    )
    engine = DeploymentQualityGateEngine(max_slice_drop=0.02)
    res = engine.evaluate_gates(cand, champ)
    assert res["passed_all"] is False
    assert res["checks"]["slice_regression"]["passed"] is False

Troubleshooting

Troubleshooting: Day 190 - The ML Project Lifecycle

Common Issues

Unchecked Subgroups:
- Cause: Missing slice dictionary entries.
- Fix: Use .get(key, 0.0) with fallback.

Security notes

Security & Privacy: Day 190 - The ML Project Lifecycle

Security Guidance

All gate validations run locally without network transmission.

Hands-on lab — Day 190: The ML Project Lifecycle

Commands

Setup

Run

Test

File tree

Lab README

Lab: Day 190 -- The ML Project Lifecycle

Lesson

Purpose

Learning objectives

Prerequisites

Supported operating systems

Hardware requirements

Required software

Free and open-source options

Installation

File structure

How to run

What the commands do

Expected output

Validation steps

Tests

Cleanup

Troubleshooting

Security notes

Extension exercises

Navigation

Expected output

FIELDS.md

examples-run.txt

measured-values.txt

starter-run.txt

test-run.txt

Source files

Troubleshooting

Troubleshooting: Day 190 - The ML Project Lifecycle

Common Issues

Security notes

Security & Privacy: Day 190 - The ML Project Lifecycle

Security Guidance