Deployment, MLOps, and Security › Operating AI in Production › Day 341
Hands-on lab — Day 341: Rollouts, A/B Tests, and Feature Flags
- ← Back to the Day 341 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/deployment-mlops-and-security/day-341-rollouts-ab-tests-and-feature-flags/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/feature_flags.py Test
bash tests/run_tests.sh File tree
examples/feature_flags.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/feature_flags.py tests/run_tests.sh tests/test_feature_flags.py troubleshooting.md
Lab README
Day 341 Lab: Rollouts, A/B Tests, and Feature Flags
Lesson
- Lesson title: Rollouts, A/B Tests, and Feature Flags
- Day number: 341 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-341-rollouts-ab-tests-and-feature-flags
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-341-rollouts-ab-tests-and-feature-flagswhen the site is running.
Purpose
Build an AI Feature Flag & Shadow Routing Engine in Python implementing deterministic user hashing, percentage canary splits, and asynchronous shadow traffic mirroring.
Learning objectives
- Implement deterministic user hashing across persistent buckets.
- Route user traffic based on configurable percentage thresholds.
- Execute shadow traffic mirroring without affecting live response latency.
- Log shadow execution metrics for offline model comparison.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/feature_flags.py: Starter implementation skeletonexamples/feature_flags.py: Verified reference implementationtests/test_feature_flags.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/feature_flags.py
What the commands do
- Evaluates deterministic user hashing, tests canary thresholds, and verifies shadow logging.
Expected output
{'user_id': 'u1', 'bucket': 7, 'assigned_variant': 'CANDIDATE_MODEL_V2', 'response': "[CANDIDATE_MODEL_V2] Response for 'hello'", 'shadow_executed': True}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Deterministic hashing producing identical buckets for same user
- Correct routing to Candidate when bucket < canary percentage
- Correct routing to Baseline when bucket >= canary percentage
- Shadow execution and audit logging when enabled
- Zero shadow execution when disabled
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Use MD5 or SHA256 for deterministic hashing across runs.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement a Thompson Sampling Multi-Armed Bandit router.
Navigation
Day number: 341 of 365
Expected output
FIELDS.md
# Measured Values
- `routes_executed`: 10
- `shadow_logs_captured`: 4
examples-run.txt
{'user_id': 'u1', 'bucket': 7, 'assigned_variant': 'CANDIDATE_MODEL_V2', 'response': "[CANDIDATE_MODEL_V2] Response for 'hello'", 'shadow_executed': True}
measured-values.txt
routes_executed=10
shadow_logs_captured=4
run-output.txt
{'user_id': 'u1', 'bucket': 7, 'assigned_variant': 'CANDIDATE_MODEL_V2', 'response': "[CANDIDATE_MODEL_V2] Response for 'hello'", 'shadow_executed': True}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.feature_flags.AIFeatureFlagRouter object at 0x10b63a650>
user_id = 'alice'
def get_user_bucket(self, user_id: str) -> int:
> raise NotImplementedError('TASK 1: implement get_user_bucket.')
E NotImplementedError: TASK 1: implement get_user_bucket.
examples/feature_flags.py:21: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_feature_flags.py::test_deterministic_user_hashing - NotImpl...
FAILED tests/test_feature_flags.py::test_canary_routing_allocation - NotImple...
FAILED tests/test_feature_flags.py::test_shadow_traffic_execution_when_enabled
FAILED tests/test_feature_flags.py::test_shadow_traffic_disabled - NotImpleme...
FAILED tests/test_feature_flags.py::test_different_flags_produce_different_buckets
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-341-rollouts-ab-tests-and-feature-flags
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_feature_flags.py::test_deterministic_user_hashing PASSED [ 20%]
tests/test_feature_flags.py::test_canary_routing_allocation PASSED [ 40%]
tests/test_feature_flags.py::test_shadow_traffic_execution_when_enabled PASSED [ 60%]
tests/test_feature_flags.py::test_shadow_traffic_disabled PASSED [ 80%]
tests/test_feature_flags.py::test_different_flags_produce_different_buckets PASSED [100%]
============================== 5 passed in 0.02s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/deployment-mlops-and-security/day-341-rollouts-ab-tests-and-feature-flags
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_feature_flags.py::test_deterministic_user_hashing PASSED [ 20%]
tests/test_feature_flags.py::test_canary_routing_allocation PASSED [ 40%]
tests/test_feature_flags.py::test_shadow_traffic_execution_when_enabled PASSED [ 60%]
tests/test_feature_flags.py::test_shadow_traffic_disabled PASSED [ 80%]
tests/test_feature_flags.py::test_different_flags_produce_different_buckets PASSED [100%]
============================== 5 passed in 0.02s ===============================
Source files
examples/feature_flags.py (1815 bytes)
import hashlib
import time
from typing import Dict, Any, List, Optional
class AIFeatureFlagRouter:
def __init__(self, flag_name: str, canary_percentage: int = 10, shadow_enabled: bool = False):
self.flag_name = flag_name
self.canary_pct = max(0, min(100, int(canary_percentage)))
self.shadow_enabled = bool(shadow_enabled)
self.shadow_logs: List[Dict[str, Any]] = []
def get_user_bucket(self, user_id: str) -> int:
hash_input = f"{self.flag_name}:{user_id}".encode("utf-8")
hash_hex = hashlib.md5(hash_input).hexdigest()
return int(hash_hex[:8], 16) % 100
def route_request(self, user_id: str, prompt: str) -> Dict[str, Any]:
bucket = self.get_user_bucket(user_id)
if bucket < self.canary_pct:
live_variant = "CANDIDATE_MODEL_V2"
else:
live_variant = "BASELINE_MODEL_V1"
live_response = f"[{live_variant}] Response for '{prompt}'"
shadow_executed = False
if self.shadow_enabled:
shadow_variant = "SHADOW_EXPERIMENTAL_V3"
shadow_response = f"[{shadow_variant}] Response for '{prompt}'"
self.shadow_logs.append({
"user_id": user_id,
"prompt": prompt,
"shadow_variant": shadow_variant,
"shadow_response": shadow_response,
"timestamp": time.time()
})
shadow_executed = True
return {
"user_id": user_id,
"bucket": bucket,
"assigned_variant": live_variant,
"response": live_response,
"shadow_executed": shadow_executed
}
if __name__ == "__main__":
r = AIFeatureFlagRouter("test_flag", 20, True)
print(r.route_request("u1", "hello"))
metadata.yml (400 bytes)
lesson_id: D341
day: 341
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/feature_flags.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/feature_flags.py (1068 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import hashlib
import time
from typing import Dict, Any, List, Optional
class AIFeatureFlagRouter:
def __init__(self, flag_name: str, canary_percentage: int=10, shadow_enabled: bool=False):
self.flag_name = flag_name
self.canary_pct = max(0, min(100, int(canary_percentage)))
self.shadow_enabled = bool(shadow_enabled)
self.shadow_logs: List[Dict[str, Any]] = []
def get_user_bucket(self, user_id: str) -> int:
raise NotImplementedError('TASK 1: implement get_user_bucket.')
def route_request(self, user_id: str, prompt: str) -> Dict[str, Any]:
raise NotImplementedError('TASK 2: implement route_request.')
if __name__ == '__main__':
r = AIFeatureFlagRouter('test_flag', 20, True)
print(r.route_request('u1', 'hello'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_feature_flags.py (2071 bytes)
import pytest
from examples.feature_flags import AIFeatureFlagRouter
def test_deterministic_user_hashing():
router = AIFeatureFlagRouter("test_flag", canary_percentage=20)
bucket1 = router.get_user_bucket("user_alice")
bucket2 = router.get_user_bucket("user_alice")
bucket_bob = router.get_user_bucket("user_bob")
assert bucket1 == bucket2
assert 0 <= bucket1 < 100
assert 0 <= bucket_bob < 100
def test_canary_routing_allocation():
# Set 100% canary -> all users must get CANDIDATE_MODEL_V2
router_all_canary = AIFeatureFlagRouter("flag1", canary_percentage=100)
res1 = router_all_canary.route_request("u1", "test")
assert res1["assigned_variant"] == "CANDIDATE_MODEL_V2"
# Set 0% canary -> all users must get BASELINE_MODEL_V1
router_zero_canary = AIFeatureFlagRouter("flag1", canary_percentage=0)
res2 = router_zero_canary.route_request("u1", "test")
assert res2["assigned_variant"] == "BASELINE_MODEL_V1"
def test_shadow_traffic_execution_when_enabled():
router = AIFeatureFlagRouter("flag_shadow", canary_percentage=10, shadow_enabled=True)
res = router.route_request("usr_100", "Summarize article")
assert res["shadow_executed"] is True
assert len(router.shadow_logs) == 1
assert router.shadow_logs[0]["user_id"] == "usr_100"
assert router.shadow_logs[0]["shadow_variant"] == "SHADOW_EXPERIMENTAL_V3"
def test_shadow_traffic_disabled():
router = AIFeatureFlagRouter("flag_no_shadow", canary_percentage=10, shadow_enabled=False)
res = router.route_request("usr_100", "Summarize article")
assert res["shadow_executed"] is False
assert len(router.shadow_logs) == 0
def test_different_flags_produce_different_buckets():
r1 = AIFeatureFlagRouter("flag_a", 50)
r2 = AIFeatureFlagRouter("flag_b", 50)
# Flags with different salts generally produce different buckets for the same user
b1 = r1.get_user_bucket("alice")
b2 = r2.get_user_bucket("alice")
# Both are valid [0, 99]
assert 0 <= b1 < 100 and 0 <= b2 < 100