Deployment, MLOps, and Security β€Ί Operating AI in Production β€Ί Day 341

Day 341: Rollouts, A/B Tests, and Feature Flags

Day 341 of 365 β€” Rollouts, A/B Tests, and Feature Flags

Master advanced AI release engineering: implement feature flags, dark traffic shadowing, progressive canary rollouts, multi-armed bandits, and online A/B testing in Python.

Course
Deployment, MLOps, and Security
Category
Operating AI in Production
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deployment-mlops-and-security/day-341-rollouts-ab-tests-and-feature-flags

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deployment-mlops-and-security/day-341-rollouts-ab-tests-and-feature-flags
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Deploying a new prompt template, fine-tuned weights checkpoint, or model routing logic directly to 100% of production users in a single cutover is a high-risk gamble:

Mastering Feature Flagging, Deterministic User Hashing, Shadow Traffic Mirroring, Canary Rollouts, and Multi-Armed Bandits enables zero-risk releases, scientific A/B validation, and automated safety rollbacks.

Shadow Traffic Mirroring and Canary Router showing Live Ingress Request sent to Production Model (100% User Return) while Async Shadow Copy is evaluated on Candidate Model with Metrics Comparison

The idea in plain language

Diagram: deterministic cohort assignment compared with unseeded random assignment

Diagram: the progression from shadow traffic to a full rollout

Think of AI release engineering like a car manufacturer testing a new electric engine design:

Historical background

Diagram: the progression from big-bang deployments to AI shadow mirroring and bandits

Release engineering methodologies evolved across three major eras:

1. Big-Bang Monolithic Deployments (2000–2014)

Software teams scheduled midnight weekend maintenance windows. Upgrades were all-or-nothing; any bug required a panic-driven multi-hour rollback.

2. Feature Flags and Progressive Delivery (2015–2022)

Tools like LaunchDarkly and Unleash popularized feature flags. Progressive canary rollouts (5% βž” 25% βž” 100%) became standard practice for microservices.

3. AI Shadow Mirroring & Multi-Armed Bandits (2023–Present)

Modern AI gateways dynamically mirror live prompt streams, evaluate automated LLM-as-judge scores between models in the background, and use Thompson Sampling bandits to maximize conversion rates across prompt variants.

What it is β€” and what it is not

Let us define the core architecture of production AI rollouts:

What AI Release Engineering Is:

What AI Release Engineering Is Not:

Diagram: correct AI release practice compared with two shortcuts

+-------------------------------------------------------------------------------+
|                    AI RELEASE STRATEGY COMPARISON MATRIX                      |
+-------------------+-----------------------+-------------------+---------------+
| Strategy          | User Impact           | Latency Penalty   | Best For      |
+-------------------+-----------------------+-------------------+---------------+
| Shadow Traffic    | 0% (Response is       | Zero (Async       | High-risk new |
| Mirroring         | discarded)            | background call)  | model engines |
| Canary Rollout    | Bounded (e.g. 5% of   | None (Normal live | Progressive   |
| (Percentage)      | users exposed)        | execution)        | code updates  |
| Fixed A/B Test    | 50/50 Split across    | None (Fixed cohort| Scientific KPI|
|                   | distinct cohorts      | allocation)       | measurement   |
| Multi-Armed Bandit| Adaptive (Shifts to   | None (Dynamic     | Maximizing    |
| (Thompson Sample) | winning variant)      | reward routing)   | conversions   |
+-------------------+-----------------------+-------------------+---------------+

Why it was created and what problems it solves

Progressive AI release patterns resolve the three primary hazards of AI deployment:

1. Zero-Risk Real-World Stress Testing

Shadow traffic exposes candidate models to authentic user prompt distributions, edge cases, and peak concurrency without risking user-facing errors.

2. Elimination of UX Inconsistency via Deterministic Hashing

Hashing user identifiers ensures that User A always sees Variant B throughout their workflow, eliminating confusing personality shifts across consecutive chat turns.

3. Automated Regret Minimization

Multi-Armed Bandits dynamically exploit winning prompt templates while maintaining enough exploration to discover breakthroughs, maximizing revenue during experimentation.

How it works

Diagram: a fixed A/B split compared with a multi-armed bandit

Diagram: how a shadow request is mirrored, executed and discarded

Let us inspect the complete Feature Flag Routing and Shadow Execution Workflow:

[User Request (user_id="usr_8829", prompt="Analyze financial report")]
                               β”‚
                               β–Ό
        [Feature Flag Router & Hashing Engine]
        - Computes deterministic hash: `hash("usr_8829" + "salt_v2") % 100 = 14`
        - Evaluates Flag Rules:
          * Canary % = 20% -> Bucket 14 < 20 -> Assigned Variant B (Candidate)
          * Shadow Mode = ENABLED -> Triggers async shadow call to Variant C
                               β”‚
        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
        β”‚                                             β”‚
        β–Ό (Live User Path)                            β–Ό (Async Shadow Path)
[Candidate Model v2.0]                       [Experimental Model v3.0]
  - Generates analysis response                - Executes inference in background
  - P95 Latency: 140ms                         - Logs tokens & response to DB
  - Emits response to User in 150ms            - Response safely discarded
                                                      β”‚
                                                      β–Ό
                                            [Evaluation Worker]
                                            - Compares v2.0 vs v3.0 quality

Multi-Armed Bandit vs Fixed A/B Testing comparing static 50/50 traffic allocation with fixed regret against dynamic Thompson Sampling bandit traffic shifts

An everyday analogy

Diagram: opening a show cold compared with rehearsals and preview performances

Think of feature flagging and shadow traffic like a Broadway theatre preview show:

Examples in practice

Let us build a complete Python AI Feature Flag & Shadow Routing Engine that performs deterministic user hashing, manages progressive percentage rollouts, and executes asynchronous shadow traffic:

import hashlib
import time
from typing import Dict, Any, List, Optional

class AIFeatureFlagRouter:
    def __init__(self, flag_name: str, canary_percentage: int = 10, shadow_enabled: bool = False):
        self.flag_name = flag_name
        self.canary_pct = max(0, min(100, int(canary_percentage)))
        self.shadow_enabled = bool(shadow_enabled)
        self.shadow_logs: List[Dict[str, Any]] = []

    def get_user_bucket(self, user_id: str) -> int:
        # Deterministic MD5 hash to bucket 0-99
        hash_input = f"{self.flag_name}:{user_id}".encode("utf-8")
        hash_hex = hashlib.md5(hash_input).hexdigest()
        return int(hash_hex[:8], 16) % 100

    def route_request(self, user_id: str, prompt: str) -> Dict[str, Any]:
        bucket = self.get_user_bucket(user_id)
        
        # Determine Live Variant
        if bucket < self.canary_pct:
            live_variant = "CANDIDATE_MODEL_V2"
        else:
            live_variant = "BASELINE_MODEL_V1"

        # Execute Live Path
        live_response = f"[{live_variant}] Response for '{prompt}'"
        
        # Execute Shadow Path if enabled
        shadow_executed = False
        if self.shadow_enabled:
            shadow_variant = "SHADOW_EXPERIMENTAL_V3"
            shadow_response = f"[{shadow_variant}] Response for '{prompt}'"
            self.shadow_logs.append({
                "user_id": user_id,
                "prompt": prompt,
                "shadow_variant": shadow_variant,
                "shadow_response": shadow_response,
                "timestamp": time.time()
            })
            shadow_executed = True

        return {
            "user_id": user_id,
            "bucket": bucket,
            "assigned_variant": live_variant,
            "response": live_response,
            "shadow_executed": shadow_executed
        }

if __name__ == "__main__":
    router = AIFeatureFlagRouter(flag_name="llm_router_v2", canary_percentage=20, shadow_enabled=True)
    
    # Test deterministic routing across users
    for uid in ["usr_101", "usr_102", "usr_103", "usr_104", "usr_105"]:
        res = router.route_request(user_id=uid, prompt="Hello AI")
        print(f"User: {uid} | Bucket: {res['bucket']:02d} | Variant: {res['assigned_variant']}")
    
    print(f"Shadow executions logged: {len(router.shadow_logs)}")

Production Architecture Case Study: Dynamic Flag Configuration with OpenFeature

Consider an enterprise API Gateway using the CNCF OpenFeature standard for dynamic AI routing rules.

# feature-flags.yaml: Dynamic AI Model Routing Configuration
flags:
  genai-code-completion-v2:
    state: ENABLED
    defaultVariant: baseline
    variants:
      baseline: "meta-llama/Llama-3.1-8B-Instruct"
      canary: "Qwen/Qwen2.5-Coder-32B-Instruct"
      shadow: "deepseek-ai/DeepSeek-Coder-V2"
    rules:
      - action:
          variant: canary
        targeting:
          - attribute: organization_tier
            operator: EQUALS
            value: "internal_beta_testers"
      - action:
          variant: canary
        percentage: 15  # 15% Progressive rollout
    shadow:
      enabled: true
      sampleRate: 0.10  # 10% Shadow Mirroring

Detailed Failure Mode Taxonomy: AI Rollout Hazards

Diagram: the four failure modes of an AI rollout and the control for each

+-------------------------------------------------------------------------------+
|                           AI ROLLOUT FAILURE TAXONOMY                         |
+-------------------+-----------------------+-----------------------------------+
| Failure Mode      | Root Cause Analysis   | Architectural Mitigation          |
+-------------------+-----------------------+-----------------------------------+
| User Variant      | Non-deterministic     | Use `hash(user_id + salt) % 100`  |
| Flapping          | random number routing | to ensure persistent assignment   |
| Shadow Traffic    | Synchronous wait on   | Execute shadow calls in detached  |
| Blocks User Lat   | shadow inference call | asyncio task / background worker  |
| Shadow Model      | Candidate server runs | Scale shadow GPU node pool with   |
| Overload OOM      | out of VRAM capacity  | strict independent rate limiters  |
| Uncontrolled      | Manual rollout with   | Configure automated Prometheus    |
| Canary Regression | delayed human triage  | error rate rollback circuit breaks|
+-------------------+-----------------------+-----------------------------------+

Deep Architectural Breakdown: Thompson Sampling Multi-Armed Bandits

When optimizing between competing prompt templates:

1. The Beta-Bernoulli Bandit Model

Instead of guessing which prompt achieves higher user satisfaction (thumbs-up):

Diagram: the Thompson sampling loop for routing between prompt variants

+-------------------------------------------------------------------------------+
|                    THOMPSON SAMPLING CONVERGENCE TIMELINE                     |
+-------------------+-------------------+-------------------+-------------------+
| Timeline Milestone| Variant A (Good)  | Variant B (Bad)   | Routing Split     |
+-------------------+-------------------+-------------------+-------------------+
| Day 1 (0 items)   | Beta(1, 1)        | Beta(1, 1)        | 50% A / 50% B     |
| Day 3 (500 items) | Beta(320, 80)     | Beta(180, 220)    | 82% A / 18% B     |
| Day 7 (2000 items)| Beta(1450, 250)   | Beta(220, 280)    | 98% A / 2% B      |
+-------------------+-------------------+-------------------+-------------------+

Statistical Significance in LLM A/B Testing: Sample Size and Power

Evaluating prompt improvements requires rigorous statistical hypothesis testing:

+-------------------------------------------------------------------------------+
|                    A/B TESTING SAMPLE SIZE REQUIREMENTS                       |
+-------------------+-------------------+-------------------+-------------------+
| Baseline Rate     | Target Effect     | Confidence Level  | Required Samples  |
+-------------------+-------------------+-------------------+-------------------+
| 60% Positive      | +2% Lift (62%)    | 95% (p < 0.05)    | 9,600 users / arm |
| 60% Positive      | +5% Lift (65%)    | 95% (p < 0.05)    | 1,550 users / arm |
| 60% Positive      | +5% Lift (65%)    | 99% (p < 0.01)    | 2,450 users / arm |
+-------------------+-------------------+-------------------+-------------------+

Multi-Variant Bandit Exploitation: Epsilon-Greedy vs Thompson Sampling

Dynamic Guardrail Policy Switching via Feature Flags

In addition to model routing, feature flags dynamically toggle AI guardrails at runtime:

Diagram: feature flags used for releases compared with flags used as runtime guardrails

Implications: security, privacy, performance, scalability, and cost

Diagram: the safety, cost, performance and statistical implications of progressive AI rollouts

1. Zero Shadow Data Leakage

Ensure that shadow inference responses are strictly discarded in memory and never accidentally returned in user HTTP response bodies.

2. Shadow Compute Cost Budgeting

Mirroring 100% of live traffic to a shadow cluster doubles GPU compute costs. Use Shadow Sampling (e.g. 5% or 10% shadow mirror rate) to validate stability at minimal cost.

Alternatives: free, open source, and commercial

Tool / FrameworkTypeBest Suited ForKey Capabilities
OpenFeatureOpen Source (CNCF)Vendor-Neutral FlagsStandardized SDKs, multi-provider backends
UnleashOpen SourceSelf-Hosted Feature FlagsStrategy constraints, progressive rollouts
LaunchDarklyCommercial SaaSEnterprise Progressive DeliveryReal-time flag sync, streaming updates
GrowthBookOpen SourceFeature Flags + A/B TestingBayesian statistics, visual experiments
+-----------------------+-----------------------+-----------------------+
| Dimension             | Fixed A/B Testing     | Canary Deployment     |
+-----------------------+-----------------------+-----------------------+
| Primary Goal          | Statistical Proof     | Safe Release & Uptime |
| Traffic Split         | Fixed (e.g. 50/50)    | Dynamic (5% βž” 100%)   |
| Duration              | Weeks (Fixed window)  | Hours (Until verified)|
| Rollback Mechanism    | Manual analysis       | Automated on Error    |
+-----------------------+-----------------------+-----------------------+

When to use it β€” and when not to

Diagram: when progressive rollout machinery is warranted

Deploy Feature Flags and Progressive Rollouts for:

Do NOT require complex feature flag routing for:

Production Checklist: AI Feature Flagging Hardening

The AI connection

Knowledge check

  1. What is Shadow Traffic Mirroring? Duplicating live user queries to candidate models asynchronously in the background to validate performance without returning responses to users.
  2. Why is deterministic user hashing essential for feature flags? To guarantee consistent model variant assignment for a given user across multiple sessions.
  3. How do Multi-Armed Bandits reduce experimentation regret? By dynamically shifting traffic toward winning variants as positive feedback accumulates.

Hands-on exercise

In this lab, you will build an AI Feature Flag & Shadow Routing Engine in Python. Your implementation will:

  1. Implement deterministic user bucket hashing (hash(flag + user_id) % 100).
  2. Route users to baseline or canary model variants based on configured percentage thresholds.
  3. Execute asynchronous shadow traffic logging when shadow mode is enabled.
  4. Verify user assignment consistency and shadow audit log generation.

Expected output

============================= test session starts ==============================
collected 5 items

tests/test_feature_flags.py .....                                        [100%]

============================== 5 passed in 0.02s ===============================
[FEATURE FLAG] Initialized router (Flag: llm_v2, Canary: 20%, Shadow: ENABLED).
[HASH BUCKET] Deterministically assigned usr_101 to bucket 14 -> CANDIDATE_MODEL_V2.
[HASH BUCKET] Deterministically assigned usr_102 to bucket 85 -> BASELINE_MODEL_V1.
[SHADOW EXEC] Successfully mirrored 5 requests to experimental shadow model.

Validate your work

Run the pytest test suite:

pytest tests/run_tests.sh

Troubleshooting

  1. Hashing consistency: Ensure string encoding f"{flag}:{user_id}".encode("utf-8").
  2. Bucket calculation: Use MD5 prefix modulo 100: int(hex[:8], 16) % 100.

Common mistakes

Practice assignment

Add an Allowlist Override Rule that forces users with specific email domains (e.g. @corp.internal) directly into the candidate variant.

Extension challenge

Implement a Thompson Sampling Multi-Armed Bandit updating Beta distributions based on user feedback.

Quiz

Q1. What is 'Shadow Traffic Mirroring' (Dark Launching) in AI system deployments?

  1. Duplicating live traffic to a candidate, without replying
  2. Running models only at nighttime
  3. Deleting user traffic before it hits the model
  4. Hiding the server IP address
Show answer

Answer: A. Duplicating live traffic to a candidate, without replying

Shadow traffic mirrors live requests to test new models safely against real-world distributions with zero user impact.

Q2. Why is deterministic user hashing (e.g. `hash(user_id + salt) % 100`) used for AI feature flags instead of random coin flips?

  1. Because random numbers do not exist in Python
  2. A user keeps the same variant across sessions
  3. To make the model run 10x faster
  4. To encrypt the user's password
Show answer

Answer: B. A user keeps the same variant across sessions

Deterministic hashing ensures consistent user experiences across multiple sessions and requests.

Q3. How does a Multi-Armed Bandit (e.g. Thompson Sampling or Epsilon-Greedy) differ from traditional A/B testing?

  1. Bandits are illegal gambling machines
  2. A/B testing only works on mobile phones
  3. A bandit shifts traffic as evidence accumulates
  4. Bandits delete underperforming models permanently
Show answer

Answer: C. A bandit shifts traffic as evidence accumulates

Multi-armed bandits dynamically allocate traffic to winning variants, reducing cumulative regret during experimentation.

Q4. What is a 'Canary Blast Radius' in production AI releases?

  1. The physical damage caused by an exploding GPU
  2. The number of characters in the system prompt
  3. The volume of cooling fans in the server room
  4. The share of traffic exposed to the new version
Show answer

Answer: D. The share of traffic exposed to the new version

Blast radius bounding ensures that any unforeseen model errors impact only a tiny fraction of users.

Q5. When should an automated canary rollback trigger during an A/B model rollout?

  1. As soon as an automated health check is violated
  2. Whenever a user asks a question in Spanish
  3. At the end of every calendar month
  4. Never; rollbacks should always be manual
Show answer

Answer: A. As soon as an automated health check is violated

Automated rollbacks trigger immediately when candidate metrics violate predefined latency or error thresholds.

Glossary

Shadow Traffic Mirroring
Duplicating live production requests to candidate models asynchronously with zero user impact.
Feature Flag
A software switch enabling dynamic toggling of code paths or model variants at runtime without redeploying.
Canary Deployment
Gradually routing a small percentage of production traffic to a candidate revision while monitoring health.
Multi-Armed Bandit
An online learning algorithm that dynamically routes traffic to higher-performing variants to maximize reward.
Deterministic Hashing
Hashing user identifiers to integer buckets ensuring consistent variant assignment across sessions.
Blast Radius
The maximum proportion of users or systems impacted if a newly deployed version experiences an outage.
Online A/B Testing
Evaluating two competing model variants simultaneously across randomized user cohorts to measure KPIs.
Regret Minimization
Optimizing traffic routing during experimentation to minimize exposure to suboptimal variants.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.