Deployment, MLOps, and Security β€Ί Operating AI in Production β€Ί Day 337

Day 337: Deploying to a Cloud Service

Day 337 of 365 β€” Deploying to a Cloud Service

Master cloud AI infrastructure provisioning: deploy containerized AI workloads to serverless and container orchestration services (Cloud Run with GPU, AWS ECS Fargate, GCP GKE) using declarative Terraform and Python.

Course
Deployment, MLOps, and Security
Category
Operating AI in Production
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deployment-mlops-and-security/day-337-deploying-to-a-cloud-service

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deployment-mlops-and-security/day-337-deploying-to-a-cloud-service
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Moving an AI model from a local laptop or training environment into a production cloud infrastructure is one of the most critical transitions in modern software engineering. Manual click-ops deployments in cloud web consoles lead to catastrophic operational vulnerabilities:

Mastering Cloud AI Deployment with Infrastructure as Code (Terraform), Cloud Run / ECS GPU hosting, and Blue-Green zero-downtime cutover establishes a reproducible, rock-solid foundation for enterprise AI applications.

Declarative IaC Cloud Deployment Pipeline showing Terraform Plan & Apply, Secure Container Registry, IAM Role Attachment, and Cloud Run / Kubernetes Pod Provisioning with Zero Downtime Cutover

The idea in plain language

Diagram: the stages of a declarative cloud deployment pipeline

Diagram: manual console configuration compared with declarative infrastructure as code

Think of cloud AI deployment like launching a high-end luxury restaurant chain:

Historical background

Diagram: the progression from manual VM provisioning to serverless GPU

Cloud deployment architectures for machine learning evolved across three distinct eras:

1. Manual Virtual Machine Provisioning (2016–2019)

Engineers manually launched AWS EC2 p2.xlarge instances, installed CUDA drivers via shell scripts, and ran Flask apps in tmux sessions. Any instance termination required hours of manual reconstruction.

2. Containerized Cloud Platforms (2020–2023)

Docker containers standardized runtime environments. Platforms like AWS ECS and Google Cloud Run emerged, though GPU support remained restricted to complex dedicated Kubernetes clusters.

3. Serverless GPU & Modern Declarative Cloud (2024–Present)

Google Cloud Run launched native Nvidia L4 GPU serverless hosting with per-second billing. Terraform and Pulumi became industry standards for codifying complete GPU AI architectures with automated canary verification.

What it is β€” and what it is not

Let us define the core mechanisms of production cloud deployments:

What Cloud AI Deployment Is:

What Cloud AI Deployment Is Not:

Diagram: what cloud AI deployment is compared with two practices it excludes

+-------------------------------------------------------------------------------+
|                    CLOUD HOSTING PLATFORM COMPARISON MATRIX                   |
+-------------------+-----------------------+-------------------+---------------+
| Platform          | Execution Model       | Pricing Model     | Best For      |
+-------------------+-----------------------+-------------------+---------------+
| Google Cloud Run  | Serverless Container  | Per-millisecond   | Spiky traffic,|
| (with GPU)        | with Scale-to-Zero    | active compute    | sub-5k req/day|
| AWS ECS Fargate / | Managed Container     | Hourly instance   | Predictable   |
| EC2 GPU Tasks     | Orchestration         | + commitment disc | steady traffic|
| Kubernetes (GKE / | Full Distributed      | Hourly node pool  | High-concurr, |
| EKS Multi-Node)   | Cluster Control       | + cluster mgmt fee| multi-GPU TP  |
| Modal / RunPod    | Specialized Serverless| Per-second GPU    | Fast cold-start|
| Serverless        | Cloud                 | execution billing | custom kernels|
+-------------------+-----------------------+-------------------+---------------+

Why it was created and what problems it solves

Cloud deployment patterns resolve the three primary failure modes of enterprise AI operations:

1. Eliminating Environment Drift via Declarative Terraform

IaC guarantees that Staging and Production environments use identical CUDA driver versions, memory limits, and networking rules, eliminating β€œworks on staging but crashes in prod” incidents.

2. Zero-Downtime Model Version Upgrades

Blue-Green and Canary traffic routing ensure that when updating an LLM weights checkpoint or server version, active streaming responses finish cleanly without truncation.

3. Slashing Cloud Spend via Commitment Modeling

Balancing spot GPU autoscaling for peak bursts with 1-year savings plan commitments for steady-state traffic cuts monthly cloud bills by 40% to 60%.

How it works

Diagram: blue-green cutover compared with a canary traffic ramp

Diagram: how declarative infrastructure is planned, locked, applied and verified

Let us inspect the complete Declarative Cloud Deployment Workflow:

[Developer pushes git commit to main branch]
                    β”‚
                    β–Ό
[GitHub Actions CI/CD Pipeline]
  1. Executes pytest suite & ruff linting
  2. Builds multi-stage GPU Docker image
  3. Pushes image to Artifact Registry: `us-docker.pkg.dev/ai/vllm:v2.4.0`
                    β”‚
                    β–Ό
[Terraform Apply Automation]
  - Updates Service Definition with new image tag
  - Provisions Green Revision (Target: 2x Nvidia L4 GPUs)
                    β”‚
                    β–Ό
[Automated Green Revision Health Verification]
  - Polls `/health` Readiness Probe until model weights load (45s)
  - Dispatches Canary Test Request: "Ping" βž” "Pong" (200 OK)
                    β”‚
                    β–Ό
    Did the Green revision pass health checks?
                    β”œβ”€β”€ YES βž” Cloud Load Balancer shifts 100% traffic to Green
                    β”‚        Blue revision gracefully drains active streams (30s)
                    β”‚
                    └── NO  βž” Alerts SRE; aborts deployment; 100% traffic stays on Blue

Blue-Green vs Canary Deployment Traffic Split comparing instantaneous 100% Blue-to-Green cutover against progressive 5% to 25% to 100% Canary traffic ramp-up with automated rollback on error spikes

An everyday analogy

Diagram: cutting over airport signage mid-service compared with installing in parallel

Think of cloud deployments like upgrading the electronic signage and gate systems at a busy international airport:

Examples in practice

Let us build a complete Python Cloud Deployment Orchestration Simulator that manages declarative infrastructure states, container registry digests, IAM permission verification, and Blue-Green zero-downtime cutover:

import time
from typing import Dict, Any, List, Optional

class CloudDeploymentOrchestrator:
    def __init__(self):
        self.active_environment = "BLUE"  # BLUE or GREEN
        self.revisions = {
            "BLUE": {"image_tag": "v1.0.0", "healthy": True, "active_traffic_pct": 100},
            "GREEN": {"image_tag": "v1.0.0", "healthy": False, "active_traffic_pct": 0}
        }
        self.deployment_history: List[Dict[str, Any]] = []

    def deploy_new_revision(self, new_image_tag: str, simulate_health_pass: bool = True) -> Dict[str, Any]:
        target_env = "GREEN" if self.active_environment == "BLUE" else "BLUE"
        current_env = self.active_environment
        
        # 1. Provision target environment with new image
        self.revisions[target_env]["image_tag"] = new_image_tag
        self.revisions[target_env]["healthy"] = False
        
        # 2. Execute health check probe
        if simulate_health_pass:
            self.revisions[target_env]["healthy"] = True
            # Zero-downtime cutover
            self.revisions[target_env]["active_traffic_pct"] = 100
            self.revisions[current_env]["active_traffic_pct"] = 0
            self.active_environment = target_env
            
            event = {
                "status": "DEPLOYMENT_SUCCESS",
                "active_env": target_env,
                "deployed_image": new_image_tag,
                "timestamp": time.time()
            }
            self.deployment_history.append(event)
            return event
        else:
            # Health check failed; abort cutover
            self.revisions[target_env]["healthy"] = False
            event = {
                "status": "DEPLOYMENT_FAILED_HEALTH_CHECK",
                "active_env": current_env,
                "rolled_back_to": current_env,
                "timestamp": time.time()
            }
            self.deployment_history.append(event)
            return event

if __name__ == "__main__":
    orchestrator = CloudDeploymentOrchestrator()
    print("Initial State:", orchestrator.active_environment, orchestrator.revisions)
    
    # Deploy v2.0.0 successfully
    res1 = orchestrator.deploy_new_revision("v2.0.0", simulate_health_pass=True)
    print("Deploy v2:", res1["status"], "Active:", orchestrator.active_environment)
    
    # Attempt bad deployment v2.1.0
    res2 = orchestrator.deploy_new_revision("v2.1.0", simulate_health_pass=False)
    print("Deploy v2.1 (Bad):", res2["status"], "Active:", orchestrator.active_environment)

Production Architecture Case Study: Terraform Cloud Run GPU Deployment

Consider an enterprise SaaS company deploying vLLM on Google Cloud Run with Nvidia L4 GPUs via Terraform.

# main.tf: Cloud Run Service with Nvidia L4 GPU
resource "google_cloud_run_v2_service" "vllm_serving" {
  name     = "vllm-llama3-serving"
  location = "us-central1"
  ingress  = "INGRESS_TRAFFIC_ALL"

  template {
    scaling {
      min_instance_count = 0  # Scale-to-zero when idle
      max_instance_count = 8  # Autoscale to 8 GPUs under load
    }

    containers {
      image = "us-docker.pkg.dev/enterprise-ai/serving/vllm:v2.4.0"
      
      resources {
        limits = {
          cpu    = "8"
          memory = "32Gi"
          "nvidia.com/gpu" = "1"
        }
      }

      env {
        name  = "MODEL_NAME"
        value = "meta-llama/Meta-Llama-3.1-8B-Instruct"
      }
      
      startup_probe {
        http_get {
          path = "/health"
          port = 8000
        }
        initial_delay_seconds = 15
        period_seconds        = 5
        failure_threshold     = 30
      }
    }
    
    node_selector = {
      "cloud.google.com/gke-accelerator" = "nvidia-l4"
    }
  }

  traffic {
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
    percent = 100
  }
}

Detailed Failure Mode Taxonomy: Cloud Deployment Hazards

Diagram: the four failure modes of a cloud AI deployment and their controls

+-------------------------------------------------------------------------------+
|                        CLOUD DEPLOYMENT FAILURE TAXONOMY                      |
+-------------------+-----------------------+-----------------------------------+
| Failure Mode      | Root Cause Analysis   | Architectural Mitigation          |
+-------------------+-----------------------+-----------------------------------+
| Premature Traffic | Load balancer shifts  | Configure Startup Probes with     |
| Cutover OOM       | traffic before model  | generous initial delay (60-120s)  |
| IAM Permission    | Service account lacks | Grant specific IAM bucket reader  |
| AccessDenied      | S3/GCS bucket KMS key | roles to container service account|
| Quota Limit Exceed| Cloud region runs out | Configure multi-region failover   |
| on GPU Autoscale  | of available Nvidia L4| in Terraform deployment definitions|
| Secret Leak in    | API tokens baked into | Mount secrets via AWS Secrets     |
| Docker Image      | container layer       | Manager / GCP Secret Manager      |
+-------------------+-----------------------+-----------------------------------+

Deep Architectural Breakdown: Cloud AI Infrastructure Security and Cost Arbitrage

Deploying AI models in production cloud environments introduces complex security and cost challenges:

1. VPC Peering and Private Service Connect (PSC)

Publicly exposed AI inference endpoints invite Distributed Denial of Service (DDoS) and unauthorized prompt fuzzing attacks. Production cloud architectures enforce:

Diagram: a publicly exposed inference endpoint compared with a private one behind a gateway

2. GPU Cloud Cost Optimization Matrix

Balancing reserved compute, savings plans, and spot instances yields dramatic cost savings:

Diagram: matching baseline, peak and batch traffic to committed, on-demand and spot capacity

+-------------------------------------------------------------------------------+
|                    ENTERPRISE CLOUD GPU COST ARBITRAGE                        |
+-------------------+-----------------------+-------------------+---------------+
| Workload Type     | Infrastructure Tier   | Pricing Model     | Cost Savings  |
+-------------------+-----------------------+-------------------+---------------+
| Core Baseline 24/7| EKS Dedicated NodePool| 3-Year Savings Pl | 52% Discount  |
| Daytime Spikes    | Cloud Run / EKS Auto  | On-Demand Billing | Standard Rate |
| Nightly Batch ETL | Ray Data on Spot Nodes| Spot Preemptible  | 72% Discount  |
| Staging / Dev QA  | Serverless Scale-to-0 | Per-second Active | 90% Savings   |
+-------------------+-----------------------+-------------------+---------------+

Cloud Deployment Infrastructure Deep Dive: Multi-Region Ingress and Network Topology

In enterprise production deployments, deploying across multiple cloud regions requires orchestrating global load balancers with health-checked backends:

+-------------------------------------------------------------------------------+
|                    GLOBAL MULTI-REGION DEPLOYMENT TOPOLOGY                    |
+-------------------+-------------------+-------------------+-------------------+
| Region Zone       | Primary Role      | Accelerator Tier  | Target Latency    |
+-------------------+-------------------+-------------------+-------------------+
| us-east1 (N. VA)  | 50% North America | 8x Nvidia L4 (24GB)| < 45 ms TTFT      |
| us-central1 (Iowa)| 30% Backup / Burst| 8x Nvidia A100    | < 55 ms TTFT      |
| europe-west1 (Bel)| 20% EU Sovereign  | 4x Nvidia L4 (24GB)| < 40 ms TTFT      |
+-------------------+-------------------+-------------------+-------------------+

Terraform State Management & Locking Best Practices

When teams collaborate on cloud AI infrastructure, state locking prevents concurrent apply collisions:

Implications: security, privacy, performance, scalability, and cost

Diagram: the security, privacy, performance, resilience and cost implications of cloud AI deployment

1. Ephemeral Secret Injection

Never store OpenAI, Anthropic, or HuggingFace API tokens inside git repositories or environment variables in Dockerfiles. Inject secrets dynamically at runtime using cloud secret managers with automated rotation.

2. Multi-Region Disaster Recovery

Maintain identical Terraform definitions across at least two geographical cloud regions (e.g. us-east1 and us-central1). If an entire datacenter suffers power or fiber disruptions, DNS automatically routes traffic to the surviving region.

Alternatives: free, open source, and commercial

Tool / FrameworkTypeBest Suited ForKey Capabilities
TerraformOpen Source / CommDeclarative Multi-Cloud IaCMulti-provider state management, drift detection
PulumiOpen Source / CommProgrammatic IaC (Python/TS)Native programming language infrastructure
Google Cloud RunManaged CloudServerless GPU ContainersFast scale-to-zero, Nvidia L4 GPUs
AWS ECS FargateManaged CloudServerless Container ServingDeep AWS ecosystem integration
+-----------------------+-----------------------+-----------------------+
| Dimension             | Manual Click-Ops      | Terraform Declarative |
+-----------------------+-----------------------+-----------------------+
| Reproducibility       | 0% (Manual memory)    | 100% Version-Controlled|
| Drift Detection       | None (Silent errors)  | `terraform plan` check|
| Multi-Environment Sync| Fragile & slow        | Identical staging/prod|
| Disaster Recovery     | Days to reconstruct   | Minutes to re-apply   |
+-----------------------+-----------------------+-----------------------+

When to use it β€” and when not to

Diagram: when infrastructure as code is worth its overhead

Deploy Cloud AI Infrastructure for:

Do NOT require complex multi-region Terraform pipelines for:

Production Checklist: Cloud AI Deployment Hardening

The AI connection

Knowledge check

  1. Why is Infrastructure as Code (IaC) mandatory for production AI? Because it ensures declarative, reproducible, and version-controlled infrastructure definitions, eliminating configuration drift between environments.
  2. What is the difference between Blue-Green and Canary deployments? Blue-Green switches 100% of traffic instantly between two identical environments; Canary routes a small percentage (e.g. 5%) progressively while monitoring metrics.
  3. How does serverless GPU hosting reduce costs for spiky workloads? By scaling to zero when idle and billing per millisecond of active execution time.

Hands-on exercise

In this lab, you will build a Cloud Deployment Orchestrator Simulator in Python. Your implementation will:

  1. Manage Blue-Green deployment environments and active traffic routing percentages.
  2. Deploy new container image revisions to the staging environment.
  3. Execute automated health verification probes before switching traffic.
  4. Execute instantaneous zero-downtime cutover upon health check success and abort/rollback upon health check failure.

Expected output

============================= test session starts ==============================
collected 5 items

tests/test_cloud_deployment.py .....                                     [100%]

============================== 5 passed in 0.02s ===============================
[ORCHESTRATOR] Initialized Blue-Green environment (Active: BLUE, Traffic: 100%).
[DEPLOYMENT] Deployed v2.0.0 to GREEN; health probe passed.
[CUTOVER] Shifted 100% traffic to GREEN; Blue drained.
[ROLLBACK] Detected health check failure on v2.1.0; preserved 100% traffic on GREEN.

Validate your work

Run the pytest test suite:

pytest tests/run_tests.sh

Troubleshooting

  1. Active environment tracking: Ensure active_environment only flips when simulate_health_pass is True.
  2. Traffic percentage sum: Verify active traffic percentages always sum to 100%.

Common mistakes

Practice assignment

Add a Canary Rollout Strategy that shifts traffic in increments of 10%, 25%, 50%, and 100% with simulated error monitoring.

Extension challenge

Implement a Terraform State Parser that parses .tf JSON plans and validates that all GPU node pools include non-root security contexts.

Quiz

Q1. What is the primary architectural differentiator of Serverless GPU Hosting (e.g. Google Cloud Run with Nvidia L4 GPUs, Modal, RunPod Serverless) compared to Dedicated Kubernetes Clusters?

  1. Serverless GPUs do not support Python
  2. Dedicated clusters cannot run Docker containers at all
  3. Serverless GPUs have infinite memory
  4. Per-millisecond billing and scale-to-zero, but cold starts
Show answer

Answer: D. Per-millisecond billing and scale-to-zero, but cold starts

Serverless GPU platforms bill exclusively for active compute time with scale-to-zero, trading cold-start delays for massive cost savings.

Q2. Why is Infrastructure as Code (IaC) with Terraform essential for production AI deployments?

  1. Declarative, versioned infrastructure with no drift
  2. It formats Markdown files
  3. It makes models 10x faster
  4. It compiles Python code into C++
Show answer

Answer: A. Declarative, versioned infrastructure with no drift

Terraform ensures declarative, reproducible, and drift-free infrastructure provisioning across environments.

Q3. What is the security principle of 'Least Privilege IAM' for AI container deployments?

  1. Giving containers full root admin access to the entire cloud account
  2. Grant only the minimal permissions the container needs
  3. Disabling all passwords across the team
  4. Running all services anonymously without authentication
Show answer

Answer: B. Grant only the minimal permissions the container needs

Least privilege IAM ensures compromised containers cannot access unauthorized cloud resources.

Q4. What is a 'Blue-Green Deployment' strategy in cloud AI serving?

  1. Painting the server rack blue and green
  2. Deleting the database before deployment
  3. Two identical environments, switched once green is healthy
  4. Restarting the model every 5 minutes
Show answer

Answer: C. Two identical environments, switched once green is healthy

Blue-Green deployments maintain two identical environments, switching load balancer traffic instantaneously once verified.

Q5. How do Cloud Cost Savings Plans / 1-Year Commitments reduce enterprise GPU spending?

  1. By turning off the GPUs on weekends
  2. By converting all of the models to 1-bit integers
  3. By deleting model weights from memory
  4. Commit to a baseline spend for a large discount
Show answer

Answer: D. Commit to a baseline spend for a large discount

Commitment-based savings plans provide 30-55% cost reductions for steady-state baseline AI workloads.

Glossary

Infrastructure as Code (IaC)
Managing and provisioning computing infrastructure through version-controlled definition files (e.g. Terraform).
Serverless GPU
A cloud execution model where GPU instances are dynamically provisioned and billed per millisecond of active execution.
Blue-Green Deployment
A deployment strategy utilizing two identical environments to achieve instantaneous zero-downtime cutover.
Canary Deployment
Gradually rolling out a new model version to a small percentage of user traffic (e.g. 5%) before full rollout.
IAM Role / Service Account
An identity created in cloud provider IAM that has specific permissions to interact with cloud services.
Container Registry
A secure, managed repository for storing, versioning, and distributing Docker container images (ECR / Artifact Registry).
Cold-Start Penalty
The latency overhead incurred when a serverless container initializes hardware, loads dependencies, and mounts weights.
Spot / Preemptible Instances
Discounted spare cloud compute capacity that can be reclaimed by the cloud provider with short notice.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.