Deployment, MLOps, and Security › Containers and Cloud › Day 331
Day 331: Dockerizing an AI Application
Sort everything that could go into an AI image into build-only, runtime and data, then measure what each decision costs. Discover that the multi-stage build every tutorial teaches saves 1.17x while moving the model weights out saves 3.97x, and that a rollout to twenty nodes falls from four and a half hours to one.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deployment-mlops-and-security/day-331-dockerizing-an-ai-application
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
cd labs/sections/deployment-mlops-and-security/day-331-dockerizing-an-ai-application - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Compute an image size from its stages, and explain why only the final stage ships.
- Separate build-only components from runtime ones, which is what multi-stage removes.
- Separate runtime components from data, which is what mounting relocates.
- Convert image size into rollout time across a fleet, the unit that is actually felt.
- Choose between cuda-devel and cuda-runtime, and say why the vendor ships both.
Prerequisites
Why this matters
Yesterday you learned the rule that decides how long a build takes. Today the question is what ends up inside the image, and it is a different problem with a different answer.
Almost every guide to writing a good Dockerfile was written for web applications. Its advice is correct and it is aimed at the wrong target. In a web image the biggest thing is the framework, so a multi-stage build that drops the compiler is a large win. In an AI image the biggest thing is usually the model, and no amount of multi-stage discipline touches it.
Here is the measurement from today’s lab, on one service packaged three ways:
multi-stage 21128 -> 18053 MB (3075 MB, 1.17x)
mount the weights 18053 -> 4553 MB (13500 MB, 3.97x)
The famous optimisation buys 1.17×. The one nobody mentions buys 3.97×. Both are worth doing, and if you only do the famous one you have taken the smaller half and probably believe you are finished.
That matters because image size is not a disk-space problem. A rollout pulls the image onto every node, so 21 GB across twenty nodes is four and a half hours of pulling before anything serves traffic.
The idea in plain language
Sort everything that could be in your image into three buckets, and the decisions follow.
Build-only. Compilers, headers, cuda-devel, anything whose job is to produce something. The running process never calls them. These are removed by a multi-stage build: do the compiling in one stage, then start a fresh stage from a small base and copy just the artefacts across. The builder is discarded entirely.
Runtime. The interpreter, cuda-runtime, the framework, your code. The process genuinely needs these, so they ship. You can choose a slimmer base, but you cannot drop them.
Data. Model weights, embeddings, static assets. Needed at run time — but unlike runtime code, they do not have to live in the image. Mount a volume, or fetch from object storage on start.
That third bucket is the one people do not draw, because in a web application it is empty. Drawing it is the difference between a 18 GB image and a 4.5 GB one.
The reason it works is that weights change on a different schedule from code. Baking them in couples the two: ship a one-line fix and every node re-pulls thirteen gigabytes of unchanged floating point.
Historical background
Static linking and the fat binary
Long before containers, shipping software meant deciding what to bundle. A statically linked binary trades size for independence — the same trade an image makes, at a different granularity.
Multi-stage builds, Docker 17.05 (2017)
Before them the standard trick was the “builder pattern”: two Dockerfiles and a shell script that copied artefacts between them. FROM ... AS builder with COPY --from= collapsed that into one file, and slim runtime images became the default rather than an advanced technique.
Distroless, 2017 onward
Google’s distroless images contain an application’s runtime dependencies and nothing else — no shell, no package manager. The size win is real and the security win is larger: an attacker who achieves execution finds no tools. The cost is that debugging a running container becomes genuinely hard.
Model registries and weight formats
Hugging Face Hub, MLflow’s model registry and formats like safetensors emerged because the industry converged on the same conclusion this lesson reaches: model weights are data with a lifecycle of their own, and they want versioning, access control and distribution separate from the code that loads them.
The GPU base image problem
NVIDIA publishes cuda:*-devel and cuda:*-runtime variants precisely because the build/runtime split matters so much here — the devel image is roughly 4 GB against the runtime’s 1.6 GB. That the vendor ships two images is the clearest possible signal about which one belongs in production.
What it is — and what it is not
Dockerizing an AI application is deciding what belongs in the image, in which stage, and what belongs outside it entirely.
What it is
It is a sorting problem first and a Dockerfile second. Get the three buckets right and the file writes itself.
It is about deploy time, not disk. Nobody minds a large file on a laptop; everybody minds a slow rollout.
It is a licensing decision when weights are involved. Pushing an image containing a model you may not redistribute is redistribution.
What it is not
It is not “make the image as small as possible”. A distroless image you cannot debug at 3am has costs that do not show up in megabytes.
It is not a reason to avoid large frameworks. Torch is 2.4 GB because it does a great deal; the answer is to ship it once, not to avoid it.
It is not solved by compression. The numbers here are already compressed sizes.
Why it was created and what problems it solves
Build toolchains in production images. Hundreds of megabytes and a compiler for anyone who gets execution. Solution: multi-stage builds.
Rollouts that take hours. Solution: get the image small enough that pulling it is not the dominant cost.
Code changes that force a weights re-pull. Solution: separate their lifecycles — weights on a volume or in object storage.
Images that cannot be shared because of what is inside them. Solution: keep licensed weights out, so the image can be public and the model private.
GPU images that are enormous for no reason. Solution: cuda-runtime, not cuda-devel, in the final stage.
How it works
Declare a builder stage from a full base. It gets the compiler, the headers, cuda-devel — whatever is needed to produce wheels or compile an extension.
Declare the final stage from a slim base. python:3.12-slim plus cuda-runtime, not the devel package.
Copy artefacts, not tooling. COPY --from=builder /wheels /wheels then install from them. Copying the toolchain forward defeats the whole arrangement — which is why today’s lab checks the carried components as well as the stage’s own.
Leave the weights out. Mount a volume, or fetch on startup with a checksum. The image then contains the code that loads a model rather than the model.
Add a USER. An AI service typically has a writable cache directory, a mounted volume and a network listener. Root inside a container with a writable mount is the combination that turns an escape from theoretical into practical.
Measure in rollout seconds, not megabytes. Size × nodes is the number that a deploy actually spends.
An everyday analogy
Think about moving house.
Build-only is the scaffolding used to get the piano through the window. You do not load it onto the van; you take it down and leave it behind. That is the multi-stage build, and everyone understands it immediately.
Runtime is the furniture. It goes on the van because you need it in the new house.
Data is the contents of the loft — heavy, and going to the same address, and not necessarily on this van. It can be shipped separately, and it should be if you are going to move house again next month, because otherwise every move carries the loft.
The mistake the analogy exposes is the common one: people become very disciplined about the scaffolding, feel efficient, and load the loft onto the van without noticing that it outweighs the furniture four to one.
Examples in practice
The measured output from today’s lab, three packagings of one service:
--- 1. single stage, full base, weights baked in ---
image 21128 MB one pull 845.1s rollout to 20 nodes 16902.0s
[build-tooling-shipped] saves 4380 MB
[weights-baked-in] saves 13500 MB
[fat-base] saves 295 MB
--- 2. multi-stage, slim runtime, weights baked in ---
image 18053 MB one pull 722.1s rollout to 20 nodes 14442.0s
[weights-baked-in] saves 13500 MB
--- 3. multi-stage, slim runtime, weights mounted ---
image 4553 MB one pull 182.1s rollout to 20 nodes 3642.0s
no findings
Plan 2 is a correctly written Dockerfile by every standard piece of advice. Multi-stage, slim base, no build tooling. It is 18 GB, and a rollout to twenty nodes takes four hours.
The finding that remains is the one no web-application guide would have taught you to look for.
rollout to 20 nodes: 16902s -> 3642s
Four hours and 42 minutes to one hour and one minute. Same service, same framework, same model — the model is simply no longer travelling inside the container image.
Note also that plan 1 ships cuda-devel at 4.1 GB when the running process needs cuda-runtime at 1.6 GB. NVIDIA publishing those as two separate images is the vendor telling you which one production wants.
Implications: security, privacy, performance, scalability, and cost
Security. The size and security arguments coincide here, which is rare. Build tooling in a runtime image is a compiler available to anyone who achieves execution. A slim base has fewer packages that can have CVEs — most vulnerabilities reported against an application image are in components it never calls.
Supply chain. Mounting weights makes the mount an input. Whatever can write to that volume can replace your model. Verify a checksum on load; Day 347 treats this properly.
Privacy and licensing. An image containing licensed weights, pushed to a public registry, is publication. Keeping weights out means an image leak is not a model leak.
Performance. Image size does not affect inference speed at all. It affects how long a node waits before it can serve, which is felt during rollouts, autoscaling events and incident recovery — precisely the moments you are least willing to wait.
Cost. Registry egress is billed per gigabyte on most providers. A 21 GB image pulled by twenty nodes on every deploy is 420 GB of egress per rollout.
Alternatives: free, open source, and commercial
Multi-stage Dockerfiles — free, standard, and what this lesson assumes. The right default.
Distroless base images — Google’s, Apache-2.0. Smaller still and no shell, which is excellent for security and painful the first time you need to debug a running pod.
Chainguard Images — hardened, minimal, continuously rebuilt against current CVEs. Free tier for many images, commercial for the full catalogue and support.
docker buildx cache mounts and build secrets — free and built in. A cache mount keeps the pip cache across builds without shipping it; a build secret makes a token available during a RUN without it entering any layer. Both solve problems people otherwise solve badly.
Model registries — Hugging Face Hub, MLflow, or plain object storage with versioned keys. This is where weights belong, and any of them beats baking the model in.
Buildpacks — no Dockerfile; the platform decides. Fine for a web service, and a poor fit here precisely because it will make the weights decision for you.
The honest default: a two-stage Dockerfile on a slim base with cuda-runtime, a non-root USER, and weights fetched at startup from versioned object storage with a checksum.
Comparison with related concepts
Multi-stage build versus slim base. Different mechanisms for different buckets. Multi-stage removes build-only components entirely; a slim base reduces what the runtime bucket costs. Doing one does not do the other.
cuda-devel versus cuda-runtime. Compiler toolkit versus the libraries a compiled program links against. About 4.1 GB versus 1.6 GB. Confusing them is the single most expensive mistake available in a GPU Dockerfile.
Baked-in versus mounted weights. Baked-in is simpler and reproducible: the image is the whole system. Mounted is smaller, decouples the lifecycles, and adds a supply-chain input to verify. For anything above a few hundred megabytes, mounted wins.
Image size versus layer count. Squashing layers reduces count and rarely reduces size much. What reduces size is not putting the thing in.
When to use it — and when not to
Always separate build from runtime, even for a small service. It costs a few lines and it is the difference between shipping a compiler and not.
Mount the weights once the model is larger than roughly a gigabyte, or once it changes on a different schedule from the code. Both are usually true at once.
Bake them in when the model is small and reproducibility matters more than size — an embedding model of a few hundred megabytes in an image you want to be a single self-contained artefact is a defensible choice, and the lab’s model-weights-small entry exists to let you check the arithmetic.
Do not chase the last hundred megabytes. Once the weights are out, you are usually looking at a framework you cannot remove. Distroless will buy a little more and cost you debuggability.
Do not put a model in a public image without checking its licence. This is not an engineering decision and it is not reversible.
The AI connection
-
The famous optimisation is the smaller half here. Multi-stage buys 1.17x on an AI image; getting the weights out buys 3.97x, and no web guide mentions it.
-
Weights are data with their own lifecycle, which is why Hugging Face Hub, MLflow’s registry and safetensors all exist as separate things from the code.
-
A rollout is where image size is actually felt — during autoscaling and incident recovery, precisely the moments nobody is willing to wait.
-
Licensing is an engineering constraint here. Pushing an image containing weights you may not redistribute is redistribution, and it is not reversible.
-
A mounted volume is a supply-chain input. Whatever can write to it can replace your model, which is why Day 347 asks for a checksum on load.
Knowledge check
- Why does a multi-stage build save so much less on an AI image than on a web application image?
- What is the difference between
cuda-develandcuda-runtime, and which belongs in the final stage? - Why is image size a deployment problem rather than a disk-space problem?
- Mounting weights makes the image far smaller. What does it add that you must then handle?
- Why does copying a build artefact forward preserve the benefit of a multi-stage build, while copying the toolchain forward destroys it?
Hands-on exercise
You will size an image from its stages, work out what is making it big, and convert the answer into rollout time.
Open starter/image_plan.py in the lab directory and implement the stubbed functions, then run the tests. Docker is not required and no weights are downloaded.
Expected output
Running python3 examples/image_demo.py sizes three packagings of one service:
--- what each change bought ---
multi-stage 21128 -> 18053 MB (3075 MB, 1.17x)
mount the weights 18053 -> 4553 MB (13500 MB, 3.97x)
both 21128 -> 4553 MB (16575 MB, 4.64x)
rollout to 20 nodes: 16902s -> 3642s
The second line is four times the first. That is the finding.
Validate your work
Run bash tests/run_tests.sh. All twenty tests must pass.
Then check three things by inspection. Only the final stage may count toward the size — if the builder’s four gigabytes appear in the total, multi-stage will appear to buy nothing at all. An unknown component must contribute zero rather than raising, because naming a package the table does not know is a normal thing to do. And a final_stage that names no existing stage must raise KeyError rather than quietly using the first one.
Troubleshooting
Multi-stage appears to save nothing. ImagePlan.size_mb is summing every stage. Only the final one ships.
A clean plan produces a finding. You are treating cuda-runtime as build tooling. It is runtime; only cuda-devel is build-only.
build-tooling-shipped misses a case. Check the carried list as well as the final stage’s own components.
Zero bandwidth crashes. Return 0.0 rather than dividing.
Common mistakes
Summing all stages, which silently destroys the entire multi-stage argument.
Confusing the CUDA devel and runtime packages — a 2.5 GB error in either direction.
Reporting savings in megabytes only. Multiply by the fleet; that is what people wait for.
Treating weights as immovable because they are needed at run time. Needed at run time and needed in the image are different claims.
Practice assignment
Add layer-level granularity. Model each component as its own layer with an identity, and compute what a rebuild transfers when a registry already holds most of the layers.
Then write a test showing the case that matters: changing only app-code should transfer three megabytes, not the whole image. This is why the weights argument is even stronger in practice than the first-pull numbers suggest — a node that already has the weights layer does not re-pull it, but a node that has never seen the image does, and autoscaling creates new nodes precisely when you are busiest.
Extension challenge
Take a real AI service you have built and put it through the review.
List every component with an honest purpose: BUILD, RUNTIME or DATA. The exercise is the sorting, not the arithmetic — and the interesting output is not the total. It is the one component you had never considered optional.
For most people that is either a dataset copied in “for testing”, a set of dev dependencies that came along with the requirements file, or a model that could perfectly well be fetched at startup. Then compute what removing it would save across your actual node count, and decide whether you would have guessed the answer before measuring it.
Quiz
Q1. Why does a multi-stage build save proportionally much less on an AI image than on a web application image?
- Because multi-stage builds do not work correctly with GPU base images
- Because compressed layers behave differently when they contain binary data
- Because AI frameworks cannot be compiled in a separate stage
- The weights are runtime data, not build tooling
Show answer
Answer: D. The weights are runtime data, not build tooling
Most Docker guidance was written for applications where the biggest thing is the framework. In an AI image the biggest thing is usually the model.
Q2. What is the difference between cuda-devel and cuda-runtime, and which belongs in the final stage?
- devel builds at 4.1 GB; runtime links at 1.6 GB
- They are the same image under two names; either is fine
- cuda-runtime is for inference and cuda-devel is for training, so both ship
- cuda-devel is smaller because it excludes the drivers
Show answer
Answer: A. devel builds at 4.1 GB; runtime links at 1.6 GB
NVIDIA publishing them as separate images is the vendor signalling which one belongs in a runtime stage. Confusing them is a multi-gigabyte error.
Q3. Why is image size a deployment problem rather than a disk-space problem?
- Because container registries charge by image size regardless of pulls
- Size times node count is time before anything serves
- Because large images run more slowly once started
- Because disk is cheap only on cloud providers
Show answer
Answer: B. Size times node count is time before anything serves
In the lab a 21 GB image costs 16,902 seconds across twenty nodes. The same service at 4.5 GB costs 3,642.
Q4. Mounting weights instead of baking them in makes the image far smaller. What does it add that you must then handle?
- Nothing; it is strictly better in every respect
- A requirement to run the container as root
- A supply-chain input — verify a checksum on load
- A dependency on a specific container runtime
Show answer
Answer: C. A supply-chain input — verify a checksum on load
Decoupling the lifecycles is the win. The cost is that the model now arrives from somewhere the image does not control.
Q5. Why does copying a build artefact forward preserve the benefit of a multi-stage build while copying the toolchain forward destroys it?
- Because artefacts compress better than binaries
- Because the toolchain has to be reinstalled anyway
- Because COPY --from cannot copy directories
- The builder stage is meant to be discarded
Show answer
Answer: D. The builder stage is meant to be discarded
This is why the lab's review checks the carried components as well as the final stage's own -- carrying the toolchain forward is a silent way to undo the whole arrangement.
Glossary
- Multi-Stage Build
- A Dockerfile with several FROM instructions where a later stage copies artefacts out of an earlier one. Only the final stage ships; the builder is discarded entirely.
- Builder Stage
- An early stage carrying compilers and headers, used to produce artefacts. Its size does not affect the shipped image, which is why it can use a full base.
- Build-Only Component
- Something needed to produce artefacts and never at run time — build-essential, cuda-devel. In a final image it is pure waste and an available toolchain.
- Runtime Component
- Something the running process genuinely needs — the interpreter, cuda-runtime, the framework, your code. It can be made slimmer but not dropped.
- Data Component
- Model weights and assets. Needed at run time but not needed IN the image, which is the distinction that produces the larger saving.
- cuda-devel versus cuda-runtime
- The CUDA compiler toolkit against the libraries a compiled program links against — roughly 4.1 GB against 1.6 GB. Only the runtime belongs in a final stage.
- Slim Base
- A base image carrying the interpreter without the build toolchain. Smaller, and fewer packages that can carry a vulnerability.
- Distroless
- An image containing an application's runtime dependencies and nothing else — no shell, no package manager. Excellent for security, harder to debug.
- Rollout Time
- Image size multiplied by the number of nodes that must pull it. The unit in which image size is actually experienced.
- Baked-In Weights
- Model weights copied into the image. Simple and reproducible, and it couples the model's lifecycle to the code's, so a one-line fix re-pulls gigabytes.
- Mounted Weights
- Weights supplied at run time from a volume or object storage. Decouples the lifecycles, shrinks the image, and makes the mount a supply-chain input to verify.
Sources and further reading
- Multi-Stage Builds — Docker Documentation (accessed 2026-08-31)
- NVIDIA CUDA Container Images — base, runtime and devel tags — NVIDIA (accessed 2026-08-31)
- Distroless Container Images — Google Container Tools (accessed 2026-08-31)
- Build Secrets — Docker Documentation (accessed 2026-08-31)
- Safetensors — A Safe Format for Storing Model Weights — Hugging Face (accessed 2026-08-31)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.