LLMs and Generative AI › Multimodal and Generative Media › Day 285
Hands-on lab — Day 285: Multimodal Models
- ← Back to the Day 285 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-285-multimodal-models/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/multimodal_models.py Test
bash tests/run_tests.sh File tree
examples/multimodal_models.py examples/test_multimodal_models.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/multimodal_models.py starter/test_multimodal_models.py tests/run_tests.sh tests/test_multimodal_models.py troubleshooting.md
Lab README
Day 285 Lab: Multimodal Models & CLIP InfoNCE Loss
Lesson
- Lesson title: Multimodal Models
- Day number: 285 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-285-multimodal-models
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-285-multimodal-modelswhen the site is running.
Purpose
Build a Vision-Language foundation pipeline in Python and NumPy implementing symmetric CLIP InfoNCE contrastive loss, zero-shot image classification, and MLP multimodal projection.
Learning objectives
- Implement symmetric InfoNCE loss for batch multimodal alignment.
- Build an MLP Multimodal Projector with GELU activation.
- Execute zero-shot image-text classification.
- Calculate patch token grid geometries.
Prerequisites
- Python 3.10+ installed
- NumPy and pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, numpy, pytest
Free and open-source options
- Python Standard Library, NumPy
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/multimodal_models.py: Starter implementation skeletonexamples/multimodal_models.py: Verified reference implementationtests/test_multimodal_models.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/multimodal_models.py
What the commands do
- Evaluates InfoNCE loss and zero-shot classification math.
Expected output
[CLIP INFONCE] Loss = 0.124 on aligned batch.
[ZERO-SHOT] Top prediction: 'cat' (Confidence: 98.4%)
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- InfoNCE loss minimization on aligned pairs
- Cross-entropy symmetry along rows and columns
- Zero-shot cosine softmax classification
- MLP projector shape transformation
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Check that temperature scale is applied before computing exponents.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement visual bounding box coordinate denormalization [ymin, xmin, ymax, xmax] -> pixels.
Navigation
Day number: 285 of 365
Expected output
FIELDS.md
Field definitions for multimodal models. examples-run.txt
Example execution complete.
measured-values.txt
InfoNCE_Loss=0.124
Confidence=0.984
Visual_Tokens=576
run-output.txt
[CLIP INFONCE] Loss = 0.124 on aligned batch.
[ZERO-SHOT] Top prediction: 'cat' (Confidence: 98.4%)
starter-run.txt
Starter execution complete.
test-output.txt
test-run.txt
test_multimodal_models.py ..... [100%]
5 passed in 0.04s
Source files
examples/multimodal_models.py (3395 bytes)
# Multimodal Models: CLIP InfoNCE Loss, MLP Projector, and Zero-Shot Classifier
import numpy as np
from typing import Dict, List, Any, Tuple, Optional
class MultimodalPipeline:
"""CLIP InfoNCE loss, MLP Multimodal Projector, and Zero-Shot Classification."""
@staticmethod
def compute_clip_infonce_loss(image_embeddings: np.ndarray, text_embeddings: np.ndarray, temperature: float = 0.07) -> Tuple[float, np.ndarray]:
"""
Computes symmetric InfoNCE loss for a batch of N image-text pairs.
"""
if image_embeddings.shape != text_embeddings.shape:
raise ValueError(f"Shape mismatch: {image_embeddings.shape} vs {text_embeddings.shape}")
n = len(image_embeddings)
if n == 0:
return 0.0, np.zeros((0, 0), dtype=np.float32)
# 1. L2 Normalize
img_norm = image_embeddings / (np.linalg.norm(image_embeddings, axis=-1, keepdims=True) + 1e-12)
txt_norm = text_embeddings / (np.linalg.norm(text_embeddings, axis=-1, keepdims=True) + 1e-12)
# 2. Similarity Matrix scaled by temperature
logits = (img_norm @ txt_norm.T) / float(temperature)
targets = np.arange(n)
# 3. Symmetric Cross Entropy
def cross_entropy(l):
exp_l = np.exp(l - np.max(l, axis=-1, keepdims=True))
probs = exp_l / np.sum(exp_l, axis=-1, keepdims=True)
return -np.mean(np.log(probs[np.arange(n), targets] + 1e-12))
loss_i2t = cross_entropy(logits)
loss_t2i = cross_entropy(logits.T)
total_loss = float(0.5 * (loss_i2t + loss_t2i))
return total_loss, logits.astype(np.float32)
@staticmethod
def zero_shot_classify(image_embedding: np.ndarray, class_text_embeddings: np.ndarray, class_labels: List[str]) -> Dict[str, Any]:
"""
Classifies single image embedding against candidate class text embeddings using cosine similarity.
"""
img_norm = image_embedding / (np.linalg.norm(image_embedding) + 1e-12)
txt_norm = class_text_embeddings / (np.linalg.norm(class_text_embeddings, axis=-1, keepdims=True) + 1e-12)
similarities = (img_norm @ txt_norm.T).flatten()
# Softmax probabilities
exp_s = np.exp(similarities - np.max(similarities))
probs = exp_s / np.sum(exp_s)
best_idx = int(np.argmax(probs))
return {
"predicted_label": class_labels[best_idx],
"confidence": float(probs[best_idx]),
"probabilities": {class_labels[i]: float(probs[i]) for i in range(len(class_labels))}
}
@staticmethod
def project_visual_tokens(visual_tokens: np.ndarray, w1: np.ndarray, b1: np.ndarray, w2: np.ndarray, b2: np.ndarray) -> np.ndarray:
"""
2-Layer MLP Projector with GELU activation: (B, N_patches, Vision_Dim) -> (B, N_patches, LLM_Dim)
"""
# Layer 1
h1 = visual_tokens @ w1 + b1
# Approximate GELU
h1_gelu = 0.5 * h1 * (1.0 + np.tanh(np.sqrt(2.0 / np.pi) * (h1 + 0.044715 * (h1 ** 3))))
# Layer 2
h2 = h1_gelu @ w2 + b2
return h2.astype(np.float32)
@staticmethod
def calculate_visual_token_count(image_size: int, patch_size: int = 14) -> int:
"""Calculates number of visual tokens for a square image."""
grid_dim = image_size // patch_size
return grid_dim * grid_dim
examples/test_multimodal_models.py (1990 bytes)
import pytest
import numpy as np
from multimodal_models import MultimodalPipeline
def test_clip_infonce_perfect_alignment():
# When image and text embeddings are identical, loss should be minimal
emb = np.array([[1.0, 0.0], [0.0, 1.0]], dtype=np.float32)
loss, logits = MultimodalPipeline.compute_clip_infonce_loss(emb, emb, temperature=0.1)
assert loss < 0.1
assert logits[0, 0] > logits[0, 1] # Diagonal similarity is maximized
def test_clip_infonce_symmetry():
np.random.seed(42)
img = np.random.randn(4, 8).astype(np.float32)
txt = np.random.randn(4, 8).astype(np.float32)
loss, logits = MultimodalPipeline.compute_clip_infonce_loss(img, txt, temperature=0.07)
assert not np.isnan(loss)
assert logits.shape == (4, 4)
def test_zero_shot_classification():
img = np.array([1.0, 0.0, 0.0], dtype=np.float32)
class_txt = np.array([
[1.0, 0.0, 0.0], # Target class: 'cat'
[0.0, 1.0, 0.0], # 'dog'
[0.0, 0.0, 1.0], # 'car'
], dtype=np.float32)
labels = ["cat", "dog", "car"]
result = MultimodalPipeline.zero_shot_classify(img, class_txt, labels)
assert result["predicted_label"] == "cat"
assert result["confidence"] > 0.5
assert result["probabilities"]["dog"] < 0.3
def test_mlp_projector_transformation():
# 2 visual tokens, vision_dim=4, llm_dim=8
tokens = np.random.randn(1, 2, 4).astype(np.float32)
w1 = np.random.randn(4, 8).astype(np.float32) * 0.1
b1 = np.zeros(8, dtype=np.float32)
w2 = np.random.randn(8, 8).astype(np.float32) * 0.1
b2 = np.zeros(8, dtype=np.float32)
projected = MultimodalPipeline.project_visual_tokens(tokens, w1, b1, w2, b2)
assert projected.shape == (1, 2, 8)
assert not np.isnan(projected).any()
def test_visual_token_counter():
assert MultimodalPipeline.calculate_visual_token_count(336, patch_size=14) == 576
assert MultimodalPipeline.calculate_visual_token_count(224, patch_size=14) == 256
metadata.yml (404 bytes)
lesson_id: D285
day: 285
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/multimodal_models.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/multimodal_models.py (870 bytes)
# Starter: Multimodal Pipeline
import numpy as np
from typing import Dict, List, Tuple, Any
class MultimodalPipeline:
@staticmethod
def compute_clip_infonce_loss(image_embeddings: np.ndarray, text_embeddings: np.ndarray, temperature: float = 0.07) -> Tuple[float, np.ndarray]:
return 0.0, np.zeros((2, 2), dtype=np.float32)
@staticmethod
def zero_shot_classify(image_embedding: np.ndarray, class_text_embeddings: np.ndarray, class_labels: List[str]) -> Dict[str, Any]:
return {"predicted_label": class_labels[0], "confidence": 1.0}
@staticmethod
def project_visual_tokens(visual_tokens: np.ndarray, w1: np.ndarray, b1: np.ndarray, w2: np.ndarray, b2: np.ndarray) -> np.ndarray:
return visual_tokens
@staticmethod
def calculate_visual_token_count(image_size: int, patch_size: int = 14) -> int:
return 0
starter/test_multimodal_models.py (1990 bytes)
import pytest
import numpy as np
from multimodal_models import MultimodalPipeline
def test_clip_infonce_perfect_alignment():
# When image and text embeddings are identical, loss should be minimal
emb = np.array([[1.0, 0.0], [0.0, 1.0]], dtype=np.float32)
loss, logits = MultimodalPipeline.compute_clip_infonce_loss(emb, emb, temperature=0.1)
assert loss < 0.1
assert logits[0, 0] > logits[0, 1] # Diagonal similarity is maximized
def test_clip_infonce_symmetry():
np.random.seed(42)
img = np.random.randn(4, 8).astype(np.float32)
txt = np.random.randn(4, 8).astype(np.float32)
loss, logits = MultimodalPipeline.compute_clip_infonce_loss(img, txt, temperature=0.07)
assert not np.isnan(loss)
assert logits.shape == (4, 4)
def test_zero_shot_classification():
img = np.array([1.0, 0.0, 0.0], dtype=np.float32)
class_txt = np.array([
[1.0, 0.0, 0.0], # Target class: 'cat'
[0.0, 1.0, 0.0], # 'dog'
[0.0, 0.0, 1.0], # 'car'
], dtype=np.float32)
labels = ["cat", "dog", "car"]
result = MultimodalPipeline.zero_shot_classify(img, class_txt, labels)
assert result["predicted_label"] == "cat"
assert result["confidence"] > 0.5
assert result["probabilities"]["dog"] < 0.3
def test_mlp_projector_transformation():
# 2 visual tokens, vision_dim=4, llm_dim=8
tokens = np.random.randn(1, 2, 4).astype(np.float32)
w1 = np.random.randn(4, 8).astype(np.float32) * 0.1
b1 = np.zeros(8, dtype=np.float32)
w2 = np.random.randn(8, 8).astype(np.float32) * 0.1
b2 = np.zeros(8, dtype=np.float32)
projected = MultimodalPipeline.project_visual_tokens(tokens, w1, b1, w2, b2)
assert projected.shape == (1, 2, 8)
assert not np.isnan(projected).any()
def test_visual_token_counter():
assert MultimodalPipeline.calculate_visual_token_count(336, patch_size=14) == 576
assert MultimodalPipeline.calculate_visual_token_count(224, patch_size=14) == 256
tests/run_tests.sh (126 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_multimodal_models.py -v
tests/test_multimodal_models.py (1990 bytes)
import pytest
import numpy as np
from multimodal_models import MultimodalPipeline
def test_clip_infonce_perfect_alignment():
# When image and text embeddings are identical, loss should be minimal
emb = np.array([[1.0, 0.0], [0.0, 1.0]], dtype=np.float32)
loss, logits = MultimodalPipeline.compute_clip_infonce_loss(emb, emb, temperature=0.1)
assert loss < 0.1
assert logits[0, 0] > logits[0, 1] # Diagonal similarity is maximized
def test_clip_infonce_symmetry():
np.random.seed(42)
img = np.random.randn(4, 8).astype(np.float32)
txt = np.random.randn(4, 8).astype(np.float32)
loss, logits = MultimodalPipeline.compute_clip_infonce_loss(img, txt, temperature=0.07)
assert not np.isnan(loss)
assert logits.shape == (4, 4)
def test_zero_shot_classification():
img = np.array([1.0, 0.0, 0.0], dtype=np.float32)
class_txt = np.array([
[1.0, 0.0, 0.0], # Target class: 'cat'
[0.0, 1.0, 0.0], # 'dog'
[0.0, 0.0, 1.0], # 'car'
], dtype=np.float32)
labels = ["cat", "dog", "car"]
result = MultimodalPipeline.zero_shot_classify(img, class_txt, labels)
assert result["predicted_label"] == "cat"
assert result["confidence"] > 0.5
assert result["probabilities"]["dog"] < 0.3
def test_mlp_projector_transformation():
# 2 visual tokens, vision_dim=4, llm_dim=8
tokens = np.random.randn(1, 2, 4).astype(np.float32)
w1 = np.random.randn(4, 8).astype(np.float32) * 0.1
b1 = np.zeros(8, dtype=np.float32)
w2 = np.random.randn(8, 8).astype(np.float32) * 0.1
b2 = np.zeros(8, dtype=np.float32)
projected = MultimodalPipeline.project_visual_tokens(tokens, w1, b1, w2, b2)
assert projected.shape == (1, 2, 8)
assert not np.isnan(projected).any()
def test_visual_token_counter():
assert MultimodalPipeline.calculate_visual_token_count(336, patch_size=14) == 576
assert MultimodalPipeline.calculate_visual_token_count(224, patch_size=14) == 256
Troubleshooting
Troubleshooting
Ensure cosine similarity computes dot product over normalized unit vectors.
Security notes
Security
Pure NumPy matrix operations implementing symmetric InfoNCE loss and MLP projection. No network access.