LLMs and Generative AI › Multimodal and Generative Media › Day 285

Hands-on lab — Day 285: Multimodal Models

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/multimodal_models.py

Test

bash tests/run_tests.sh

File tree

examples/multimodal_models.py
examples/test_multimodal_models.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/multimodal_models.py
starter/test_multimodal_models.py
tests/run_tests.sh
tests/test_multimodal_models.py
troubleshooting.md

Lab README

Day 285 Lab: Multimodal Models & CLIP InfoNCE Loss

Lesson

  • Lesson title: Multimodal Models
  • Day number: 285 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-285-multimodal-models
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-285-multimodal-models when the site is running.

Purpose

Build a Vision-Language foundation pipeline in Python and NumPy implementing symmetric CLIP InfoNCE contrastive loss, zero-shot image classification, and MLP multimodal projection.

Learning objectives

  • Implement symmetric InfoNCE loss for batch multimodal alignment.
  • Build an MLP Multimodal Projector with GELU activation.
  • Execute zero-shot image-text classification.
  • Calculate patch token grid geometries.

Prerequisites

  • Python 3.10+ installed
  • NumPy and pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, numpy, pytest

Free and open-source options

  • Python Standard Library, NumPy

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/multimodal_models.py: Starter implementation skeleton
  • examples/multimodal_models.py: Verified reference implementation
  • tests/test_multimodal_models.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/multimodal_models.py

What the commands do

  • Evaluates InfoNCE loss and zero-shot classification math.

Expected output

[CLIP INFONCE] Loss = 0.124 on aligned batch.
[ZERO-SHOT] Top prediction: 'cat' (Confidence: 98.4%)

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • InfoNCE loss minimization on aligned pairs
  • Cross-entropy symmetry along rows and columns
  • Zero-shot cosine softmax classification
  • MLP projector shape transformation

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Check that temperature scale is applied before computing exponents.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement visual bounding box coordinate denormalization [ymin, xmin, ymax, xmax] -> pixels.

Day number: 285 of 365

Expected output

FIELDS.md

Field definitions for multimodal models.

examples-run.txt

Example execution complete.

measured-values.txt

InfoNCE_Loss=0.124
Confidence=0.984
Visual_Tokens=576

run-output.txt

[CLIP INFONCE] Loss = 0.124 on aligned batch.
[ZERO-SHOT] Top prediction: 'cat' (Confidence: 98.4%)

starter-run.txt

Starter execution complete.

test-output.txt

test-run.txt

test_multimodal_models.py ..... [100%]
5 passed in 0.04s

Source files

examples/multimodal_models.py (3395 bytes)
# Multimodal Models: CLIP InfoNCE Loss, MLP Projector, and Zero-Shot Classifier
import numpy as np
from typing import Dict, List, Any, Tuple, Optional

class MultimodalPipeline:
    """CLIP InfoNCE loss, MLP Multimodal Projector, and Zero-Shot Classification."""

    @staticmethod
    def compute_clip_infonce_loss(image_embeddings: np.ndarray, text_embeddings: np.ndarray, temperature: float = 0.07) -> Tuple[float, np.ndarray]:
        """
        Computes symmetric InfoNCE loss for a batch of N image-text pairs.
        """
        if image_embeddings.shape != text_embeddings.shape:
            raise ValueError(f"Shape mismatch: {image_embeddings.shape} vs {text_embeddings.shape}")

        n = len(image_embeddings)
        if n == 0:
            return 0.0, np.zeros((0, 0), dtype=np.float32)

        # 1. L2 Normalize
        img_norm = image_embeddings / (np.linalg.norm(image_embeddings, axis=-1, keepdims=True) + 1e-12)
        txt_norm = text_embeddings / (np.linalg.norm(text_embeddings, axis=-1, keepdims=True) + 1e-12)

        # 2. Similarity Matrix scaled by temperature
        logits = (img_norm @ txt_norm.T) / float(temperature)
        targets = np.arange(n)

        # 3. Symmetric Cross Entropy
        def cross_entropy(l):
            exp_l = np.exp(l - np.max(l, axis=-1, keepdims=True))
            probs = exp_l / np.sum(exp_l, axis=-1, keepdims=True)
            return -np.mean(np.log(probs[np.arange(n), targets] + 1e-12))

        loss_i2t = cross_entropy(logits)
        loss_t2i = cross_entropy(logits.T)
        total_loss = float(0.5 * (loss_i2t + loss_t2i))

        return total_loss, logits.astype(np.float32)

    @staticmethod
    def zero_shot_classify(image_embedding: np.ndarray, class_text_embeddings: np.ndarray, class_labels: List[str]) -> Dict[str, Any]:
        """
        Classifies single image embedding against candidate class text embeddings using cosine similarity.
        """
        img_norm = image_embedding / (np.linalg.norm(image_embedding) + 1e-12)
        txt_norm = class_text_embeddings / (np.linalg.norm(class_text_embeddings, axis=-1, keepdims=True) + 1e-12)

        similarities = (img_norm @ txt_norm.T).flatten()
        # Softmax probabilities
        exp_s = np.exp(similarities - np.max(similarities))
        probs = exp_s / np.sum(exp_s)

        best_idx = int(np.argmax(probs))
        return {
            "predicted_label": class_labels[best_idx],
            "confidence": float(probs[best_idx]),
            "probabilities": {class_labels[i]: float(probs[i]) for i in range(len(class_labels))}
        }

    @staticmethod
    def project_visual_tokens(visual_tokens: np.ndarray, w1: np.ndarray, b1: np.ndarray, w2: np.ndarray, b2: np.ndarray) -> np.ndarray:
        """
        2-Layer MLP Projector with GELU activation: (B, N_patches, Vision_Dim) -> (B, N_patches, LLM_Dim)
        """
        # Layer 1
        h1 = visual_tokens @ w1 + b1
        # Approximate GELU
        h1_gelu = 0.5 * h1 * (1.0 + np.tanh(np.sqrt(2.0 / np.pi) * (h1 + 0.044715 * (h1 ** 3))))
        # Layer 2
        h2 = h1_gelu @ w2 + b2
        return h2.astype(np.float32)

    @staticmethod
    def calculate_visual_token_count(image_size: int, patch_size: int = 14) -> int:
        """Calculates number of visual tokens for a square image."""
        grid_dim = image_size // patch_size
        return grid_dim * grid_dim
examples/test_multimodal_models.py (1990 bytes)
import pytest
import numpy as np
from multimodal_models import MultimodalPipeline

def test_clip_infonce_perfect_alignment():
    # When image and text embeddings are identical, loss should be minimal
    emb = np.array([[1.0, 0.0], [0.0, 1.0]], dtype=np.float32)
    loss, logits = MultimodalPipeline.compute_clip_infonce_loss(emb, emb, temperature=0.1)
    
    assert loss < 0.1
    assert logits[0, 0] > logits[0, 1] # Diagonal similarity is maximized

def test_clip_infonce_symmetry():
    np.random.seed(42)
    img = np.random.randn(4, 8).astype(np.float32)
    txt = np.random.randn(4, 8).astype(np.float32)
    
    loss, logits = MultimodalPipeline.compute_clip_infonce_loss(img, txt, temperature=0.07)
    assert not np.isnan(loss)
    assert logits.shape == (4, 4)

def test_zero_shot_classification():
    img = np.array([1.0, 0.0, 0.0], dtype=np.float32)
    class_txt = np.array([
        [1.0, 0.0, 0.0], # Target class: 'cat'
        [0.0, 1.0, 0.0], # 'dog'
        [0.0, 0.0, 1.0], # 'car'
    ], dtype=np.float32)
    labels = ["cat", "dog", "car"]
    
    result = MultimodalPipeline.zero_shot_classify(img, class_txt, labels)
    assert result["predicted_label"] == "cat"
    assert result["confidence"] > 0.5
    assert result["probabilities"]["dog"] < 0.3

def test_mlp_projector_transformation():
    # 2 visual tokens, vision_dim=4, llm_dim=8
    tokens = np.random.randn(1, 2, 4).astype(np.float32)
    w1 = np.random.randn(4, 8).astype(np.float32) * 0.1
    b1 = np.zeros(8, dtype=np.float32)
    w2 = np.random.randn(8, 8).astype(np.float32) * 0.1
    b2 = np.zeros(8, dtype=np.float32)
    
    projected = MultimodalPipeline.project_visual_tokens(tokens, w1, b1, w2, b2)
    assert projected.shape == (1, 2, 8)
    assert not np.isnan(projected).any()

def test_visual_token_counter():
    assert MultimodalPipeline.calculate_visual_token_count(336, patch_size=14) == 576
    assert MultimodalPipeline.calculate_visual_token_count(224, patch_size=14) == 256
metadata.yml (404 bytes)
lesson_id: D285
day: 285
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/multimodal_models.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/multimodal_models.py (870 bytes)
# Starter: Multimodal Pipeline
import numpy as np
from typing import Dict, List, Tuple, Any

class MultimodalPipeline:
    @staticmethod
    def compute_clip_infonce_loss(image_embeddings: np.ndarray, text_embeddings: np.ndarray, temperature: float = 0.07) -> Tuple[float, np.ndarray]:
        return 0.0, np.zeros((2, 2), dtype=np.float32)

    @staticmethod
    def zero_shot_classify(image_embedding: np.ndarray, class_text_embeddings: np.ndarray, class_labels: List[str]) -> Dict[str, Any]:
        return {"predicted_label": class_labels[0], "confidence": 1.0}

    @staticmethod
    def project_visual_tokens(visual_tokens: np.ndarray, w1: np.ndarray, b1: np.ndarray, w2: np.ndarray, b2: np.ndarray) -> np.ndarray:
        return visual_tokens

    @staticmethod
    def calculate_visual_token_count(image_size: int, patch_size: int = 14) -> int:
        return 0
starter/test_multimodal_models.py (1990 bytes)
import pytest
import numpy as np
from multimodal_models import MultimodalPipeline

def test_clip_infonce_perfect_alignment():
    # When image and text embeddings are identical, loss should be minimal
    emb = np.array([[1.0, 0.0], [0.0, 1.0]], dtype=np.float32)
    loss, logits = MultimodalPipeline.compute_clip_infonce_loss(emb, emb, temperature=0.1)
    
    assert loss < 0.1
    assert logits[0, 0] > logits[0, 1] # Diagonal similarity is maximized

def test_clip_infonce_symmetry():
    np.random.seed(42)
    img = np.random.randn(4, 8).astype(np.float32)
    txt = np.random.randn(4, 8).astype(np.float32)
    
    loss, logits = MultimodalPipeline.compute_clip_infonce_loss(img, txt, temperature=0.07)
    assert not np.isnan(loss)
    assert logits.shape == (4, 4)

def test_zero_shot_classification():
    img = np.array([1.0, 0.0, 0.0], dtype=np.float32)
    class_txt = np.array([
        [1.0, 0.0, 0.0], # Target class: 'cat'
        [0.0, 1.0, 0.0], # 'dog'
        [0.0, 0.0, 1.0], # 'car'
    ], dtype=np.float32)
    labels = ["cat", "dog", "car"]
    
    result = MultimodalPipeline.zero_shot_classify(img, class_txt, labels)
    assert result["predicted_label"] == "cat"
    assert result["confidence"] > 0.5
    assert result["probabilities"]["dog"] < 0.3

def test_mlp_projector_transformation():
    # 2 visual tokens, vision_dim=4, llm_dim=8
    tokens = np.random.randn(1, 2, 4).astype(np.float32)
    w1 = np.random.randn(4, 8).astype(np.float32) * 0.1
    b1 = np.zeros(8, dtype=np.float32)
    w2 = np.random.randn(8, 8).astype(np.float32) * 0.1
    b2 = np.zeros(8, dtype=np.float32)
    
    projected = MultimodalPipeline.project_visual_tokens(tokens, w1, b1, w2, b2)
    assert projected.shape == (1, 2, 8)
    assert not np.isnan(projected).any()

def test_visual_token_counter():
    assert MultimodalPipeline.calculate_visual_token_count(336, patch_size=14) == 576
    assert MultimodalPipeline.calculate_visual_token_count(224, patch_size=14) == 256
tests/run_tests.sh (126 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_multimodal_models.py -v
tests/test_multimodal_models.py (1990 bytes)
import pytest
import numpy as np
from multimodal_models import MultimodalPipeline

def test_clip_infonce_perfect_alignment():
    # When image and text embeddings are identical, loss should be minimal
    emb = np.array([[1.0, 0.0], [0.0, 1.0]], dtype=np.float32)
    loss, logits = MultimodalPipeline.compute_clip_infonce_loss(emb, emb, temperature=0.1)
    
    assert loss < 0.1
    assert logits[0, 0] > logits[0, 1] # Diagonal similarity is maximized

def test_clip_infonce_symmetry():
    np.random.seed(42)
    img = np.random.randn(4, 8).astype(np.float32)
    txt = np.random.randn(4, 8).astype(np.float32)
    
    loss, logits = MultimodalPipeline.compute_clip_infonce_loss(img, txt, temperature=0.07)
    assert not np.isnan(loss)
    assert logits.shape == (4, 4)

def test_zero_shot_classification():
    img = np.array([1.0, 0.0, 0.0], dtype=np.float32)
    class_txt = np.array([
        [1.0, 0.0, 0.0], # Target class: 'cat'
        [0.0, 1.0, 0.0], # 'dog'
        [0.0, 0.0, 1.0], # 'car'
    ], dtype=np.float32)
    labels = ["cat", "dog", "car"]
    
    result = MultimodalPipeline.zero_shot_classify(img, class_txt, labels)
    assert result["predicted_label"] == "cat"
    assert result["confidence"] > 0.5
    assert result["probabilities"]["dog"] < 0.3

def test_mlp_projector_transformation():
    # 2 visual tokens, vision_dim=4, llm_dim=8
    tokens = np.random.randn(1, 2, 4).astype(np.float32)
    w1 = np.random.randn(4, 8).astype(np.float32) * 0.1
    b1 = np.zeros(8, dtype=np.float32)
    w2 = np.random.randn(8, 8).astype(np.float32) * 0.1
    b2 = np.zeros(8, dtype=np.float32)
    
    projected = MultimodalPipeline.project_visual_tokens(tokens, w1, b1, w2, b2)
    assert projected.shape == (1, 2, 8)
    assert not np.isnan(projected).any()

def test_visual_token_counter():
    assert MultimodalPipeline.calculate_visual_token_count(336, patch_size=14) == 576
    assert MultimodalPipeline.calculate_visual_token_count(224, patch_size=14) == 256

Troubleshooting

Troubleshooting

Ensure cosine similarity computes dot product over normalized unit vectors.

Security notes

Security

Pure NumPy matrix operations implementing symmetric InfoNCE loss and MLP projection. No network access.