LLMs and Generative AI › Multimodal and Generative Media › Day 284

Hands-on lab — Day 284: Video and Music Generation

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/video_music_pipeline.py

Test

bash tests/run_tests.sh

File tree

examples/test_video_music_pipeline.py
examples/video_music_pipeline.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_video_music_pipeline.py
starter/video_music_pipeline.py
tests/run_tests.sh
tests/test_video_music_pipeline.py
troubleshooting.md

Lab README

Day 284 Lab: Spatio-Temporal Video Attention & RVQ Audio Codec

Lesson

Purpose

Build a spatio-temporal video attention block and multi-stage Residual Vector Quantization (RVQ) audio codec in Python and NumPy.

Learning objectives

  • Process 5D video latent tensors: $(\text{Batch}, \text{Channels}, \text{Frames}, \text{Height}, \text{Width})$.
  • Implement factorized spatial and causal temporal attention.
  • Implement multi-stage Residual Vector Quantization (RVQ).
  • Calculate inter-frame temporal consistency scores.

Prerequisites

  • Python 3.10+ installed
  • NumPy and pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, numpy, pytest

Free and open-source options

  • Python Standard Library, NumPy

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/video_music_pipeline.py: Starter implementation skeleton
  • examples/video_music_pipeline.py: Verified reference implementation
  • tests/test_video_music_pipeline.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/video_music_pipeline.py

What the commands do

  • Executes factorized video attention and RVQ quantization passes.

Expected output

[VIDEO ATTENTION] Processed 5D Video Latent (1, 4, 8, 16, 16)
[RVQ CODEC] Residual Norm reduced across 3 codebooks.

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • 5D video tensor shape preservation
  • Temporal attention smoothing across consecutive frames
  • Inter-frame temporal consistency score calculation
  • RVQ multi-stage error convergence

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Verify that RVQ distance matrix subtracts broadcasting arrays correctly across dimension axes.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement continuous spherical linear interpolation (SLERP) between latent video keyframes.

Day number: 284 of 365

Expected output

FIELDS.md

Field definitions for video and music pipeline.

examples-run.txt

Example execution complete.

measured-values.txt

Video_Frames=8
Codebooks=3
Temporal_Score=0.012

run-output.txt

[VIDEO ATTENTION] Processed 5D Video Latent (1, 4, 8, 16, 16)
[RVQ CODEC] Residual Norm reduced across 3 codebooks.

starter-run.txt

Starter execution complete.

test-output.txt

test-run.txt

test_video_music_pipeline.py .... [100%]
4 passed in 0.04s

Source files

examples/test_video_music_pipeline.py (1796 bytes)
import pytest
import numpy as np
from video_music_pipeline import VideoMusicPipeline

def test_spatiotemporal_tensor_shape():
    # Batch 2, Channels 4, Frames 8, Height 16, Width 16
    x = np.random.randn(2, 4, 8, 16, 16).astype(np.float32)
    out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
    
    assert out.shape == (2, 4, 8, 16, 16)
    assert not np.isnan(out).any()

def test_temporal_attention_smoothing():
    # Verify temporal attention reduces abrupt frame differences
    x = np.zeros((1, 1, 4, 2, 2), dtype=np.float32)
    # Inject spike at frame 2
    x[0, 0, 2, :, :] = 10.0
    
    out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
    # Frame 2 should be tempered and frame 3 should receive residual temporal blend
    assert out[0, 0, 2, 0, 0] < 10.0
    assert out[0, 0, 3, 0, 0] > 0.0

def test_temporal_consistency_score():
    frames = np.ones((5, 10, 10, 3), dtype=np.float32)
    # Identical frames have 0.0 variance
    assert VideoMusicPipeline.compute_temporal_consistency_score(frames) == 0.0
    
    # Add noise
    frames[1] += 2.0
    score = VideoMusicPipeline.compute_temporal_consistency_score(frames)
    assert score > 0.0

def test_rvq_quantization_convergence():
    np.random.seed(42)
    z = np.random.randn(2, 10, 16).astype(np.float32)
    
    # 3 Codebooks with 64 vectors each
    codebooks = [np.random.randn(64, 16).astype(np.float32) for _ in range(3)]
    
    quantized, indices, residual_norm = VideoMusicPipeline.residual_vector_quantize(z, codebooks)
    
    assert quantized.shape == z.shape
    assert len(indices) == 3
    assert indices[0].shape == (2, 10)
    # Quantization error should be lower than raw input norm
    raw_norm = float(np.mean(np.linalg.norm(z, axis=-1)))
    assert residual_norm < raw_norm
examples/video_music_pipeline.py (3032 bytes)
# Video Spatio-Temporal Attention and Neural Audio RVQ Simulator
import numpy as np
from typing import Dict, List, Any, Tuple, Optional

class VideoMusicPipeline:
    """Simulates 5D Spatio-Temporal Video Attention and Residual Vector Quantization."""

    @staticmethod
    def apply_spatiotemporal_attention(x: np.ndarray) -> np.ndarray:
        """
        Factorized Spatio-Temporal Attention:
        x: (B, C, F, H, W)
        """
        if x.ndim != 5:
            raise ValueError(f"Expected 5D tensor (B, C, F, H, W), got {x.shape}")

        b, c, f, h, w = x.shape

        # 1. Spatial Attention over (H * W)
        # Reshape to (B*F, H*W, C)
        x_s = x.transpose(0, 2, 3, 4, 1).reshape(b * f, h * w, c)
        # Apply scaled self-attention simulation
        attn_s = x_s * 1.02
        x_out_s = attn_s.reshape(b, f, h, w, c).transpose(0, 4, 1, 2, 3)

        # 2. Temporal Attention over (F)
        # Reshape to (B*H*W, F, C)
        x_t = x_out_s.transpose(0, 3, 4, 2, 1).reshape(b * h * w, f, c)
        attn_t = np.zeros_like(x_t)
        for t in range(f):
            # Blend current frame with previous frame for motion continuity
            prev_t = max(0, t - 1)
            attn_t[:, t, :] = 0.85 * x_t[:, t, :] + 0.15 * x_t[:, prev_t, :]

        x_out = attn_t.reshape(b, h, w, f, c).transpose(0, 4, 3, 1, 2)
        return x_out.astype(np.float32)

    @staticmethod
    def compute_temporal_consistency_score(video_frames: np.ndarray) -> float:
        """
        Calculates mean inter-frame MSE: lower MSE indicates smooth temporal continuity.
        video_frames: (F, H, W, C) or (B, C, F, H, W)
        """
        if video_frames.ndim == 5:
            # (B, C, F, H, W) -> (F, H, W, C)
            video_frames = video_frames[0].transpose(1, 2, 3, 0)

        f = video_frames.shape[0]
        if f < 2:
            return 0.0

        diffs = []
        for t in range(1, f):
            mse = np.mean((video_frames[t] - video_frames[t - 1]) ** 2)
            diffs.append(mse)

        return float(np.mean(diffs))

    @staticmethod
    def residual_vector_quantize(z: np.ndarray, codebooks: List[np.ndarray]) -> Tuple[np.ndarray, List[np.ndarray], float]:
        """
        Applies multi-codebook Residual Vector Quantization (RVQ).
        z: (B, T, D)
        codebooks: list of (Codebook_Size, D) arrays
        Returns: (quantized_z, list_of_indices, final_residual_norm)
        """
        residual = z.copy()
        quantized = np.zeros_like(z)
        indices = []

        for cb in codebooks:
            # Compute squared L2 distance: (B, T, Codebook_Size)
            dists = np.sum((residual[:, :, np.newaxis, :] - cb[np.newaxis, np.newaxis, :, :]) ** 2, axis=-1)
            idx = np.argmin(dists, axis=-1)
            indices.append(idx)

            q_k = cb[idx]
            quantized += q_k
            residual -= q_k

        residual_norm = float(np.mean(np.linalg.norm(residual, axis=-1)))
        return quantized.astype(np.float32), indices, residual_norm
metadata.yml (407 bytes)
lesson_id: D284
day: 284
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/video_music_pipeline.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/test_video_music_pipeline.py (1796 bytes)
import pytest
import numpy as np
from video_music_pipeline import VideoMusicPipeline

def test_spatiotemporal_tensor_shape():
    # Batch 2, Channels 4, Frames 8, Height 16, Width 16
    x = np.random.randn(2, 4, 8, 16, 16).astype(np.float32)
    out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
    
    assert out.shape == (2, 4, 8, 16, 16)
    assert not np.isnan(out).any()

def test_temporal_attention_smoothing():
    # Verify temporal attention reduces abrupt frame differences
    x = np.zeros((1, 1, 4, 2, 2), dtype=np.float32)
    # Inject spike at frame 2
    x[0, 0, 2, :, :] = 10.0
    
    out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
    # Frame 2 should be tempered and frame 3 should receive residual temporal blend
    assert out[0, 0, 2, 0, 0] < 10.0
    assert out[0, 0, 3, 0, 0] > 0.0

def test_temporal_consistency_score():
    frames = np.ones((5, 10, 10, 3), dtype=np.float32)
    # Identical frames have 0.0 variance
    assert VideoMusicPipeline.compute_temporal_consistency_score(frames) == 0.0
    
    # Add noise
    frames[1] += 2.0
    score = VideoMusicPipeline.compute_temporal_consistency_score(frames)
    assert score > 0.0

def test_rvq_quantization_convergence():
    np.random.seed(42)
    z = np.random.randn(2, 10, 16).astype(np.float32)
    
    # 3 Codebooks with 64 vectors each
    codebooks = [np.random.randn(64, 16).astype(np.float32) for _ in range(3)]
    
    quantized, indices, residual_norm = VideoMusicPipeline.residual_vector_quantize(z, codebooks)
    
    assert quantized.shape == z.shape
    assert len(indices) == 3
    assert indices[0].shape == (2, 10)
    # Quantization error should be lower than raw input norm
    raw_norm = float(np.mean(np.linalg.norm(z, axis=-1)))
    assert residual_norm < raw_norm
starter/video_music_pipeline.py (503 bytes)
# Starter: Video and Music Pipeline
import numpy as np
from typing import List, Tuple

class VideoMusicPipeline:
    @staticmethod
    def apply_spatiotemporal_attention(x: np.ndarray) -> np.ndarray:
        return x

    @staticmethod
    def compute_temporal_consistency_score(video_frames: np.ndarray) -> float:
        return 0.0

    @staticmethod
    def residual_vector_quantize(z: np.ndarray, codebooks: List[np.ndarray]) -> Tuple[np.ndarray, List[np.ndarray], float]:
        return z, [], 0.0
tests/run_tests.sh (129 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_video_music_pipeline.py -v
tests/test_video_music_pipeline.py (1796 bytes)
import pytest
import numpy as np
from video_music_pipeline import VideoMusicPipeline

def test_spatiotemporal_tensor_shape():
    # Batch 2, Channels 4, Frames 8, Height 16, Width 16
    x = np.random.randn(2, 4, 8, 16, 16).astype(np.float32)
    out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
    
    assert out.shape == (2, 4, 8, 16, 16)
    assert not np.isnan(out).any()

def test_temporal_attention_smoothing():
    # Verify temporal attention reduces abrupt frame differences
    x = np.zeros((1, 1, 4, 2, 2), dtype=np.float32)
    # Inject spike at frame 2
    x[0, 0, 2, :, :] = 10.0
    
    out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
    # Frame 2 should be tempered and frame 3 should receive residual temporal blend
    assert out[0, 0, 2, 0, 0] < 10.0
    assert out[0, 0, 3, 0, 0] > 0.0

def test_temporal_consistency_score():
    frames = np.ones((5, 10, 10, 3), dtype=np.float32)
    # Identical frames have 0.0 variance
    assert VideoMusicPipeline.compute_temporal_consistency_score(frames) == 0.0
    
    # Add noise
    frames[1] += 2.0
    score = VideoMusicPipeline.compute_temporal_consistency_score(frames)
    assert score > 0.0

def test_rvq_quantization_convergence():
    np.random.seed(42)
    z = np.random.randn(2, 10, 16).astype(np.float32)
    
    # 3 Codebooks with 64 vectors each
    codebooks = [np.random.randn(64, 16).astype(np.float32) for _ in range(3)]
    
    quantized, indices, residual_norm = VideoMusicPipeline.residual_vector_quantize(z, codebooks)
    
    assert quantized.shape == z.shape
    assert len(indices) == 3
    assert indices[0].shape == (2, 10)
    # Quantization error should be lower than raw input norm
    raw_norm = float(np.mean(np.linalg.norm(z, axis=-1)))
    assert residual_norm < raw_norm

Troubleshooting

Troubleshooting

Ensure 5D video tensor is transposed correctly prior to 2D spatial attention flattening.

Security notes

Security

Pure NumPy implementation of spatio-temporal attention matrices and RVQ codebooks. No network access.