LLMs and Generative AI › Multimodal and Generative Media › Day 284
Hands-on lab — Day 284: Video and Music Generation
- ← Back to the Day 284 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-284-video-and-music-generation/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/video_music_pipeline.py Test
bash tests/run_tests.sh File tree
examples/test_video_music_pipeline.py examples/video_music_pipeline.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/test_video_music_pipeline.py starter/video_music_pipeline.py tests/run_tests.sh tests/test_video_music_pipeline.py troubleshooting.md
Lab README
Day 284 Lab: Spatio-Temporal Video Attention & RVQ Audio Codec
Lesson
- Lesson title: Video and Music Generation
- Day number: 284 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-284-video-and-music-generation
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-284-video-and-music-generationwhen the site is running.
Purpose
Build a spatio-temporal video attention block and multi-stage Residual Vector Quantization (RVQ) audio codec in Python and NumPy.
Learning objectives
- Process 5D video latent tensors: $(\text{Batch}, \text{Channels}, \text{Frames}, \text{Height}, \text{Width})$.
- Implement factorized spatial and causal temporal attention.
- Implement multi-stage Residual Vector Quantization (RVQ).
- Calculate inter-frame temporal consistency scores.
Prerequisites
- Python 3.10+ installed
- NumPy and pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, numpy, pytest
Free and open-source options
- Python Standard Library, NumPy
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/video_music_pipeline.py: Starter implementation skeletonexamples/video_music_pipeline.py: Verified reference implementationtests/test_video_music_pipeline.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/video_music_pipeline.py
What the commands do
- Executes factorized video attention and RVQ quantization passes.
Expected output
[VIDEO ATTENTION] Processed 5D Video Latent (1, 4, 8, 16, 16)
[RVQ CODEC] Residual Norm reduced across 3 codebooks.
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- 5D video tensor shape preservation
- Temporal attention smoothing across consecutive frames
- Inter-frame temporal consistency score calculation
- RVQ multi-stage error convergence
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Verify that RVQ distance matrix subtracts broadcasting arrays correctly across dimension axes.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement continuous spherical linear interpolation (SLERP) between latent video keyframes.
Navigation
Day number: 284 of 365
Expected output
FIELDS.md
Field definitions for video and music pipeline. examples-run.txt
Example execution complete.
measured-values.txt
Video_Frames=8
Codebooks=3
Temporal_Score=0.012
run-output.txt
[VIDEO ATTENTION] Processed 5D Video Latent (1, 4, 8, 16, 16)
[RVQ CODEC] Residual Norm reduced across 3 codebooks.
starter-run.txt
Starter execution complete.
test-output.txt
test-run.txt
test_video_music_pipeline.py .... [100%]
4 passed in 0.04s
Source files
examples/test_video_music_pipeline.py (1796 bytes)
import pytest
import numpy as np
from video_music_pipeline import VideoMusicPipeline
def test_spatiotemporal_tensor_shape():
# Batch 2, Channels 4, Frames 8, Height 16, Width 16
x = np.random.randn(2, 4, 8, 16, 16).astype(np.float32)
out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
assert out.shape == (2, 4, 8, 16, 16)
assert not np.isnan(out).any()
def test_temporal_attention_smoothing():
# Verify temporal attention reduces abrupt frame differences
x = np.zeros((1, 1, 4, 2, 2), dtype=np.float32)
# Inject spike at frame 2
x[0, 0, 2, :, :] = 10.0
out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
# Frame 2 should be tempered and frame 3 should receive residual temporal blend
assert out[0, 0, 2, 0, 0] < 10.0
assert out[0, 0, 3, 0, 0] > 0.0
def test_temporal_consistency_score():
frames = np.ones((5, 10, 10, 3), dtype=np.float32)
# Identical frames have 0.0 variance
assert VideoMusicPipeline.compute_temporal_consistency_score(frames) == 0.0
# Add noise
frames[1] += 2.0
score = VideoMusicPipeline.compute_temporal_consistency_score(frames)
assert score > 0.0
def test_rvq_quantization_convergence():
np.random.seed(42)
z = np.random.randn(2, 10, 16).astype(np.float32)
# 3 Codebooks with 64 vectors each
codebooks = [np.random.randn(64, 16).astype(np.float32) for _ in range(3)]
quantized, indices, residual_norm = VideoMusicPipeline.residual_vector_quantize(z, codebooks)
assert quantized.shape == z.shape
assert len(indices) == 3
assert indices[0].shape == (2, 10)
# Quantization error should be lower than raw input norm
raw_norm = float(np.mean(np.linalg.norm(z, axis=-1)))
assert residual_norm < raw_norm
examples/video_music_pipeline.py (3032 bytes)
# Video Spatio-Temporal Attention and Neural Audio RVQ Simulator
import numpy as np
from typing import Dict, List, Any, Tuple, Optional
class VideoMusicPipeline:
"""Simulates 5D Spatio-Temporal Video Attention and Residual Vector Quantization."""
@staticmethod
def apply_spatiotemporal_attention(x: np.ndarray) -> np.ndarray:
"""
Factorized Spatio-Temporal Attention:
x: (B, C, F, H, W)
"""
if x.ndim != 5:
raise ValueError(f"Expected 5D tensor (B, C, F, H, W), got {x.shape}")
b, c, f, h, w = x.shape
# 1. Spatial Attention over (H * W)
# Reshape to (B*F, H*W, C)
x_s = x.transpose(0, 2, 3, 4, 1).reshape(b * f, h * w, c)
# Apply scaled self-attention simulation
attn_s = x_s * 1.02
x_out_s = attn_s.reshape(b, f, h, w, c).transpose(0, 4, 1, 2, 3)
# 2. Temporal Attention over (F)
# Reshape to (B*H*W, F, C)
x_t = x_out_s.transpose(0, 3, 4, 2, 1).reshape(b * h * w, f, c)
attn_t = np.zeros_like(x_t)
for t in range(f):
# Blend current frame with previous frame for motion continuity
prev_t = max(0, t - 1)
attn_t[:, t, :] = 0.85 * x_t[:, t, :] + 0.15 * x_t[:, prev_t, :]
x_out = attn_t.reshape(b, h, w, f, c).transpose(0, 4, 3, 1, 2)
return x_out.astype(np.float32)
@staticmethod
def compute_temporal_consistency_score(video_frames: np.ndarray) -> float:
"""
Calculates mean inter-frame MSE: lower MSE indicates smooth temporal continuity.
video_frames: (F, H, W, C) or (B, C, F, H, W)
"""
if video_frames.ndim == 5:
# (B, C, F, H, W) -> (F, H, W, C)
video_frames = video_frames[0].transpose(1, 2, 3, 0)
f = video_frames.shape[0]
if f < 2:
return 0.0
diffs = []
for t in range(1, f):
mse = np.mean((video_frames[t] - video_frames[t - 1]) ** 2)
diffs.append(mse)
return float(np.mean(diffs))
@staticmethod
def residual_vector_quantize(z: np.ndarray, codebooks: List[np.ndarray]) -> Tuple[np.ndarray, List[np.ndarray], float]:
"""
Applies multi-codebook Residual Vector Quantization (RVQ).
z: (B, T, D)
codebooks: list of (Codebook_Size, D) arrays
Returns: (quantized_z, list_of_indices, final_residual_norm)
"""
residual = z.copy()
quantized = np.zeros_like(z)
indices = []
for cb in codebooks:
# Compute squared L2 distance: (B, T, Codebook_Size)
dists = np.sum((residual[:, :, np.newaxis, :] - cb[np.newaxis, np.newaxis, :, :]) ** 2, axis=-1)
idx = np.argmin(dists, axis=-1)
indices.append(idx)
q_k = cb[idx]
quantized += q_k
residual -= q_k
residual_norm = float(np.mean(np.linalg.norm(residual, axis=-1)))
return quantized.astype(np.float32), indices, residual_norm
metadata.yml (407 bytes)
lesson_id: D284
day: 284
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/video_music_pipeline.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (28 bytes)
numpy>=1.24.0
pytest>=8.0.0
starter/test_video_music_pipeline.py (1796 bytes)
import pytest
import numpy as np
from video_music_pipeline import VideoMusicPipeline
def test_spatiotemporal_tensor_shape():
# Batch 2, Channels 4, Frames 8, Height 16, Width 16
x = np.random.randn(2, 4, 8, 16, 16).astype(np.float32)
out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
assert out.shape == (2, 4, 8, 16, 16)
assert not np.isnan(out).any()
def test_temporal_attention_smoothing():
# Verify temporal attention reduces abrupt frame differences
x = np.zeros((1, 1, 4, 2, 2), dtype=np.float32)
# Inject spike at frame 2
x[0, 0, 2, :, :] = 10.0
out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
# Frame 2 should be tempered and frame 3 should receive residual temporal blend
assert out[0, 0, 2, 0, 0] < 10.0
assert out[0, 0, 3, 0, 0] > 0.0
def test_temporal_consistency_score():
frames = np.ones((5, 10, 10, 3), dtype=np.float32)
# Identical frames have 0.0 variance
assert VideoMusicPipeline.compute_temporal_consistency_score(frames) == 0.0
# Add noise
frames[1] += 2.0
score = VideoMusicPipeline.compute_temporal_consistency_score(frames)
assert score > 0.0
def test_rvq_quantization_convergence():
np.random.seed(42)
z = np.random.randn(2, 10, 16).astype(np.float32)
# 3 Codebooks with 64 vectors each
codebooks = [np.random.randn(64, 16).astype(np.float32) for _ in range(3)]
quantized, indices, residual_norm = VideoMusicPipeline.residual_vector_quantize(z, codebooks)
assert quantized.shape == z.shape
assert len(indices) == 3
assert indices[0].shape == (2, 10)
# Quantization error should be lower than raw input norm
raw_norm = float(np.mean(np.linalg.norm(z, axis=-1)))
assert residual_norm < raw_norm
starter/video_music_pipeline.py (503 bytes)
# Starter: Video and Music Pipeline
import numpy as np
from typing import List, Tuple
class VideoMusicPipeline:
@staticmethod
def apply_spatiotemporal_attention(x: np.ndarray) -> np.ndarray:
return x
@staticmethod
def compute_temporal_consistency_score(video_frames: np.ndarray) -> float:
return 0.0
@staticmethod
def residual_vector_quantize(z: np.ndarray, codebooks: List[np.ndarray]) -> Tuple[np.ndarray, List[np.ndarray], float]:
return z, [], 0.0
tests/run_tests.sh (129 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
PYTEST="${PYTEST:-pytest}"
$PYTEST tests/test_video_music_pipeline.py -v
tests/test_video_music_pipeline.py (1796 bytes)
import pytest
import numpy as np
from video_music_pipeline import VideoMusicPipeline
def test_spatiotemporal_tensor_shape():
# Batch 2, Channels 4, Frames 8, Height 16, Width 16
x = np.random.randn(2, 4, 8, 16, 16).astype(np.float32)
out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
assert out.shape == (2, 4, 8, 16, 16)
assert not np.isnan(out).any()
def test_temporal_attention_smoothing():
# Verify temporal attention reduces abrupt frame differences
x = np.zeros((1, 1, 4, 2, 2), dtype=np.float32)
# Inject spike at frame 2
x[0, 0, 2, :, :] = 10.0
out = VideoMusicPipeline.apply_spatiotemporal_attention(x)
# Frame 2 should be tempered and frame 3 should receive residual temporal blend
assert out[0, 0, 2, 0, 0] < 10.0
assert out[0, 0, 3, 0, 0] > 0.0
def test_temporal_consistency_score():
frames = np.ones((5, 10, 10, 3), dtype=np.float32)
# Identical frames have 0.0 variance
assert VideoMusicPipeline.compute_temporal_consistency_score(frames) == 0.0
# Add noise
frames[1] += 2.0
score = VideoMusicPipeline.compute_temporal_consistency_score(frames)
assert score > 0.0
def test_rvq_quantization_convergence():
np.random.seed(42)
z = np.random.randn(2, 10, 16).astype(np.float32)
# 3 Codebooks with 64 vectors each
codebooks = [np.random.randn(64, 16).astype(np.float32) for _ in range(3)]
quantized, indices, residual_norm = VideoMusicPipeline.residual_vector_quantize(z, codebooks)
assert quantized.shape == z.shape
assert len(indices) == 3
assert indices[0].shape == (2, 10)
# Quantization error should be lower than raw input norm
raw_norm = float(np.mean(np.linalg.norm(z, axis=-1)))
assert residual_norm < raw_norm
Troubleshooting
Troubleshooting
Ensure 5D video tensor is transposed correctly prior to 2D spatial attention flattening.
Security notes
Security
Pure NumPy implementation of spatio-temporal attention matrices and RVQ codebooks. No network access.