LLMs and Generative AI › Embeddings and Vector Search › Day 260

Hands-on lab — Day 260: What Embeddings Are

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/what_embeddings_are_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/test_what_embeddings_are_lib.py
examples/what_embeddings_are_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_what_embeddings_are_lib.py
starter/what_embeddings_are_lib.py
tests/run_tests.sh
tests/test_what_embeddings_are_lib.py
troubleshooting.md

Lab README

Lab: Day 260 -- What Embeddings Are

Lesson

Day number: 260 of 365. Course: Course06-SS02 (LLMs and Generative AI - Retrieval and Customization). Topic: Dense Embeddings, Vector Spaces, and Similarity Mathematics.

Purpose

Build and test a Pure NumPy Vector Similarity Engine. Implement L2 vector normalization, cosine similarity, Euclidean distance, multi-document semantic ranking, and Matryoshka dimension truncation.

Learning objectives

  • Normalize multi-dimensional vectors to unit sphere length.
  • Calculate cosine similarity and Euclidean distance between vectors.
  • Rank a collection of candidate document vectors against an incoming query vector.
  • Implement Matryoshka sub-vector truncation and re-normalization.

Prerequisites

  • Day 259 (Building a CLI Assistant).
  • Python 3.11+ with NumPy and Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

NumPy and Python standard libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/what_embeddings_are_lib.py: Student scaffold file.
  • examples/what_embeddings_are_lib.py: Complete reference implementation.
  • tests/test_what_embeddings_are_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/what_embeddings_are_lib.py

What the commands do

  • Normalizes sample vectors.
  • Computes cosine similarity and L2 distance.
  • Runs unit test assertions.

Expected output

Embeddings Demo Executed. Cosine Sim: 1.0000, L2 Dist: 0.0000

Validation steps

  1. Verify L2 normalization produces vectors of length 1.0.
  2. Confirm identical vectors produce cosine similarity 1.0.
  3. Validate orthogonal vectors produce cosine similarity 0.0.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Zero division error: Handle zero-magnitude vectors gracefully in normalization functions.

Security notes

All vector arithmetic runs locally in memory.

Extension exercises

  1. Implement a K-Means vector clustering algorithm in pure NumPy.
  2. Build an embedding dimensionality reduction benchmark.
  • Lesson title: What Embeddings Are
  • Day number: 260 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-260-what-embeddings-are
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-260-what-embeddings-are when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 260

- `Embeddings Demo`: Status of embedding similarity execution.

examples-run.txt

Embeddings Demo Executed. Cosine Sim: 1.0000, L2 Dist: 0.0000

measured-values.txt

Embeddings Demo: Success
Cosine Similarity: 1.0000
L2 Distance: 0.0000

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_what_embeddings_are_lib.py::test_normalization_and_cosine_similarity PASSED [ 25%]
tests/test_what_embeddings_are_lib.py::test_document_ranking PASSED                     [ 50%]
tests/test_what_embeddings_are_lib.py::test_euclidean_distance PASSED                   [ 75%]
tests/test_what_embeddings_are_lib.py::test_matryoshka_truncation PASSED                 [100%]

============================== 4 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/test_what_embeddings_are_lib.py (1646 bytes)
import pytest
import numpy as np
from examples.what_embeddings_are_lib import (
    VectorSimilarityEngine
)

def test_normalization_and_cosine_similarity():
    engine = VectorSimilarityEngine(normalize=True)
    v1 = np.array([3.0, 4.0])
    norm_v1 = engine.l2_normalize(v1)
    
    assert np.isclose(np.linalg.norm(norm_v1), 1.0)
    assert np.isclose(norm_v1[0], 0.6)
    assert np.isclose(norm_v1[1], 0.8)

    # Identical vectors should have cosine similarity 1.0
    assert np.isclose(engine.cosine_similarity(v1, v1), 1.0)

    # Orthogonal vectors should have cosine similarity 0.0
    v_ortho = np.array([-4.0, 3.0])
    assert np.isclose(engine.cosine_similarity(v1, v_ortho), 0.0)

def test_document_ranking():
    engine = VectorSimilarityEngine(normalize=True)
    query = np.array([1.0, 0.0, 0.0])
    
    doc0 = np.array([0.9, 0.1, 0.0]) # High similarity
    doc1 = np.array([0.0, 1.0, 0.0]) # Zero similarity (orthogonal)
    doc2 = np.array([0.5, 0.5, 0.0]) # Medium similarity

    ranked = engine.rank_documents(query, [doc0, doc1, doc2])
    assert ranked[0][0] == 0
    assert ranked[1][0] == 2
    assert ranked[2][0] == 1

def test_euclidean_distance():
    engine = VectorSimilarityEngine(normalize=True)
    p1 = np.array([0.0, 0.0])
    p2 = np.array([3.0, 4.0])
    assert np.isclose(engine.euclidean_distance(p1, p2), 5.0)

def test_matryoshka_truncation():
    engine = VectorSimilarityEngine(normalize=True)
    vec = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
    truncated = engine.matryoshka_truncate(vec, target_dim=3)
    
    assert len(truncated) == 3
    assert np.isclose(np.linalg.norm(truncated), 1.0)
examples/what_embeddings_are_lib.py (1594 bytes)
import numpy as np
from typing import List, Tuple

class VectorSimilarityEngine:
    def __init__(self, normalize: bool = True):
        self.normalize = normalize

    def l2_normalize(self, v: np.ndarray) -> np.ndarray:
        norm = np.linalg.norm(v)
        if norm == 0:
            return v
        return v / norm

    def cosine_similarity(self, u: np.ndarray, v: np.ndarray) -> float:
        norm_u = self.l2_normalize(u) if self.normalize else u
        norm_v = self.l2_normalize(v) if self.normalize else v
        return float(np.dot(norm_u, norm_v))

    def euclidean_distance(self, u: np.ndarray, v: np.ndarray) -> float:
        return float(np.linalg.norm(u - v))

    def rank_documents(self, query_vec: np.ndarray, doc_vecs: List[np.ndarray]) -> List[Tuple[int, float]]:
        scores = []
        for idx, doc_vec in enumerate(doc_vecs):
            sim = self.cosine_similarity(query_vec, doc_vec)
            scores.append((idx, sim))
        scores.sort(key=lambda x: x[1], reverse=True)
        return scores

    def matryoshka_truncate(self, v: np.ndarray, target_dim: int) -> np.ndarray:
        truncated = v[:target_dim]
        return self.l2_normalize(truncated)

def run_embeddings_demo():
    engine = VectorSimilarityEngine(normalize=True)
    v1 = np.array([1.0, 2.0, 3.0])
    v2 = np.array([1.0, 2.0, 3.0])
    sim = engine.cosine_similarity(v1, v2)
    dist = engine.euclidean_distance(v1, v2)
    print(f"Embeddings Demo Executed. Cosine Sim: {sim:.4f}, L2 Dist: {dist:.4f}")
    return sim, dist

if __name__ == "__main__":
    run_embeddings_demo()
metadata.yml (432 bytes)
lesson_id: D260
day: 260
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/what_embeddings_are_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_what_embeddings_are_lib.py (1646 bytes)
import pytest
import numpy as np
from examples.what_embeddings_are_lib import (
    VectorSimilarityEngine
)

def test_normalization_and_cosine_similarity():
    engine = VectorSimilarityEngine(normalize=True)
    v1 = np.array([3.0, 4.0])
    norm_v1 = engine.l2_normalize(v1)
    
    assert np.isclose(np.linalg.norm(norm_v1), 1.0)
    assert np.isclose(norm_v1[0], 0.6)
    assert np.isclose(norm_v1[1], 0.8)

    # Identical vectors should have cosine similarity 1.0
    assert np.isclose(engine.cosine_similarity(v1, v1), 1.0)

    # Orthogonal vectors should have cosine similarity 0.0
    v_ortho = np.array([-4.0, 3.0])
    assert np.isclose(engine.cosine_similarity(v1, v_ortho), 0.0)

def test_document_ranking():
    engine = VectorSimilarityEngine(normalize=True)
    query = np.array([1.0, 0.0, 0.0])
    
    doc0 = np.array([0.9, 0.1, 0.0]) # High similarity
    doc1 = np.array([0.0, 1.0, 0.0]) # Zero similarity (orthogonal)
    doc2 = np.array([0.5, 0.5, 0.0]) # Medium similarity

    ranked = engine.rank_documents(query, [doc0, doc1, doc2])
    assert ranked[0][0] == 0
    assert ranked[1][0] == 2
    assert ranked[2][0] == 1

def test_euclidean_distance():
    engine = VectorSimilarityEngine(normalize=True)
    p1 = np.array([0.0, 0.0])
    p2 = np.array([3.0, 4.0])
    assert np.isclose(engine.euclidean_distance(p1, p2), 5.0)

def test_matryoshka_truncation():
    engine = VectorSimilarityEngine(normalize=True)
    vec = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
    truncated = engine.matryoshka_truncate(vec, target_dim=3)
    
    assert len(truncated) == 3
    assert np.isclose(np.linalg.norm(truncated), 1.0)
starter/what_embeddings_are_lib.py (914 bytes)
import numpy as np
from typing import List, Tuple

class VectorSimilarityEngine:
    def __init__(self, normalize: bool = True):
        # TODO: Initialize vector similarity engine
        pass

    def l2_normalize(self, v: np.ndarray) -> np.ndarray:
        # TODO: Normalize vector to unit length (L2 norm = 1)
        pass

    def cosine_similarity(self, u: np.ndarray, v: np.ndarray) -> float:
        # TODO: Compute cosine similarity
        pass

    def euclidean_distance(self, u: np.ndarray, v: np.ndarray) -> float:
        # TODO: Compute Euclidean distance
        pass

    def rank_documents(self, query_vec: np.ndarray, doc_vecs: List[np.ndarray]) -> List[Tuple[int, float]]:
        # TODO: Rank document vectors by descending similarity
        pass

    def matryoshka_truncate(self, v: np.ndarray, target_dim: int) -> np.ndarray:
        # TODO: Truncate vector and re-normalize
        pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 260 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_what_embeddings_are_lib.py (1646 bytes)
import pytest
import numpy as np
from examples.what_embeddings_are_lib import (
    VectorSimilarityEngine
)

def test_normalization_and_cosine_similarity():
    engine = VectorSimilarityEngine(normalize=True)
    v1 = np.array([3.0, 4.0])
    norm_v1 = engine.l2_normalize(v1)
    
    assert np.isclose(np.linalg.norm(norm_v1), 1.0)
    assert np.isclose(norm_v1[0], 0.6)
    assert np.isclose(norm_v1[1], 0.8)

    # Identical vectors should have cosine similarity 1.0
    assert np.isclose(engine.cosine_similarity(v1, v1), 1.0)

    # Orthogonal vectors should have cosine similarity 0.0
    v_ortho = np.array([-4.0, 3.0])
    assert np.isclose(engine.cosine_similarity(v1, v_ortho), 0.0)

def test_document_ranking():
    engine = VectorSimilarityEngine(normalize=True)
    query = np.array([1.0, 0.0, 0.0])
    
    doc0 = np.array([0.9, 0.1, 0.0]) # High similarity
    doc1 = np.array([0.0, 1.0, 0.0]) # Zero similarity (orthogonal)
    doc2 = np.array([0.5, 0.5, 0.0]) # Medium similarity

    ranked = engine.rank_documents(query, [doc0, doc1, doc2])
    assert ranked[0][0] == 0
    assert ranked[1][0] == 2
    assert ranked[2][0] == 1

def test_euclidean_distance():
    engine = VectorSimilarityEngine(normalize=True)
    p1 = np.array([0.0, 0.0])
    p2 = np.array([3.0, 4.0])
    assert np.isclose(engine.euclidean_distance(p1, p2), 5.0)

def test_matryoshka_truncation():
    engine = VectorSimilarityEngine(normalize=True)
    vec = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
    truncated = engine.matryoshka_truncate(vec, target_dim=3)
    
    assert len(truncated) == 3
    assert np.isclose(np.linalg.norm(truncated), 1.0)

Troubleshooting

Troubleshooting: Day 260 - What Embeddings Are

Common Issues

  1. Cosine similarity out of range [-1, 1]:
    • Cause: Numerical floating-point roundoff error in dot products.
    • Fix: Use np.clip(sim, -1.0, 1.0) before downstream arcsin/arccos calculations.

Security notes

Security & Privacy: Day 260 - What Embeddings Are

Security Guidance

  • Restrict access to raw vector embeddings of sensitive personal identifiable information (PII) to prevent embedding inversion extraction attacks.