LLMs and Generative AI β€Ί Embeddings and Vector Search β€Ί Day 260

Day 260: What Embeddings Are

Day 260 of 365 β€” What Embeddings Are

Master vector representations of semantic meaning: understand dense vs. sparse vector spaces, contrast word-level (Word2Vec/GloVe) vs. sentence/passage contextual embeddings (text-embedding-3-small, bge-large-en-v1.5, e5-mistral), analyze dimensionality reduction and cosine distance mathematics in high-dimensional vector spaces, compute similarity dot products, and build a pure NumPy embedding similarity calculator.

Course
LLMs and Generative AI
Category
Embeddings and Vector Search
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-260-what-embeddings-are

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-260-what-embeddings-are
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Computers do not understand language; they operate exclusively on numbers, matrix multiplications, and tensor algebra. If you build an AI application relying strictly on traditional lexical keyword search (such as SQL LIKE queries or raw regex patterns), your system suffers from fundamental blind spots:

Embeddings solve the semantic search problem. By converting words, sentences, and entire technical documents into dense numerical vectors in continuous geometric space, embeddings allow software to quantify semantic meaning, intent, and contextual similarity.

In modern AI engineering:

Today, you will master the mathematics of dense vs. sparse vector spaces, understand how Transformer bi-encoders extract semantic representations, analyze cosine similarity and Matryoshka Representation Learning, and build a pure NumPy embedding similarity engine.


The idea in plain language

Diagram: what a text embedding is and what its geometry means

Think of a massive 3D library where every book is placed at a specific (x, y, z) spatial coordinate:

An embedding is simply the multi-dimensional GPS coordinate of a piece of text. Instead of 3 dimensions, modern AI models use 384, 768, 1536, or 3072 dimensions to capture subtle nuances like tone, domain, syntax, and subject matter.


Historical background

Diagram: the historical progression of text representation methods

Dense vector semantic space projection diagram

  1. 1950s (Distributional Semantics): Linguist J.R. Firth formulated the foundational principle of modern NLP: β€œYou shall know a word by the company it keeps.”
  2. 1970s–1990s (Sparse Vector Space Models & TF-IDF): Gerard Salton pioneered BM25 and TF-IDF, representing documents as massive sparse vectors indexed by dictionary word frequencies.
  3. 2013 (Word2Vec & Dense Vectors): Tomas Mikolov at Google published Word2Vec, demonstrating that predicting surrounding words yields dense vectors with algebraic properties (e.g. Vector("King") - Vector("Man") + Vector("Woman") β‰ˆ Vector("Queen")).
  4. 2019 (Sentence-BERT & Bi-Encoders): Nils Reimers introduced SBERT, fine-tuning Siamese Transformers to generate contextualized sentence-level embeddings in milliseconds.
  5. 2022–2025 (Matryoshka Embeddings & MTEB): Modern models (OpenAI text-embedding-3, BGE-M3, E5-Mistral) introduced nested dimensional truncation and 8k token context windows, standardizing retrieval benchmarks worldwide.

What it is β€” and what it is not

What an Embedding IS:

What it is NOT:


Why it was created and what problems it solves

Dense embeddings solve 4 major shortcomings of classic information retrieval:

  1. Vocabulary Mismatch: Resolves synonyms effortlessly ("attorney" maps close to "lawyer", "automobile" maps close to "car").
  2. Cross-Lingual Retrieval: Maps identical concepts across different languages into adjacent vector clusters ("dog", "perro", "chien").
  3. Contextual Disambiguation: Differentiates polysemous words based on context ("Apple announced new hardware" vs "apple orchard harvest").
  4. Sub-Linear Search Scaling: Enables vector databases to search millions of documents in sub-10ms using geometric graph indexing.

How it works

Let us analyze vector representation mathematics, encoder architectures, and geometric distance metrics.


1. Sparse vs. Dense Representations

Diagram: sparse keyword representations compared with dense learned embeddings

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   SPARSE VS DENSE VECTOR ARCHITECTURE                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Dimension          β”‚ Sparse (BM25 / One-Hot)     β”‚ Dense Embedding     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Dimensionality     β”‚ High (V = 50,000 - 100,000) β”‚ Low (d = 384 - 1536)β”‚
β”‚ Value Type         β”‚ 0.0 or Term Frequency       β”‚ Continuous float32  β”‚
β”‚ Semantic Capture   β”‚ Lexical word match only     β”‚ Latent concept spaceβ”‚
β”‚ Storage Density    β”‚ Highly sparse (> 99.9% 0s)  β”‚ 100% dense non-zerosβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

2. Transformer Bi-Encoder Architecture & Mean Pooling

Diagram: how a transformer produces a single vector for a whole passage

Embedding models evolution and dimensionality architecture diagram

Modern embedding models process text through a Transformer encoder:

  1. Tokenization: Converts input text into subword token IDs [T_1, T_2, ..., T_N].

  2. Self-Attention Layers: Computes all-to-all contextual token vectors of shape (batch, seq_len, hidden_dim).

  3. Mean Pooling: Averages all token vectors across sequence length, masking out padding tokens: v_{\text{pooled}} = \frac{\sum_{i=1}^{N} h_i \cdot \text{mask}_i}{\sum_{i=1}^{N} \text{mask}_i}

  4. L2-Normalization: Scales the output vector so its Euclidean length equals 1.0: v_{\text{norm}} = \frac{v_{\text{pooled}}}{\|v_{\text{pooled}}\|_2}


3. Geometric Distance & Similarity Mathematics

Given two vectors u and v:

A. Cosine Similarity

Measures the angle between two vectors regardless of magnitude: \text{CosineSimilarity}(u, v) = \frac{u \cdot v}{\|u\|_2 \|v\|_2} = \frac{\sum_{i=1}^{d} u_i v_i}{\sqrt{\sum_{i=1}^{d} u_i^2} \sqrt{\sum_{i=1}^{d} v_i^2}}

B. Dot Product on Normalized Vectors

When both u and v are unit-normalized (\|u\| = 1 and \|v\| = 1): \text{CosineSimilarity}(u, v) = u \cdot v = \sum_{i=1}^{d} u_i v_i

C. Euclidean Distance (L2)

Measures straight-line physical distance: d_{\text{L2}}(u, v) = \sqrt{\sum_{i=1}^{d} (u_i - v_i)^2}


4. Matryoshka Representation Learning (MRL)

Diagram: how nested representation training allows embeddings to be truncated

Modern models (such as OpenAI’s text-embedding-3-large and bge-m3) are trained with Matryoshka Loss:


5. Asymmetric Query vs. Passage Prefixes

Diagram: why queries and passages are embedded differently

In retrieval tasks, search queries are short and incomplete ("refund policy"), while target documents are long and explanatory ("Customers are entitled to a full refund within 30 days of purchase...").


6. Vector Quantization & Memory Footprints

Understanding memory consumption for large-scale vector storage:


7. The Curse of Dimensionality & Geometric Density Collapse

In very high-dimensional latent spaces (such as 1024d or 1536d):


8. Evaluating Embedding Models: The MTEB Leaderboard

The Massive Text Embedding Benchmark (MTEB) provides standardized evaluation across 56 diverse datasets and 8 core tasks:


9. Multi-Modal Embeddings: Text-to-Image Alignment

Models like CLIP (Contrastive Language-Image Pretraining) and SigLIP map text and images into the same shared geometric vector space:


10. Summary: The Bedrock of Modern AI

In summary, embeddings translate human language into geometric coordinates, enabling ultra-fast, semantic similarity search, document retrieval, and contextual RAG grounding across enterprise applications. By combining dense bi-encoders, Matryoshka dimension truncation, and quantization, AI platform engineers build scalable, low-latency search systems capable of indexing millions of knowledge documents.


An everyday analogy

Diagram: a list of place names compared with a map showing their positions

Think of a color wheel:


Examples in practice

Let us build a Pure NumPy Vector Similarity Engine:

import numpy as np
from typing import List, Tuple

class VectorSimilarityEngine:
    def __init__(self, normalize: bool = True):
        self.normalize = normalize

    def l2_normalize(self, v: np.ndarray) -> np.ndarray:
        norm = np.linalg.norm(v)
        if norm == 0:
            return v
        return v / norm

    def cosine_similarity(self, u: np.ndarray, v: np.ndarray) -> float:
        norm_u = self.l2_normalize(u) if self.normalize else u
        norm_v = self.l2_normalize(v) if self.normalize else v
        return float(np.dot(norm_u, norm_v))

    def euclidean_distance(self, u: np.ndarray, v: np.ndarray) -> float:
        return float(np.linalg.norm(u - v))

    def rank_documents(self, query_vec: np.ndarray, doc_vecs: List[np.ndarray]) -> List[Tuple[int, float]]:
        # Ranks documents by descending cosine similarity
        scores = []
        for idx, doc_vec in enumerate(doc_vecs):
            sim = self.cosine_similarity(query_vec, doc_vec)
            scores.append((idx, sim))
        scores.sort(key=lambda x: x[1], reverse=True)
        return scores

Implications: security, privacy, performance, scalability, and cost

Diagram: the cost, migration, privacy and staleness implications of embeddings

  1. Embedding Inversion & Privacy:
    • Embeddings are not one-way encryption hashes; researchers have demonstrated that original text can be partially reconstructed from dense vectors. Never publish raw embedding vectors of sensitive user data without access controls.
  2. Computational Scaling:
    • Performing exact dot products against 100,000 vectors takes under 2ms in NumPy with vectorization, making exact k-NN extremely practical for moderate dataset sizes before needing dedicated vector databases.

Alternatives: free, open source, and commercial

ModelProviderDimensionsContext WindowBest For
bge-large-en-v1.5Open Source (HuggingFace)1024512 tokensSelf-hosted local RAG
all-MiniLM-L6-v2Open Source (Sentence-Transformers)384256 tokensUltra-fast lightweight search
text-embedding-3-smallOpenAI API1536 (MRL to 512)8191 tokensGeneral-purpose cloud RAG
voyage-3Voyage AI102432000 tokensSpecialized domain retrieval

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   RETRIEVAL REPRESENTATION COMPARISON                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Metric             β”‚ Exact Keyword   β”‚ Sparse (BM25)    β”‚ Dense Vector β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Synonym Handling   β”‚ None            β”‚ None             β”‚ Outstanding  β”‚
β”‚ Exact Code Matches β”‚ Perfect         β”‚ Excellent        β”‚ Moderate     β”‚
β”‚ Compute Overhead   β”‚ Zero            β”‚ Negligible       β”‚ Transformer  β”‚
β”‚ Storage Size       β”‚ Raw Text        β”‚ Inverted Index   β”‚ Float Tensorsβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when dense embeddings are appropriate and when keyword search suffices

When to Use Dense Embeddings:

When NOT to use:


The AI connection

Knowledge check

  1. What is the difference between a dense vector and a sparse vector representation?
  2. Why is cosine similarity equivalent to the dot product when vectors are L2-normalized?
  3. How does Matryoshka Representation Learning (MRL) reduce vector storage costs?
  4. What is mean pooling in Transformer sentence encoders?
  5. Why is an embedding vector not a secure one-way cryptographic hash?

Hands-on exercise

In this lab, you will build and test a Pure NumPy Vector Similarity Engine: implement L2 vector normalization, compute dot products, cosine similarities, and Euclidean distances, rank candidate document vectors by descending similarity, apply Matryoshka dimension truncation, and verify output assertions against automated test suites.

Expected output

[What Embeddings Are Verification Suite]
Testing Vector Normalization & Dot Product:
  Vector L2 Norm: 1.0000 [PASS]
  Identical Vectors Cosine Similarity: 1.0000 [PASS]
  Orthogonal Vectors Cosine Similarity: 0.0000 [PASS]
Testing Multi-Document Semantic Ranking:
  Top-1 Match: Doc 0 (sim: 0.892) [PASS]
  Top-2 Match: Doc 2 (sim: 0.654) [PASS]
Testing Matryoshka Dimension Truncation (1536d -> 512d):
  Truncated Vector Retains Relative Rank Order: VERIFIED [PASS]
Test Suite: 4 passed in 0.38s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement an Embedding Clustering Pipeline using NumPy that partitions 50 vector representations into 5 semantic topic clusters using K-Means.
  2. Build a Cross-Encoder vs. Bi-Encoder Latency Benchmark comparing pairwise similarity calculation times across 1,000 document pairs.

Extension challenge

Implement a Matryoshka Truncation Efficiency Evaluator:

Quiz

Q1. What is the fundamental difference between sparse lexical search (e.g. BM25) and dense vector embeddings?

  1. There is no mathematical difference between them
  2. Dense embeddings only work with images, while sparse search works with text
  3. Sparse search requires GPUs, while dense embeddings run on floppy disks
  4. Sparse matches keywords; dense maps meaning to geometry
Show answer

Answer: D. Sparse matches keywords; dense maps meaning to geometry

Dense vectors capture semantic synonyms and conceptual relationships regardless of lexical word overlap.

Q2. Why does polysemy (e.g. "apple" the fruit vs. "Apple" the corporation) break static word embeddings like Word2Vec?

  1. Word2Vec gives one vector per token, whatever the context
  2. Because Word2Vec cannot spell English words
  3. Because Word2Vec is limited to 10 tokens
  4. Because Apple sued Word2Vec
Show answer

Answer: A. Word2Vec gives one vector per token, whatever the context

Static embeddings lack context awareness, collapsing multiple meanings into a single vector centroid.

Q3. If two dense vectors u and v are L2-normalized such that ||u|| = 1 and ||v|| = 1, how does cosine similarity simplify?

  1. It equals zero
  2. It becomes exactly the dot product
  3. It requires computing an inverted matrix determinant
  4. It becomes undefined
Show answer

Answer: B. It becomes exactly the dot product

On unit-normalized vectors, the denominator ||u|| * ||v|| equals 1, making cosine similarity equivalent to the dot product.

Q4. What is Matryoshka Representation Learning (MRL) in modern embedding models like text-embedding-3?

  1. Compressing text into MP3 audio files
  2. Translating text into Russian doll jokes
  3. Early dimensions hold the meaning, so it truncates safely
  4. Converting floating point numbers to integers
Show answer

Answer: C. Early dimensions hold the meaning, so it truncates safely

MRL structures representations hierarchically, allowing developers to truncate vector lengths while preserving retrieval accuracy.

Q5. Why do asymmetric embedding models (such as E5 and BGE) require prefix markers like "query: " and "passage: "?

  1. To slow down GPU execution
  2. Because Python requires strings to start with letters
  3. To prevent database SQL injection
  4. Queries and passages differ, so a prefix aligns them
Show answer

Answer: D. Queries and passages differ, so a prefix aligns them

Asymmetric prefixes condition the encoder to align short intent queries with detailed factual passages.

Glossary

Embedding
A continuous, dense vector representation of discrete data (text, image, audio) in high-dimensional geometric space.
Dense Vector
A numerical tensor where most values are non-zero floating-point numbers representing latent semantic features.
Sparse Vector
A high-dimensional vector (matching vocabulary size) where nearly all elements are zero except for specific word frequencies.
Cosine Similarity
A metric measuring the cosine of the angle between two vectors, ranging from -1.0 to +1.0 regardless of vector magnitude.
Dot Product
The sum of the products of corresponding elements of two vectors; equivalent to cosine similarity on unit-normalized vectors.
Bi-Encoder
A neural architecture that processes queries and documents independently through a Transformer to generate separate vector embeddings.
Matryoshka Representation Learning (MRL)
A training technique enforcing semantic information density in leading vector dimensions, permitting lossless vector truncation.
Mean Pooling
Aggregating token-level hidden states from a Transformer by taking the average across sequence length weighted by the attention mask.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.