LLMs and Generative AI βΊ Embeddings and Vector Search βΊ Day 260
Day 260: What Embeddings Are
Master vector representations of semantic meaning: understand dense vs. sparse vector spaces, contrast word-level (Word2Vec/GloVe) vs. sentence/passage contextual embeddings (text-embedding-3-small, bge-large-en-v1.5, e5-mistral), analyze dimensionality reduction and cosine distance mathematics in high-dimensional vector spaces, compute similarity dot products, and build a pure NumPy embedding similarity calculator.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-260-what-embeddings-are
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/llms-and-generative-ai/day-260-what-embeddings-are - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Differentiate between sparse lexical representations (BM25/One-Hot) and dense continuous semantic embeddings.
- Explain how Transformer bi-encoders generate fixed-length contextual sentence vectors.
- Calculate dot product, cosine similarity, and Euclidean distance on normalized vectors in NumPy.
- Understand Matryoshka Representation Learning (MRL) and dimension truncation mechanics.
- Build a standalone embedding generation and similarity comparison engine in Python.
Prerequisites
Why this matters
Computers do not understand language; they operate exclusively on numbers, matrix multiplications, and tensor algebra.
If you build an AI application relying strictly on traditional lexical keyword search (such as SQL LIKE queries or raw regex patterns), your system suffers from fundamental blind spots:
- A user searching for
"cardiac arrest symptoms"will completely miss an authoritative medical document titled"Myocardial Infarction Clinical Presentation", because they share zero words in common. - A search for
"cheap flights to Paris"will match documents discussing βun-cheap hotel rooms near Parisβ, returning irrelevant noise.
Embeddings solve the semantic search problem. By converting words, sentences, and entire technical documents into dense numerical vectors in continuous geometric space, embeddings allow software to quantify semantic meaning, intent, and contextual similarity.
In modern AI engineering:
- Embeddings are the foundation of Retrieval-Augmented Generation (RAG): They retrieve the exact documentation paragraphs needed to ground language model answers.
- Embeddings power recommendation engines, clustering pipelines, anomaly detectors, and classification systems.
Today, you will master the mathematics of dense vs. sparse vector spaces, understand how Transformer bi-encoders extract semantic representations, analyze cosine similarity and Matryoshka Representation Learning, and build a pure NumPy embedding similarity engine.
The idea in plain language
Think of a massive 3D library where every book is placed at a specific (x, y, z) spatial coordinate:
- In the Animal Section:
- A book about
"golden retrievers"sits at coordinate(10.2, 5.1, 8.4). - A book about
"puppy training"sits at coordinate(10.1, 5.3, 8.2). - Because their spatial distance is tiny (less than 0.3 units), the librarian knows they discuss nearly identical concepts without reading a single sentence (High Semantic Similarity).
- A book about
- In the Culinary Section:
- A book about
"baking sourdough pizza"sits at coordinate(-45.0, 88.2, -12.1). - The physical distance between the puppy book and the pizza book is massive (Low Semantic Similarity).
- A book about
An embedding is simply the multi-dimensional GPS coordinate of a piece of text. Instead of 3 dimensions, modern AI models use 384, 768, 1536, or 3072 dimensions to capture subtle nuances like tone, domain, syntax, and subject matter.
Historical background
- 1950s (Distributional Semantics): Linguist J.R. Firth formulated the foundational principle of modern NLP: βYou shall know a word by the company it keeps.β
- 1970sβ1990s (Sparse Vector Space Models & TF-IDF): Gerard Salton pioneered BM25 and TF-IDF, representing documents as massive sparse vectors indexed by dictionary word frequencies.
- 2013 (Word2Vec & Dense Vectors): Tomas Mikolov at Google published Word2Vec, demonstrating that predicting surrounding words yields dense vectors with algebraic properties (e.g.
Vector("King") - Vector("Man") + Vector("Woman") β Vector("Queen")). - 2019 (Sentence-BERT & Bi-Encoders): Nils Reimers introduced SBERT, fine-tuning Siamese Transformers to generate contextualized sentence-level embeddings in milliseconds.
- 2022β2025 (Matryoshka Embeddings & MTEB): Modern models (OpenAI
text-embedding-3, BGE-M3, E5-Mistral) introduced nested dimensional truncation and 8k token context windows, standardizing retrieval benchmarks worldwide.
What it is β and what it is not
What an Embedding IS:
- A Dense Continuous Vector Representation: A fixed-length 1D tensor of floating-point numbers (e.g. 1536 floats) encoding the latent semantic meaning of input text.
What it is NOT:
- Not Hash Generation: Unlike cryptographic MD5 or SHA-256 hashes (where changing a single comma changes the hash completely), embeddings are continuous and smooth: small semantic edits result in small geometric vector shifts.
Why it was created and what problems it solves
Dense embeddings solve 4 major shortcomings of classic information retrieval:
- Vocabulary Mismatch: Resolves synonyms effortlessly (
"attorney"maps close to"lawyer","automobile"maps close to"car"). - Cross-Lingual Retrieval: Maps identical concepts across different languages into adjacent vector clusters (
"dog","perro","chien"). - Contextual Disambiguation: Differentiates polysemous words based on context (
"Apple announced new hardware"vs"apple orchard harvest"). - Sub-Linear Search Scaling: Enables vector databases to search millions of documents in sub-10ms using geometric graph indexing.
How it works
Let us analyze vector representation mathematics, encoder architectures, and geometric distance metrics.
1. Sparse vs. Dense Representations
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SPARSE VS DENSE VECTOR ARCHITECTURE β
ββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββ€
β Dimension β Sparse (BM25 / One-Hot) β Dense Embedding β
ββββββββββββββββββββββΌββββββββββββββββββββββββββββββΌββββββββββββββββββββββ€
β Dimensionality β High (V = 50,000 - 100,000) β Low (d = 384 - 1536)β
β Value Type β 0.0 or Term Frequency β Continuous float32 β
β Semantic Capture β Lexical word match only β Latent concept spaceβ
β Storage Density β Highly sparse (> 99.9% 0s) β 100% dense non-zerosβ
ββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββ
- Furthermore, sparse models cannot distinguish between semantically equivalent phrases that use completely disjoint vocabularies (such as
"cardiovascular issues"versus"heart trouble"). - In contrast, dense embedding models project these synonymous concepts into adjacent regions of latent space, enabling software systems to capture deep semantic intent without requiring manual keyword synonym expansion dictionaries.
2. Transformer Bi-Encoder Architecture & Mean Pooling
Modern embedding models process text through a Transformer encoder:
-
Tokenization: Converts input text into subword token IDs
[T_1, T_2, ..., T_N]. -
Self-Attention Layers: Computes all-to-all contextual token vectors of shape
(batch, seq_len, hidden_dim). -
Mean Pooling: Averages all token vectors across sequence length, masking out padding tokens:
v_{\text{pooled}} = \frac{\sum_{i=1}^{N} h_i \cdot \text{mask}_i}{\sum_{i=1}^{N} \text{mask}_i} -
L2-Normalization: Scales the output vector so its Euclidean length equals 1.0:
v_{\text{norm}} = \frac{v_{\text{pooled}}}{\|v_{\text{pooled}}\|_2}
3. Geometric Distance & Similarity Mathematics
Given two vectors u and v:
A. Cosine Similarity
Measures the angle between two vectors regardless of magnitude:
\text{CosineSimilarity}(u, v) = \frac{u \cdot v}{\|u\|_2 \|v\|_2} = \frac{\sum_{i=1}^{d} u_i v_i}{\sqrt{\sum_{i=1}^{d} u_i^2} \sqrt{\sum_{i=1}^{d} v_i^2}}
B. Dot Product on Normalized Vectors
When both u and v are unit-normalized (\|u\| = 1 and \|v\| = 1):
\text{CosineSimilarity}(u, v) = u \cdot v = \sum_{i=1}^{d} u_i v_i
- Computing a dot product is 5x faster than full cosine similarity because it avoids square root operations during search!
C. Euclidean Distance (L2)
Measures straight-line physical distance:
d_{\text{L2}}(u, v) = \sqrt{\sum_{i=1}^{d} (u_i - v_i)^2}
- On normalized vectors, Euclidean distance is monotonically related to cosine similarity:
d_{\text{L2}}^2 = 2 - 2 \cdot \text{CosineSimilarity}(u, v)
4. Matryoshka Representation Learning (MRL)
Modern models (such as OpenAIβs text-embedding-3-large and bge-m3) are trained with Matryoshka Loss:
-
The model forces the most critical semantic information into the first
kdimensions (e.g. first 256 or 512 dimensions). -
You can slice a 1536-dimensional vector down to 512 dimensions:
truncated_vec = full_vec[:512] normalized_truncated = truncated_vec / np.linalg.norm(truncated_vec) -
Benefit: Slashes vector database memory and disk storage by 66% with less than 1.5% loss in retrieval accuracy!
-
This multi-resolution capability allows applications to perform lightning-fast initial candidate filtering using low-dimensional 256d vectors before performing final ranking with full-resolution embeddings.
5. Asymmetric Query vs. Passage Prefixes
In retrieval tasks, search queries are short and incomplete ("refund policy"), while target documents are long and explanatory ("Customers are entitled to a full refund within 30 days of purchase...").
- Models like E5 and BGE require asymmetric prefix prompts:
- Query:
query: refund policy - Passage:
passage: Customers are entitled to a full refund within 30 days...
- Query:
- The prefix conditions the Transformer attention mechanism to align query intent with factual answers.
6. Vector Quantization & Memory Footprints
Understanding memory consumption for large-scale vector storage:
-
A 1536-dimensional
float32vector requires:1536 * 4 bytes = 6144 bytes = 6.14 KB -
Storing 1,000,000 document vectors in memory requires 6.14 GB of raw RAM.
-
Scalar Quantization (SQ8): Compresses each 32-bit float into an 8-bit integer (
int8), reducing RAM to 1.53 GB (4x reduction) while retaining 99% of retrieval precision. -
Product Quantization (PQ): Decomposes vectors into sub-vectors and assigns them to centroid codebooks, slashing memory to 192 bytes per vector (32x compression), enabling billion-scale search on a single node.
7. The Curse of Dimensionality & Geometric Density Collapse
In very high-dimensional latent spaces (such as 1024d or 1536d):
- Random orthogonal vectors have an expected cosine similarity of
0.0. - However, because natural language models are trained on cohesive human corpora, actual text embeddings concentrate within a narrow geometric cone.
- As a consequence, cosine similarity scores for text typically fall within a tight band between
0.60and0.92. - Engineering Invariant: Never set an arbitrary static score threshold (e.g.
similarity > 0.95) to determine whether a document is relevant; always use relative ranking (Top-K) combined with cross-encoder rerankers.
8. Evaluating Embedding Models: The MTEB Leaderboard
The Massive Text Embedding Benchmark (MTEB) provides standardized evaluation across 56 diverse datasets and 8 core tasks:
- Retrieval: Evaluating NDCG@10 on MS MARCO, HotpotQA, and NQ.
- Reranking: Measuring Mean Average Precision across candidate sets.
- Clustering: Measuring V-Measure and adjusted Rand Index on thematic document groupings.
- Semantic Textual Similarity (STS): Measuring Spearman rank correlation against human similarity annotations.
- Reviewing MTEB leaderboards helps teams select the optimal balance between vector dimension, parameter count, and domain accuracy.
9. Multi-Modal Embeddings: Text-to-Image Alignment
Models like CLIP (Contrastive Language-Image Pretraining) and SigLIP map text and images into the same shared geometric vector space:
- The vector representation of the text string
"a red sports car parked near the beach"is geometrically close to the vector generated by a Vision Transformer looking at a photograph of a red convertible on the coast. - This shared representation allows semantic search engines to search millions of images using natural language text queries with zero manual tagging.
10. Summary: The Bedrock of Modern AI
In summary, embeddings translate human language into geometric coordinates, enabling ultra-fast, semantic similarity search, document retrieval, and contextual RAG grounding across enterprise applications. By combining dense bi-encoders, Matryoshka dimension truncation, and quantization, AI platform engineers build scalable, low-latency search systems capable of indexing millions of knowledge documents.
An everyday analogy
Think of a color wheel:
- If you represent colors as text names (
"crimson","ruby","scarlet","navy","cobalt"), a computer sees completely different words with zero common letters. - If you represent colors as numerical RGB vectors:
"crimson"=(220, 20, 60)"ruby"=(224, 17, 95)"navy"=(0, 0, 128)
- A computer can immediately calculate that crimson and ruby are separated by only a tiny mathematical distance, while navy is far away (Numerical Representation of Perceptual Meaning).
Examples in practice
Let us build a Pure NumPy Vector Similarity Engine:
import numpy as np
from typing import List, Tuple
class VectorSimilarityEngine:
def __init__(self, normalize: bool = True):
self.normalize = normalize
def l2_normalize(self, v: np.ndarray) -> np.ndarray:
norm = np.linalg.norm(v)
if norm == 0:
return v
return v / norm
def cosine_similarity(self, u: np.ndarray, v: np.ndarray) -> float:
norm_u = self.l2_normalize(u) if self.normalize else u
norm_v = self.l2_normalize(v) if self.normalize else v
return float(np.dot(norm_u, norm_v))
def euclidean_distance(self, u: np.ndarray, v: np.ndarray) -> float:
return float(np.linalg.norm(u - v))
def rank_documents(self, query_vec: np.ndarray, doc_vecs: List[np.ndarray]) -> List[Tuple[int, float]]:
# Ranks documents by descending cosine similarity
scores = []
for idx, doc_vec in enumerate(doc_vecs):
sim = self.cosine_similarity(query_vec, doc_vec)
scores.append((idx, sim))
scores.sort(key=lambda x: x[1], reverse=True)
return scores
Implications: security, privacy, performance, scalability, and cost
- Embedding Inversion & Privacy:
- Embeddings are not one-way encryption hashes; researchers have demonstrated that original text can be partially reconstructed from dense vectors. Never publish raw embedding vectors of sensitive user data without access controls.
- Computational Scaling:
- Performing exact dot products against 100,000 vectors takes under 2ms in NumPy with vectorization, making exact k-NN extremely practical for moderate dataset sizes before needing dedicated vector databases.
Alternatives: free, open source, and commercial
| Model | Provider | Dimensions | Context Window | Best For |
|---|---|---|---|---|
| bge-large-en-v1.5 | Open Source (HuggingFace) | 1024 | 512 tokens | Self-hosted local RAG |
| all-MiniLM-L6-v2 | Open Source (Sentence-Transformers) | 384 | 256 tokens | Ultra-fast lightweight search |
| text-embedding-3-small | OpenAI API | 1536 (MRL to 512) | 8191 tokens | General-purpose cloud RAG |
| voyage-3 | Voyage AI | 1024 | 32000 tokens | Specialized domain retrieval |
Comparison with related concepts
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β RETRIEVAL REPRESENTATION COMPARISON β
ββββββββββββββββββββββ¬ββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββ€
β Metric β Exact Keyword β Sparse (BM25) β Dense Vector β
ββββββββββββββββββββββΌββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββ€
β Synonym Handling β None β None β Outstanding β
β Exact Code Matches β Perfect β Excellent β Moderate β
β Compute Overhead β Zero β Negligible β Transformer β
β Storage Size β Raw Text β Inverted Index β Float Tensorsβ
ββββββββββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββ
When to use it β and when not to
When to Use Dense Embeddings:
- Semantic document search, customer question answering, cross-lingual knowledge retrieval, document recommendation, and vector-based deduplication.
When NOT to use:
- Searching for exact arbitrary strings like alphanumeric serial numbers, product SKUs, git commit hashes, or specific variable names where exact keyword matching (BM25) is superior.
The AI connection
- Embeddings are how meaning becomes arithmetic β they are what makes semantic search, clustering and retrieval possible at all.
- Every retrieval system rests on them, which makes the embedding model a more consequential choice than the language model in many applications.
- Distance in the vector space stands in for similarity in meaning, which is a useful approximation and not an identity.
- Dimensionality is a cost and quality trade β shorter vectors are cheaper to store and search, and lose some discrimination.
- The same technique aligns text with images, which is what makes cross-modal search work with no shared vocabulary.
Knowledge check
- What is the difference between a dense vector and a sparse vector representation?
- Why is cosine similarity equivalent to the dot product when vectors are L2-normalized?
- How does Matryoshka Representation Learning (MRL) reduce vector storage costs?
- What is mean pooling in Transformer sentence encoders?
- Why is an embedding vector not a secure one-way cryptographic hash?
Hands-on exercise
In this lab, you will build and test a Pure NumPy Vector Similarity Engine: implement L2 vector normalization, compute dot products, cosine similarities, and Euclidean distances, rank candidate document vectors by descending similarity, apply Matryoshka dimension truncation, and verify output assertions against automated test suites.
Expected output
[What Embeddings Are Verification Suite]
Testing Vector Normalization & Dot Product:
Vector L2 Norm: 1.0000 [PASS]
Identical Vectors Cosine Similarity: 1.0000 [PASS]
Orthogonal Vectors Cosine Similarity: 0.0000 [PASS]
Testing Multi-Document Semantic Ranking:
Top-1 Match: Doc 0 (sim: 0.892) [PASS]
Top-2 Match: Doc 2 (sim: 0.654) [PASS]
Testing Matryoshka Dimension Truncation (1536d -> 512d):
Truncated Vector Retains Relative Rank Order: VERIFIED [PASS]
Test Suite: 4 passed in 0.38s
Validate your work
Run the automated test runner:
./tests/run_tests.sh
Troubleshooting
- Ensure vectors are normalized with
np.linalg.norm(v)before calculating dot products for cosine similarity.
Common mistakes
- Omitting the denominator in non-normalized cosine similarity: If vectors are not unit-normalized,
dot(u, v)yields unscaled magnitude rather than cosine similarity.
Practice assignment
- Implement an Embedding Clustering Pipeline using NumPy that partitions 50 vector representations into 5 semantic topic clusters using K-Means.
- Build a Cross-Encoder vs. Bi-Encoder Latency Benchmark comparing pairwise similarity calculation times across 1,000 document pairs.
Extension challenge
Implement a Matryoshka Truncation Efficiency Evaluator:
- Take a collection of 1536-dimensional vectors, compute Top-10 nearest neighbors for 50 test queries at full 1536d resolution, repeat the search at truncated 512d, 256d, and 128d resolutions, and plot the Recall@10 Retention Curve vs. Dimensionality Reduction Ratio.
Quiz
Q1. What is the fundamental difference between sparse lexical search (e.g. BM25) and dense vector embeddings?
- There is no mathematical difference between them
- Dense embeddings only work with images, while sparse search works with text
- Sparse search requires GPUs, while dense embeddings run on floppy disks
- Sparse matches keywords; dense maps meaning to geometry
Show answer
Answer: D. Sparse matches keywords; dense maps meaning to geometry
Dense vectors capture semantic synonyms and conceptual relationships regardless of lexical word overlap.
Q2. Why does polysemy (e.g. "apple" the fruit vs. "Apple" the corporation) break static word embeddings like Word2Vec?
- Word2Vec gives one vector per token, whatever the context
- Because Word2Vec cannot spell English words
- Because Word2Vec is limited to 10 tokens
- Because Apple sued Word2Vec
Show answer
Answer: A. Word2Vec gives one vector per token, whatever the context
Static embeddings lack context awareness, collapsing multiple meanings into a single vector centroid.
Q3. If two dense vectors u and v are L2-normalized such that ||u|| = 1 and ||v|| = 1, how does cosine similarity simplify?
- It equals zero
- It becomes exactly the dot product
- It requires computing an inverted matrix determinant
- It becomes undefined
Show answer
Answer: B. It becomes exactly the dot product
On unit-normalized vectors, the denominator ||u|| * ||v|| equals 1, making cosine similarity equivalent to the dot product.
Q4. What is Matryoshka Representation Learning (MRL) in modern embedding models like text-embedding-3?
- Compressing text into MP3 audio files
- Translating text into Russian doll jokes
- Early dimensions hold the meaning, so it truncates safely
- Converting floating point numbers to integers
Show answer
Answer: C. Early dimensions hold the meaning, so it truncates safely
MRL structures representations hierarchically, allowing developers to truncate vector lengths while preserving retrieval accuracy.
Q5. Why do asymmetric embedding models (such as E5 and BGE) require prefix markers like "query: " and "passage: "?
- To slow down GPU execution
- Because Python requires strings to start with letters
- To prevent database SQL injection
- Queries and passages differ, so a prefix aligns them
Show answer
Answer: D. Queries and passages differ, so a prefix aligns them
Asymmetric prefixes condition the encoder to align short intent queries with detailed factual passages.
Glossary
- Embedding
- A continuous, dense vector representation of discrete data (text, image, audio) in high-dimensional geometric space.
- Dense Vector
- A numerical tensor where most values are non-zero floating-point numbers representing latent semantic features.
- Sparse Vector
- A high-dimensional vector (matching vocabulary size) where nearly all elements are zero except for specific word frequencies.
- Cosine Similarity
- A metric measuring the cosine of the angle between two vectors, ranging from -1.0 to +1.0 regardless of vector magnitude.
- Dot Product
- The sum of the products of corresponding elements of two vectors; equivalent to cosine similarity on unit-normalized vectors.
- Bi-Encoder
- A neural architecture that processes queries and documents independently through a Transformer to generate separate vector embeddings.
- Matryoshka Representation Learning (MRL)
- A training technique enforcing semantic information density in leading vector dimensions, permitting lossless vector truncation.
- Mean Pooling
- Aggregating token-level hidden states from a Transformer by taking the average across sequence length weighted by the attention mask.
Sources and further reading
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks β EMNLP / ACL Anthology (accessed 2026-08-30)
- Matryoshka Representation Learning β NeurIPS Proceedings (accessed 2026-08-30)
- MTEB: Massive Text Embedding Benchmark β EACL Proceedings (accessed 2026-08-30)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.