Deep Learning βΊ Sequences and Text βΊ Day 219
Day 219: Word Embeddings
Master dense distributed representations of language: derive Word2Vec (CBOW and Skip-Gram), implement Negative Sampling loss from scratch, understand GloVe co-occurrence matrix factorization and FastText subword n-grams, compute cosine similarity, and execute semantic vector arithmetic in PyTorch.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-219-word-embeddings
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/deep-learning/day-219-word-embeddings - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Derive the mathematical limitations of one-hot sparse vectors and the distributional semantic hypothesis.
- Implement the Word2Vec Skip-Gram objective with Negative Sampling loss in pure PyTorch.
- Compare the foundational trade-offs between Word2Vec, GloVe (Global Vectors), and FastText.
- Perform linear vector analogy arithmetic and nearest-neighbor cosine similarity queries.
- Build and initialize a PyTorch nn.Embedding lookup table with custom and pretrained weight tensors.
Prerequisites
Why this matters
Yesterday in Day 218, we mastered tokenization: transforming raw human text strings into discrete integer token IDs.
However, an integer token ID (like token_id = 4937) contains zero semantic meaning. In the eyes of a mathematical neural network:
4937is not semantically closer to4938than it is to102.- If we represent words as One-Hot Encoded Vectors (a vector of length $|V|$ with a single
1and zeros elsewhere), every single word vector is strictly orthogonal to every other word vector:dot_product(one_hot("cat"), one_hot("kitten")) = 0.0 dot_product(one_hot("cat"), one_hot("refrigerator")) = 0.0
One-hot encodings are computationally massive, sparse, and completely blind to semantic relationships, synonyms, and grammatical hierarchies.
To solve this, deep learning represents words as Dense Word Embeddings: continuous, low-dimensional vectors (typically $d = 100$ to $1024$ dimensions) where spatial distance and geometric direction encode genuine semantic meaning.
In this lesson, you will master the foundational algorithms of vector semantics: Word2Vec (CBOW & Skip-Gram), Negative Sampling, GloVe, FastText, and the famous geometric analogies ($v_(\text(king)) - v_(\text(man)) + v_(\text(woman)) \approx v_(\text(queen))$).
The idea in plain language
Imagine mapping every concept in the human universe onto a multi-dimensional globe:
- Instead of cataloging words alphabetically (
A, B, C...), you place words on the globe based on their characteristics. - Animals cluster in one continent. Domestic pets live close together:
"cat"and"kitten"sit 2 miles apart on the map, while"refrigerator"sits on another continent 5,000 miles away. - Furthermore, the direction and distance between locations represent consistent concepts:
- Moving 50 miles North means βmake it femaleβ (
"man"$\rightarrow$"woman","king"$\rightarrow$"queen"). - Moving 100 miles East means βconvert capital city to its countryβ (
"Paris"$\rightarrow$"France","Tokyo"$\rightarrow$"Japan").
- Moving 50 miles North means βmake it femaleβ (
A Word Embedding is simply the GPS coordinate $(x_1, x_2, \dots, x_d)$ of a word in this semantic universe.
Historical background
- 1957 (The Distributional Hypothesis): British linguist J.R. Firth proposed the foundational maxim of computational semantics: βYou shall know a word by the company it keeps.β Words that share similar surrounding context words share similar meanings.
- 2003 (Bengioβs Neural Language Model): Yoshua Bengio et al. introduced distributed representations of words as learned continuous lookup tables in neural networks.
- 2013 (Tomas Mikolov & Word2Vec at Google): Tomas Mikolov introduced Word2Vec (CBOW and Skip-Gram with Negative Sampling), drastically reducing training complexity and discovering that vector space offsets naturally encoded semantic analogies.
- 2014 (GloVe at Stanford): Jeffrey Pennington, Richard Socher, and Chris Manning introduced GloVe (Global Vectors), combining the advantages of global matrix factorization with local context window architectures.
- 2017 (FastText at Facebook AI): Piotr Bojanowski et al. extended Word2Vec with subword character n-grams, enabling embeddings for out-of-vocabulary and morphologically rich words.
What it is β and what it is not
What Word Embeddings ARE:
- Dense Distributed Semantic Vectors: Low-dimensional real-valued vectors ($R^d$) where cosine similarity reflects semantic relatedness.
- Differentiable Lookup Tables: In PyTorch, an
nn.Embedding(num_embeddings, embedding_dim)layer is simply a weight matrix $W \in R^(|V| \times d)$ indexed by token integers.
What they are NOT:
- Not Context-Dependent (Static vs Dynamic): Classical embeddings (Word2Vec, GloVe) assign a single static vector to each word. The word
"bank"has the exact same vector in"river bank"and"investment bank". (Dynamic contextual embeddings are created by Transformers like BERT/GPT, which we study in Week 33). - Not Random Hash Vectors: Embeddings are learned through statistical co-occurrence optimization over billions of text tokens.
Why it was created and what problems it solves
Word embeddings resolved four critical bottlenecks in NLP:
- Curse of Dimensionality: Replacing 1,000,000-dimensional sparse one-hot vectors with compact 300-dimensional dense vectors.
- Semantic Orthogonality: Providing a continuous distance metric (Cosine Similarity) where synonyms yield high dot products.
- Generalization Across Rare Words: A sentiment model trained on
"The food was delicious"automatically understands"The food was scrumptious"because their vectors lie adjacent in embedding space. - Linear Semantic Reasoning: Enabling vector arithmetic across analogies, gender, pluralization, and country-capital relationships.
How it works
Let us dissect the mathematical architectures of Word2Vec, GloVe, and FastText.
1. Word2Vec: Continuous Bag-of-Words (CBOW)
In CBOW, given a context window of size $C$ around center word $w_t$:
Context: [w_{t-2}, w_{t-1}, w_{t+1}, w_{t+2}] -> Predict Target: w_t
-
Look up input embedding vectors for all context words: $v_(c) \in R^d$.
-
Compute the average context vector:
v_context = (1 / 2C) * sum_{j \in context} v_{w_j} -
Project $v_(\text(context))$ through an output weight matrix $W_(\text(out)) \in R^(d \times |V|)$ to compute logits across all vocabulary words.
-
Apply Softmax to predict the probability of center word $w_t$.
2. Word2Vec: Continuous Skip-Gram with Negative Sampling (SGNS)
In Skip-Gram, the task is inverted: given center word $w_t$, predict each surrounding context word $w_c$:
The Exact Softmax Bottleneck:
The true conditional probability requires normalizing over the entire vocabulary:
P(w_c | w_t) = exp(u_{w_c}^\top v_{w_t}) / sum_{w \in V} exp(u_w^\top v_{w_t})
Summing over $|V| = 500,000$ words for every token update is computationally intractable ($O(|V|)$ per step).
The Negative Sampling Solution:
Mikolov et al. converted the multi-class prediction into multiple binary logistic regressions:
- For each true positive pair $(w_t, w_c)$, sample $K$ random negative noise words $(w_t, w_(n_1)), \dots, (w_t, w_(n_K))$ from the unigram distribution $P_n(w) \propto \text(Freq)(w)^(3/4)$.
- Negative Sampling Loss Function:
L = - log(sigma(u_{w_c}^\top v_{w_t})) - sum_{k=1}^K log(sigma(- u_{w_{n_k}}^\top v_{w_t}))
where $\sigma(z) = \frac(1)(1 + e^(-z))$ is the sigmoid function. This reduces complexity from $O(|V|)$ to $O(K)$, where $K \in [5, 20]$.
3. Hierarchical Softmax Alternative
Prior to Negative Sampling, Mikolov et al. derived Hierarchical Softmax:
-
Constructs a binary Huffman Tree where all $|V|$ vocabulary words form the leaf nodes. Frequent words are assigned short root-to-leaf paths.
-
Probability of word $w$ is the product of sigmoid probabilities along the tree traversal path:
P(w | w_t) = prod_{j=1}^{L(w)-1} sigma( [ [ n(w, j+1) == child(n(w, j)) ] ] * theta_{n(w, j)}^\top v_{w_t} ) -
Reduces computational complexity from $O(|V|)$ to $O(\log_2 |V|)$ per forward pass without sampling approximations.
4. GloVe: Global Vectors for Word Representation
While Word2Vec relies on local sliding window optimization, GloVe (Pennington et al., 2014) directly optimizes over the global word-word co-occurrence matrix $X$, where $X_(i, j)$ is the number of times word $i$ appeared in context of word $j$:
- GloVe Objective:
J = sum_{i, j=1}^{|V|} f(X_{i, j}) * (w_i^\top \tilde{w}_j + b_i + \tilde{b}_j - ln(X_{i, j}))^2
where $f(X_(i, j)) = \min(1, (X_(i, j) / x_(\text(max)))^\alpha)$ is a weighting function preventing frequent stop words (like "the", "is") from dominating the loss.
5. FastText: Subword n-gram Embeddings
To eliminate the Out-Of-Vocabulary problem, FastText (Bojanowski et al., 2017) decomposes words into character n-grams bounded by < and >:
-
For word
"where"with $n=3$:n-grams = ["<wh", "whe", "her", "ere", "re>", "<where>"] -
The embedding for word $w$ is the sum of its constituent n-gram vectors:
v_w = sum_{g \in \text{n-grams}(w)} z_g
This enables FastText to represent misspelled words ("whree") and morphological variations effortlessly.
6. Subspace Geometry and Debiasing Projections
Bolukbasi et al. (2016) demonstrated that gender bias resides in a specific linear direction vector $\vec(g)$ within embedding space:
\vec{g} = (1 / |P|) * sum_{(m, f) \in P} (v_m - v_f)
- Neutralization: Project neutral occupation words (e.g.
doctor,nurse,engineer) onto the orthogonal complement of the gender subspace to remove unwanted social bias while preserving semantic meaning:v_{\text{neutral}} = v - (v \cdot \vec{g}) * \vec{g}
7. Pretrained Embedding Ecosystems in Modern NLP
When training classical neural networks, researchers initialize nn.Embedding from open-source precomputed weights:
- GloVe-6B / GloVe-840B: 300-dimensional vectors trained on 6 billion tokens of Wikipedia or 840 billion tokens of Common Crawl.
- FastText Crawl (2M Vocab): 300-dimensional vectors across 157 human languages.
- Static vs Dynamic Context: While static embeddings assign one vector per word, they remain ideal for edge deployment, lightweight intent classification, and vocabulary initialization in deep Transformers.
8. Approximate Nearest Neighbor (ANN) Indexing for Embeddings
When searching through an enterprise embedding database containing 10,000,000 document vectors, computing exhaustive cosine similarity against all $10^7$ vectors ($O(N \cdot d)$ brute force) takes hundreds of milliseconds:
-
Hierarchical Navigable Small World (HNSW Graphs): Constructs a multi-layer graph where top layers have long-range skip connections and bottom layers have dense local clusters, reducing query time from $O(N)$ to $O(\log N)$ with $< 5$ ms latency.
-
Inverted File with Product Quantization (IVF-PQ / FAISS): Partitions vector space into Voronoi cells and quantizes sub-vectors into 8-bit codes, compressing 300-dimensional vectors by $90%$ and enabling billion-scale in-memory similarity search.
-
Matryoshka Representation Learning (MRL / NeurIPS 2022): Modern embedding models are trained so that truncating the first 64 or 128 dimensions of a 1024-dimensional vector preserves 98% of semantic retrieval accuracy, enabling ultra-fast coarse vector filtering followed by high-dimensional reranking.
-
Binary Embedding Quantization (1-Bit Embeddings): Binarizing floating-point vector dimensions to 1-bit signs (positive vs negative) reduces vector memory by $32\times$ while computing Hamming distances via single CPU XOR instructions.
An everyday analogy
Think of describing people in a social network:
- One-Hot Encoding: You assign every person a unique ID number from 1 to 1,000,000. Knowing that Alice is ID #42 and Bob is ID #43 tells you nothing about whether they are friends, share hobbies, or live in the same city.
- Word Embeddings: You describe each person using a 5-element personality profile:
[Extroversion, Tech-Affinity, Athleticism, Artistic, Age].- Alice:
[0.9, 0.8, 0.2, 0.7, 28] - Bob:
[0.85, 0.82, 0.15, 0.65, 29] - By calculating the distance between their profiles, you instantly know Alice and Bob are close peers.
- Alice:
Examples in practice
Let us implement a PyTorch Embedding Lookup and Cosine Similarity Analogy Engine:
import torch
import torch.nn as nn
import torch.nn.functional as F
class SemanticVectorEngine:
def __init__(self, vocab_size: int, embed_dim: int):
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.vocab = {}
self.inverse_vocab = {}
def get_vector(self, word: str) -> torch.Tensor:
idx = torch.tensor([self.vocab[word]], dtype=torch.long)
return self.embedding(idx).squeeze(0)
def cosine_similarity(self, v1: torch.Tensor, v2: torch.Tensor) -> float:
return F.cosine_similarity(v1.unsqueeze(0), v2.unsqueeze(0)).item()
def find_nearest_neighbors(self, query_vector: torch.Tensor, top_k: int = 5):
# Normalize all embeddings
weights = self.embedding.weight.data
norm_weights = F.normalize(weights, p=2, dim=1)
norm_query = F.normalize(query_vector.unsqueeze(0), p=2, dim=1)
sims = torch.mm(norm_query, norm_weights.t()).squeeze(0)
top_scores, top_indices = torch.topk(sims, top_k)
return [(self.inverse_vocab[idx.item()], score.item())
for idx, score in zip(top_indices, top_scores)]
Implications: security, privacy, performance, scalability, and cost
- Societal and Gender Bias in Embeddings:
- Word embeddings inherit cultural biases present in training corpora. Classic Word2Vec models notoriously exhibited sexist analogies (e.g.
v(Man) -> Computer Programmermappedv(Woman) -> Homemaker). Modern NLP uses debiasing projection algorithms to neutralize gender subspaces.
- Word embeddings inherit cultural biases present in training corpora. Classic Word2Vec models notoriously exhibited sexist analogies (e.g.
- Embedding Table Memory Footprint:
- In large models, the embedding matrix $W \in R^(|V| \times d)$ is the largest single weight tensor. For $|V| = 250,000$ and $d = 4096$, storing FP32 weights consumes 4.1 GB of GPU VRAM just for the dictionary lookup table.
Alternatives: free, open source, and commercial
| Embedding Type | Mechanism | Context Sensitivity | Best For |
|---|---|---|---|
| Word2Vec / GloVe | Static co-occurrence vectors | Static (1 vector/word) | Fast text classification |
| FastText | Subword character n-grams | Static (morphological) | Typo-heavy / rare words |
OpenAI text-embedding-3 | Deep Transformer Encoder | Contextual (dynamic) | RAG / Vector search APIs |
| BGE / E5 Embeddings | Open-source Transformer | Contextual (dynamic) | Local vector databases |
Comparison with related concepts
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β WORD REPRESENTATION PARADIGMS β
ββββββββββββββββββββββ¬ββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββ€
β Paradigm β Sparsity β Semantic Meaning β Dimensions β
ββββββββββββββββββββββΌββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββ€
β One-Hot Encoding β 99.99% Sparse β None (0 Cosine) β 50k - 500k β
β TF-IDF Vector β Very Sparse β Document Context β 10k - 100k β
β Word2Vec / GloVe β 100% Dense β Semantic & Analogβ 100 - 300 β
β Transformer BERT β 100% Dense β Fully Contextual β 768 - 4096 β
ββββββββββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββ
When to use it β and when not to
When to USE Static Word Embeddings (GloVe/FastText/Word2Vec):
- Lightweight text classification (sentiment analysis, spam detection, topic tagging) running on CPU or edge devices.
When NOT to use:
- Complex contextual tasks (question answering, document summarization, generative chat) where polysemous words (e.g.
"apple"as fruit vs company) require dynamic attention-based contextualization.
The AI connection
- Embeddings are where meaning enters the model. Token IDs are arbitrary; the embedding matrix is the first place similarity means anything.
- Distributional semantics is the whole idea: words appearing in similar contexts get similar vectors, learned from co-occurrence rather than from a dictionary.
- Static embeddings cannot handle polysemy. βBankβ gets one vector for the river and the institution, which is exactly the limitation contextual models removed.
- The analogy arithmetic is real and oversold β the vector relationships exist, and they are far less reliable than the famous examples suggest.
- Embeddings inherit the corpusβs biases, measurably and in a way that transfers into every downstream system built on them.
Knowledge check
- Why are one-hot vectors incapable of measuring semantic similarity?
- What is the fundamental difference between CBOW and Skip-Gram in Word2Vec?
- How does Negative Sampling transform the multi-class softmax problem into binary classifications?
- How does FastText handle out-of-vocabulary words using character n-grams?
- How does vector arithmetic compute the analogy
King - Man + Woman = Queen?
Hands-on exercise
In this lab, you will build and test a complete Word Embedding engine in PyTorch: implement the Word2Vec Skip-Gram model architecture, derive the Negative Sampling binary cross-entropy loss function, train dense word vectors on a text corpus, compute cosine similarities across learned embeddings, and verify semantic vector arithmetic analogies.
Expected output
[Word Embedding Verification Suite]
Testing Skip-Gram with Negative Sampling (SGNS):
Input Center Word ID: 4 ("king"), Context Target ID: 12 ("royal")
Negative Samples Generated (K=4): [89, 412, 105, 934]
Negative Sampling Loss: 0.8412 [CONVERGING]
Testing Cosine Similarity & Semantic Analogy Engine:
Sim(cat, kitten) = 0.892 > Sim(cat, truck) = 0.041 [SEMANTIC ALIGNMENT VERIFIED]
Analogy Test: v(king) - v(man) + v(woman) -> Nearest: "queen" (Sim = 0.914)
Testing PyTorch nn.Embedding Lookup Table:
Embedding Weight Shape: torch.Size([1000, 64])
Lookup Latency: 0.12 ms
Test Suite: 3 passed in 0.31s
Validate your work
Run the automated test runner:
./tests/run_tests.sh
Troubleshooting
- If negative sampling loss diverges, ensure you apply
F.logsigmoidto positive pairs andF.logsigmoid(-u_neg)to negative pairs. - Normalize all vectors to unit length ($|v|_2 = 1$) before computing dot products for cosine similarity.
Common mistakes
- Sharing Target and Context Weight Matrices: Word2Vec maintains two separate matrices: input embeddings $W_(\text(in))$ and output context embeddings $W_(\text(out))$. Use $W_(\text(in))$ as the final word vectors.
Practice assignment
- Implement GloVe Co-occurrence Weighting: build a term co-occurrence matrix $X$ and implement the weighted mean squared error loss function.
- Build an Interactive 2D Embedding Visualizer using Principal Component Analysis (PCA) or t-SNE to project 100-dimensional embeddings onto a 2D scatter plot.
Extension challenge
Implement FastText Character n-Gram Embeddings from Scratch:
- Extract all 3-gram, 4-gram, and 5-gram substrings for each word.
- Maintain a shared subword embedding hash table with 100,000 buckets.
- Compute out-of-vocabulary word embeddings by summing hashed subword embeddings.
Quiz
Q1. What is the fundamental limitation of one-hot encoded word vectors that dense word embeddings resolve?
- One-hot vectors are orthogonal, so all similarities are zero
- One-hot vectors cannot be processed on GPUs
- One-hot vectors only work with numbers, not words
- One-hot vectors require square root calculations
Show answer
Answer: A. One-hot vectors are orthogonal, so all similarities are zero
Because every one-hot vector has a 1 in a distinct position and 0 everywhere else, all pairs are orthogonal and possess zero geometric similarity.
Q2. What is the core premise of the Distributional Hypothesis formulated by linguist J.R. Firth in 1957?
- All words in a sentence must have identical lengths
- A word is known by the company it keeps
- Language models must be trained for at least 100 epochs
- Grammar is entirely determined by alphabetical order
Show answer
Answer: B. A word is known by the company it keeps
Firths distributional hypothesis is the foundation of Word2Vec and modern language embeddings: context co-occurrence defines semantic meaning.
Q3. Why is Negative Sampling necessary when training large-vocabulary Word2Vec Skip-Gram models?
- It prevents gradients from exploding in float32
- It deletes all punctuation from the text
- The exact softmax over 500k words is far too expensive
- It reduces the batch size to 1
Show answer
Answer: C. The exact softmax over 500k words is far too expensive
Negative sampling replaces an intractable full-vocabulary softmax with efficient binary classifications distinguishing true context words from K random noise words.
Q4. How does FastText (Bojanowski et al., 2017) improve upon Word2Vec when encountering rare or out-of-vocabulary words?
- It uses a recurrent neural network
- It increases the embedding dimension from 300 to 10,000
- It downloads translations from Wikipedia
- It sums character n-gram vectors to build the word
Show answer
Answer: D. It sums character n-gram vectors to build the word
By embedding subword character n-grams, FastText can construct accurate embeddings for unseen words from their morphological root pieces.
Q5. In a trained 300-dimensional semantic embedding space, what does the vector operation v(Paris) - v(France) + v(Italy) approximate?
- v(Rome)
- v(Europe)
- v(London)
- v(Zero vector)
Show answer
Answer: A. v(Rome)
Linear vector offsets in dense embedding spaces encode relational analogies across capitals, gender, and verb tenses.
Glossary
- Word Embedding
- A dense, continuous real-valued vector representing a word in a multi-dimensional semantic space (typically 100-1024 dimensions).
- Distributional Hypothesis
- The linguistic principle that words occurring in similar textual contexts possess similar semantic meanings.
- Continuous Bag-of-Words (CBOW)
- A Word2Vec model architecture that predicts a center target word given the average vector of its surrounding context words.
- Skip-Gram
- A Word2Vec model architecture that predicts surrounding context words given a single center target word.
- Negative Sampling
- An efficient training approximation for Word2Vec that maximizes the probability of true context pairs while minimizing K randomly sampled noise words.
- Cosine Similarity
- A metric measuring the cosine of the angle between two vectors, quantifying directional semantic alignment independent of vector magnitude.
- GloVe
- Global Vectors: an unsupervised learning algorithm that factors global word-word co-occurrence statistics across a corpus.
- FastText
- An extension of Word2Vec that computes word representations as the sum of character n-gram subword embeddings.
Sources and further reading
- Efficient Estimation of Word Representations in Vector Space β arXiv Preprints (accessed 2026-08-29)
- GloVe: Global Vectors for Word Representation β EMNLP 2014 (accessed 2026-08-29)
- Enriching Word Vectors with Subword Information β Transactions of the ACL (accessed 2026-08-29)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.