Deep Learning β€Ί Sequences and Text β€Ί Day 219

Day 219: Word Embeddings

Day 219 of 365 β€” Word Embeddings

Master dense distributed representations of language: derive Word2Vec (CBOW and Skip-Gram), implement Negative Sampling loss from scratch, understand GloVe co-occurrence matrix factorization and FastText subword n-grams, compute cosine similarity, and execute semantic vector arithmetic in PyTorch.

Course
Deep Learning
Category
Sequences and Text
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-219-word-embeddings

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-219-word-embeddings
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Yesterday in Day 218, we mastered tokenization: transforming raw human text strings into discrete integer token IDs.

However, an integer token ID (like token_id = 4937) contains zero semantic meaning. In the eyes of a mathematical neural network:

One-hot encodings are computationally massive, sparse, and completely blind to semantic relationships, synonyms, and grammatical hierarchies.

To solve this, deep learning represents words as Dense Word Embeddings: continuous, low-dimensional vectors (typically $d = 100$ to $1024$ dimensions) where spatial distance and geometric direction encode genuine semantic meaning.

In this lesson, you will master the foundational algorithms of vector semantics: Word2Vec (CBOW & Skip-Gram), Negative Sampling, GloVe, FastText, and the famous geometric analogies ($v_(\text(king)) - v_(\text(man)) + v_(\text(woman)) \approx v_(\text(queen))$).


The idea in plain language

Diagram: static word embeddings compared with contextual embeddings

Diagram: one-hot encoding compared with a learned embedding

Diagram: what a word embedding matrix is and what it encodes

Imagine mapping every concept in the human universe onto a multi-dimensional globe:

A Word Embedding is simply the GPS coordinate $(x_1, x_2, \dots, x_d)$ of a word in this semantic universe.


Historical background

Diagram: the progression of static word embedding methods

Word2Vec architecture comparison showing Continuous Bag of Words CBOW predicting center word from context and Skip-Gram predicting context words from center word

  1. 1957 (The Distributional Hypothesis): British linguist J.R. Firth proposed the foundational maxim of computational semantics: β€œYou shall know a word by the company it keeps.” Words that share similar surrounding context words share similar meanings.
  2. 2003 (Bengio’s Neural Language Model): Yoshua Bengio et al. introduced distributed representations of words as learned continuous lookup tables in neural networks.
  3. 2013 (Tomas Mikolov & Word2Vec at Google): Tomas Mikolov introduced Word2Vec (CBOW and Skip-Gram with Negative Sampling), drastically reducing training complexity and discovering that vector space offsets naturally encoded semantic analogies.
  4. 2014 (GloVe at Stanford): Jeffrey Pennington, Richard Socher, and Chris Manning introduced GloVe (Global Vectors), combining the advantages of global matrix factorization with local context window architectures.
  5. 2017 (FastText at Facebook AI): Piotr Bojanowski et al. extended Word2Vec with subword character n-grams, enabling embeddings for out-of-vocabulary and morphologically rich words.

What it is β€” and what it is not

What Word Embeddings ARE:

What they are NOT:


Why it was created and what problems it solves

Word embeddings resolved four critical bottlenecks in NLP:

  1. Curse of Dimensionality: Replacing 1,000,000-dimensional sparse one-hot vectors with compact 300-dimensional dense vectors.
  2. Semantic Orthogonality: Providing a continuous distance metric (Cosine Similarity) where synonyms yield high dot products.
  3. Generalization Across Rare Words: A sentiment model trained on "The food was delicious" automatically understands "The food was scrumptious" because their vectors lie adjacent in embedding space.
  4. Linear Semantic Reasoning: Enabling vector arithmetic across analogies, gender, pluralization, and country-capital relationships.

How it works

Let us dissect the mathematical architectures of Word2Vec, GloVe, and FastText.


1. Word2Vec: Continuous Bag-of-Words (CBOW)

In CBOW, given a context window of size $C$ around center word $w_t$:

Context: [w_{t-2}, w_{t-1}, w_{t+1}, w_{t+2}] -> Predict Target: w_t
  1. Look up input embedding vectors for all context words: $v_(c) \in R^d$.

  2. Compute the average context vector:

    v_context = (1 / 2C) * sum_{j \in context} v_{w_j}
  3. Project $v_(\text(context))$ through an output weight matrix $W_(\text(out)) \in R^(d \times |V|)$ to compute logits across all vocabulary words.

  4. Apply Softmax to predict the probability of center word $w_t$.


2. Word2Vec: Continuous Skip-Gram with Negative Sampling (SGNS)

Diagram: the softmax bottleneck and the negative sampling solution

In Skip-Gram, the task is inverted: given center word $w_t$, predict each surrounding context word $w_c$:

Geometric semantic vector space diagram showing linear relationship analogies such as King minus Man plus Woman equals Queen

The Exact Softmax Bottleneck:

The true conditional probability requires normalizing over the entire vocabulary:

P(w_c | w_t) = exp(u_{w_c}^\top v_{w_t}) / sum_{w \in V} exp(u_w^\top v_{w_t})

Summing over $|V| = 500,000$ words for every token update is computationally intractable ($O(|V|)$ per step).

The Negative Sampling Solution:

Mikolov et al. converted the multi-class prediction into multiple binary logistic regressions:

where $\sigma(z) = \frac(1)(1 + e^(-z))$ is the sigmoid function. This reduces complexity from $O(|V|)$ to $O(K)$, where $K \in [5, 20]$.


3. Hierarchical Softmax Alternative

Prior to Negative Sampling, Mikolov et al. derived Hierarchical Softmax:


4. GloVe: Global Vectors for Word Representation

While Word2Vec relies on local sliding window optimization, GloVe (Pennington et al., 2014) directly optimizes over the global word-word co-occurrence matrix $X$, where $X_(i, j)$ is the number of times word $i$ appeared in context of word $j$:

where $f(X_(i, j)) = \min(1, (X_(i, j) / x_(\text(max)))^\alpha)$ is a weighting function preventing frequent stop words (like "the", "is") from dominating the loss.


5. FastText: Subword n-gram Embeddings

To eliminate the Out-Of-Vocabulary problem, FastText (Bojanowski et al., 2017) decomposes words into character n-grams bounded by < and >:

This enables FastText to represent misspelled words ("whree") and morphological variations effortlessly.


6. Subspace Geometry and Debiasing Projections

Diagram: what vector arithmetic on embeddings does and does not reliably do

Bolukbasi et al. (2016) demonstrated that gender bias resides in a specific linear direction vector $\vec(g)$ within embedding space:

\vec{g} = (1 / |P|) * sum_{(m, f) \in P} (v_m - v_f)

7. Pretrained Embedding Ecosystems in Modern NLP

When training classical neural networks, researchers initialize nn.Embedding from open-source precomputed weights:


8. Approximate Nearest Neighbor (ANN) Indexing for Embeddings

When searching through an enterprise embedding database containing 10,000,000 document vectors, computing exhaustive cosine similarity against all $10^7$ vectors ($O(N \cdot d)$ brute force) takes hundreds of milliseconds:


An everyday analogy

Diagram: describing individuals by listed attributes compared with describing them by position in a network

Think of describing people in a social network:


Examples in practice

Let us implement a PyTorch Embedding Lookup and Cosine Similarity Analogy Engine:

import torch
import torch.nn as nn
import torch.nn.functional as F

class SemanticVectorEngine:
    def __init__(self, vocab_size: int, embed_dim: int):
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.vocab = {}
        self.inverse_vocab = {}

    def get_vector(self, word: str) -> torch.Tensor:
        idx = torch.tensor([self.vocab[word]], dtype=torch.long)
        return self.embedding(idx).squeeze(0)

    def cosine_similarity(self, v1: torch.Tensor, v2: torch.Tensor) -> float:
        return F.cosine_similarity(v1.unsqueeze(0), v2.unsqueeze(0)).item()

    def find_nearest_neighbors(self, query_vector: torch.Tensor, top_k: int = 5):
        # Normalize all embeddings
        weights = self.embedding.weight.data
        norm_weights = F.normalize(weights, p=2, dim=1)
        norm_query = F.normalize(query_vector.unsqueeze(0), p=2, dim=1)

        sims = torch.mm(norm_query, norm_weights.t()).squeeze(0)
        top_scores, top_indices = torch.topk(sims, top_k)

        return [(self.inverse_vocab[idx.item()], score.item())
                for idx, score in zip(top_indices, top_scores)]

Implications: security, privacy, performance, scalability, and cost

Diagram: the memory, bias, retrieval and initialisation implications of embeddings

  1. Societal and Gender Bias in Embeddings:
    • Word embeddings inherit cultural biases present in training corpora. Classic Word2Vec models notoriously exhibited sexist analogies (e.g. v(Man) -> Computer Programmer mapped v(Woman) -> Homemaker). Modern NLP uses debiasing projection algorithms to neutralize gender subspaces.
  2. Embedding Table Memory Footprint:
    • In large models, the embedding matrix $W \in R^(|V| \times d)$ is the largest single weight tensor. For $|V| = 250,000$ and $d = 4096$, storing FP32 weights consumes 4.1 GB of GPU VRAM just for the dictionary lookup table.

Alternatives: free, open source, and commercial

Embedding TypeMechanismContext SensitivityBest For
Word2Vec / GloVeStatic co-occurrence vectorsStatic (1 vector/word)Fast text classification
FastTextSubword character n-gramsStatic (morphological)Typo-heavy / rare words
OpenAI text-embedding-3Deep Transformer EncoderContextual (dynamic)RAG / Vector search APIs
BGE / E5 EmbeddingsOpen-source TransformerContextual (dynamic)Local vector databases

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   WORD REPRESENTATION PARADIGMS                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Paradigm           β”‚ Sparsity        β”‚ Semantic Meaning β”‚ Dimensions   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ One-Hot Encoding   β”‚ 99.99% Sparse   β”‚ None (0 Cosine)  β”‚ 50k - 500k   β”‚
β”‚ TF-IDF Vector      β”‚ Very Sparse     β”‚ Document Context β”‚ 10k - 100k   β”‚
β”‚ Word2Vec / GloVe   β”‚ 100% Dense      β”‚ Semantic & Analogβ”‚ 100 - 300    β”‚
β”‚ Transformer BERT   β”‚ 100% Dense      β”‚ Fully Contextual β”‚ 768 - 4096   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when static word embeddings are adequate

When to USE Static Word Embeddings (GloVe/FastText/Word2Vec):

When NOT to use:


The AI connection

Knowledge check

  1. Why are one-hot vectors incapable of measuring semantic similarity?
  2. What is the fundamental difference between CBOW and Skip-Gram in Word2Vec?
  3. How does Negative Sampling transform the multi-class softmax problem into binary classifications?
  4. How does FastText handle out-of-vocabulary words using character n-grams?
  5. How does vector arithmetic compute the analogy King - Man + Woman = Queen?

Hands-on exercise

In this lab, you will build and test a complete Word Embedding engine in PyTorch: implement the Word2Vec Skip-Gram model architecture, derive the Negative Sampling binary cross-entropy loss function, train dense word vectors on a text corpus, compute cosine similarities across learned embeddings, and verify semantic vector arithmetic analogies.

Expected output

[Word Embedding Verification Suite]
Testing Skip-Gram with Negative Sampling (SGNS):
  Input Center Word ID: 4 ("king"), Context Target ID: 12 ("royal")
  Negative Samples Generated (K=4): [89, 412, 105, 934]
  Negative Sampling Loss: 0.8412 [CONVERGING]
Testing Cosine Similarity & Semantic Analogy Engine:
  Sim(cat, kitten) = 0.892 > Sim(cat, truck) = 0.041 [SEMANTIC ALIGNMENT VERIFIED]
  Analogy Test: v(king) - v(man) + v(woman) -> Nearest: "queen" (Sim = 0.914)
Testing PyTorch nn.Embedding Lookup Table:
  Embedding Weight Shape: torch.Size([1000, 64])
  Lookup Latency: 0.12 ms
Test Suite: 3 passed in 0.31s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement GloVe Co-occurrence Weighting: build a term co-occurrence matrix $X$ and implement the weighted mean squared error loss function.
  2. Build an Interactive 2D Embedding Visualizer using Principal Component Analysis (PCA) or t-SNE to project 100-dimensional embeddings onto a 2D scatter plot.

Extension challenge

Implement FastText Character n-Gram Embeddings from Scratch:

Quiz

Q1. What is the fundamental limitation of one-hot encoded word vectors that dense word embeddings resolve?

  1. One-hot vectors are orthogonal, so all similarities are zero
  2. One-hot vectors cannot be processed on GPUs
  3. One-hot vectors only work with numbers, not words
  4. One-hot vectors require square root calculations
Show answer

Answer: A. One-hot vectors are orthogonal, so all similarities are zero

Because every one-hot vector has a 1 in a distinct position and 0 everywhere else, all pairs are orthogonal and possess zero geometric similarity.

Q2. What is the core premise of the Distributional Hypothesis formulated by linguist J.R. Firth in 1957?

  1. All words in a sentence must have identical lengths
  2. A word is known by the company it keeps
  3. Language models must be trained for at least 100 epochs
  4. Grammar is entirely determined by alphabetical order
Show answer

Answer: B. A word is known by the company it keeps

Firths distributional hypothesis is the foundation of Word2Vec and modern language embeddings: context co-occurrence defines semantic meaning.

Q3. Why is Negative Sampling necessary when training large-vocabulary Word2Vec Skip-Gram models?

  1. It prevents gradients from exploding in float32
  2. It deletes all punctuation from the text
  3. The exact softmax over 500k words is far too expensive
  4. It reduces the batch size to 1
Show answer

Answer: C. The exact softmax over 500k words is far too expensive

Negative sampling replaces an intractable full-vocabulary softmax with efficient binary classifications distinguishing true context words from K random noise words.

Q4. How does FastText (Bojanowski et al., 2017) improve upon Word2Vec when encountering rare or out-of-vocabulary words?

  1. It uses a recurrent neural network
  2. It increases the embedding dimension from 300 to 10,000
  3. It downloads translations from Wikipedia
  4. It sums character n-gram vectors to build the word
Show answer

Answer: D. It sums character n-gram vectors to build the word

By embedding subword character n-grams, FastText can construct accurate embeddings for unseen words from their morphological root pieces.

Q5. In a trained 300-dimensional semantic embedding space, what does the vector operation v(Paris) - v(France) + v(Italy) approximate?

  1. v(Rome)
  2. v(Europe)
  3. v(London)
  4. v(Zero vector)
Show answer

Answer: A. v(Rome)

Linear vector offsets in dense embedding spaces encode relational analogies across capitals, gender, and verb tenses.

Glossary

Word Embedding
A dense, continuous real-valued vector representing a word in a multi-dimensional semantic space (typically 100-1024 dimensions).
Distributional Hypothesis
The linguistic principle that words occurring in similar textual contexts possess similar semantic meanings.
Continuous Bag-of-Words (CBOW)
A Word2Vec model architecture that predicts a center target word given the average vector of its surrounding context words.
Skip-Gram
A Word2Vec model architecture that predicts surrounding context words given a single center target word.
Negative Sampling
An efficient training approximation for Word2Vec that maximizes the probability of true context pairs while minimizing K randomly sampled noise words.
Cosine Similarity
A metric measuring the cosine of the angle between two vectors, quantifying directional semantic alignment independent of vector magnitude.
GloVe
Global Vectors: an unsupervised learning algorithm that factors global word-word co-occurrence statistics across a corpus.
FastText
An extension of Word2Vec that computes word representations as the sum of character n-gram subword embeddings.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.