Deep Learning › Sequences and Text › Day 219

Hands-on lab — Day 219: Word Embeddings

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/word_embeddings_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/test_word_embeddings_lib.py
examples/word_embeddings_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_word_embeddings_lib.py
starter/word_embeddings_lib.py
tests/run_tests.sh
tests/test_word_embeddings_lib.py
troubleshooting.md

Lab README

Lab: Day 219 -- Word Embeddings

Lesson

Day number: 219 of 365. Course: Course05-SS02 (Deep Learning - Sequence Models and Transformers). Topic: Word Embeddings and Vector Semantics.

Purpose

Build and test a complete Word Embedding engine in PyTorch. Implement the Word2Vec Skip-Gram model architecture with Negative Sampling loss, evaluate cosine similarity metrics, and perform semantic vector arithmetic.

Learning objectives

  • Implement the Word2Vec Skip-Gram neural architecture with target and context embedding layers.
  • Derive and compute Negative Sampling binary cross-entropy loss.
  • Calculate cosine similarity across multi-dimensional embedding vectors.
  • Validate semantic geometric properties in learned embedding spaces.

Prerequisites

  • Day 218 (Text Preprocessing and Tokenization).
  • Python 3.11+ with PyTorch and NumPy.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is open-source software maintained by the Linux Foundation under a modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/word_embeddings_lib.py: Student scaffold file.
  • examples/word_embeddings_lib.py: Complete reference implementation.
  • tests/test_word_embeddings_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/word_embeddings_lib.py

What the commands do

  • Initializes target and context embedding tables in PyTorch.
  • Computes negative sampling loss over sample center, context, and noise batches.
  • Runs cosine similarity assertions across orthogonal and parallel vectors.

Expected output

Embedding Demo: Loss = 3.4716, Cosine Sim = 0.1245

Validation steps

  1. Verify SkipGramNegativeSampling.forward returns a scalar positive loss tensor.
  2. Confirm compute_cosine_similarity returns 1.0 for identical vectors and 0.0 for orthogonal vectors.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Negative Loss Values: Ensure you are minimizing - (pos_loss + neg_loss).

Security notes

All neural embedding operations execute locally on CPU or local GPU acceleration.

Extension exercises

  1. Implement GloVe weighted co-occurrence matrix regression loss.
  2. Implement FastText character n-gram hashing embedding layer.
  • Lesson title: Word Embeddings
  • Day number: 219 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-219-word-embeddings
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-219-word-embeddings when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 219

- `Loss`: Scalar value of Skip-Gram Negative Sampling loss.
- `Cosine Sim`: Cosine similarity between sample target embeddings.

examples-run.txt

Embedding Demo: Loss = 3.4716, Cosine Sim = 0.1245

measured-values.txt

Loss: 3.4716
Cosine Sim: 0.1245

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_word_embeddings_lib.py::test_skipgram_loss_computation PASSED [ 50%]
tests/test_word_embeddings_lib.py::test_cosine_similarity_bounds PASSED   [100%]

============================== 2 passed in 0.31s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/test_word_embeddings_lib.py (933 bytes)
import pytest
import torch
from examples.word_embeddings_lib import SkipGramNegativeSampling, compute_cosine_similarity

def test_skipgram_loss_computation():
    torch.manual_seed(42)
    model = SkipGramNegativeSampling(vocab_size=50, embed_dim=8)
    center = torch.tensor([1, 2])
    context = torch.tensor([3, 4])
    negatives = torch.tensor([[10, 11], [12, 13]])
    
    loss = model(center, context, negatives)
    assert isinstance(loss, torch.Tensor)
    assert loss.ndim == 0
    assert loss.item() > 0.0

def test_cosine_similarity_bounds():
    v1 = torch.tensor([1.0, 0.0, 0.0])
    v2 = torch.tensor([1.0, 0.0, 0.0])
    v3 = torch.tensor([-1.0, 0.0, 0.0])
    v4 = torch.tensor([0.0, 1.0, 0.0])

    assert pytest.approx(compute_cosine_similarity(v1, v2), 1e-4) == 1.0
    assert pytest.approx(compute_cosine_similarity(v1, v3), 1e-4) == -1.0
    assert pytest.approx(compute_cosine_similarity(v1, v4), 1e-4) == 0.0
examples/word_embeddings_lib.py (2339 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Tuple, Dict, Any

class SkipGramNegativeSampling(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int):
        super().__init__()
        self.target_embeddings = nn.Embedding(vocab_size, embed_dim)
        self.context_embeddings = nn.Embedding(vocab_size, embed_dim)
        # Initialize weights uniformly
        nn.init.uniform_(self.target_embeddings.weight, -0.5 / embed_dim, 0.5 / embed_dim)
        nn.init.uniform_(self.context_embeddings.weight, -0.5 / embed_dim, 0.5 / embed_dim)

    def forward(self, center_words: torch.Tensor, context_words: torch.Tensor,
                negative_words: torch.Tensor) -> torch.Tensor:
        # center_words: (B,)
        # context_words: (B,)
        # negative_words: (B, K)
        
        v_center = self.target_embeddings(center_words) # (B, d)
        u_context = self.context_embeddings(context_words) # (B, d)
        u_neg = self.context_embeddings(negative_words) # (B, K, d)

        # Positive score: log(sigmoid(v_center . u_context))
        pos_score = torch.sum(v_center * u_context, dim=1) # (B,)
        pos_loss = F.logsigmoid(pos_score)

        # Negative score: log(sigmoid(- v_center . u_neg))
        # (B, 1, d) x (B, d, K) -> (B, 1, K) -> (B, K)
        neg_score = torch.bmm(u_neg, v_center.unsqueeze(2)).squeeze(2) # (B, K)
        neg_loss = torch.sum(F.logsigmoid(-neg_score), dim=1) # (B,)

        total_loss = - (pos_loss + neg_loss).mean()
        return total_loss

def compute_cosine_similarity(v1: torch.Tensor, v2: torch.Tensor) -> float:
    return F.cosine_similarity(v1.unsqueeze(0), v2.unsqueeze(0)).item()

def run_embedding_demo():
    torch.manual_seed(42)
    model = SkipGramNegativeSampling(vocab_size=100, embed_dim=16)
    
    center = torch.tensor([4, 10])
    context = torch.tensor([5, 12])
    negatives = torch.tensor([
        [15, 22, 33, 44],
        [55, 66, 77, 88]
    ])

    loss = model(center, context, negatives)
    v1 = model.target_embeddings.weight[0]
    v2 = model.target_embeddings.weight[1]
    sim = compute_cosine_similarity(v1, v2)
    
    print(f"Embedding Demo: Loss = {loss.item():.4f}, Cosine Sim = {sim:.4f}")
    return model, loss.item(), sim

if __name__ == "__main__":
    run_embedding_demo()
metadata.yml (428 bytes)
lesson_id: D219
day: 219
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/word_embeddings_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_word_embeddings_lib.py (933 bytes)
import pytest
import torch
from examples.word_embeddings_lib import SkipGramNegativeSampling, compute_cosine_similarity

def test_skipgram_loss_computation():
    torch.manual_seed(42)
    model = SkipGramNegativeSampling(vocab_size=50, embed_dim=8)
    center = torch.tensor([1, 2])
    context = torch.tensor([3, 4])
    negatives = torch.tensor([[10, 11], [12, 13]])
    
    loss = model(center, context, negatives)
    assert isinstance(loss, torch.Tensor)
    assert loss.ndim == 0
    assert loss.item() > 0.0

def test_cosine_similarity_bounds():
    v1 = torch.tensor([1.0, 0.0, 0.0])
    v2 = torch.tensor([1.0, 0.0, 0.0])
    v3 = torch.tensor([-1.0, 0.0, 0.0])
    v4 = torch.tensor([0.0, 1.0, 0.0])

    assert pytest.approx(compute_cosine_similarity(v1, v2), 1e-4) == 1.0
    assert pytest.approx(compute_cosine_similarity(v1, v3), 1e-4) == -1.0
    assert pytest.approx(compute_cosine_similarity(v1, v4), 1e-4) == 0.0
starter/word_embeddings_lib.py (668 bytes)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Tuple, Dict, Any

class SkipGramNegativeSampling(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int):
        super().__init__()
        # TODO: Initialize target embeddings and context embeddings
        pass

    def forward(self, center_words: torch.Tensor, context_words: torch.Tensor,
                negative_words: torch.Tensor) -> torch.Tensor:
        # TODO: Compute Negative Sampling loss
        pass

def compute_cosine_similarity(v1: torch.Tensor, v2: torch.Tensor) -> float:
    # TODO: Compute cosine similarity between two 1D vectors
    pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 219 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_word_embeddings_lib.py (933 bytes)
import pytest
import torch
from examples.word_embeddings_lib import SkipGramNegativeSampling, compute_cosine_similarity

def test_skipgram_loss_computation():
    torch.manual_seed(42)
    model = SkipGramNegativeSampling(vocab_size=50, embed_dim=8)
    center = torch.tensor([1, 2])
    context = torch.tensor([3, 4])
    negatives = torch.tensor([[10, 11], [12, 13]])
    
    loss = model(center, context, negatives)
    assert isinstance(loss, torch.Tensor)
    assert loss.ndim == 0
    assert loss.item() > 0.0

def test_cosine_similarity_bounds():
    v1 = torch.tensor([1.0, 0.0, 0.0])
    v2 = torch.tensor([1.0, 0.0, 0.0])
    v3 = torch.tensor([-1.0, 0.0, 0.0])
    v4 = torch.tensor([0.0, 1.0, 0.0])

    assert pytest.approx(compute_cosine_similarity(v1, v2), 1e-4) == 1.0
    assert pytest.approx(compute_cosine_similarity(v1, v3), 1e-4) == -1.0
    assert pytest.approx(compute_cosine_similarity(v1, v4), 1e-4) == 0.0

Troubleshooting

Troubleshooting: Day 219 - Word Embeddings

Common Issues

  1. Dimension mismatch in batch matrix multiplication:
    • Cause: Unmatched tensor dimensions between v_center and u_neg.
    • Fix: Use torch.bmm(u_neg, v_center.unsqueeze(2)).squeeze(2).

Security notes

Security & Privacy: Day 219 - Word Embeddings

Security Guidance

  • All embedding tables and vectors reside in local process memory.