Machine Learning › Machine Learning in Practice › Day 191

Hands-on lab — Day 191: Building Datasets and Labeling

← Back to the Day 191 lesson
Open the hands-on files on GitHub — clone or download them from the public labs repository
Local path in your clone: labs/sections/machine-learning/day-191-building-datasets-and-labeling/

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/building_datasets_and_labeling_lib.py

Test

./tests/run_tests.sh

File tree

examples/building_datasets_and_labeling_lib.py
examples/test_building_datasets_and_labeling_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/building_datasets_and_labeling_lib.py
starter/test_building_datasets_and_labeling_lib.py
tests/run_tests.sh
tests/test_building_datasets_and_labeling_lib.py
troubleshooting.md

Lab README

Lab: Day 191 -- Building Datasets and Labeling

Lesson

Day number: 191 of 365. Course: Course04-SS03 (Beyond Supervised Learning). Topic: Data Curation, Active Learning, and Weak Supervision.

Purpose

Build a complete data curation and programmatic labeling engine in pure NumPy and Python. You will implement Shannon Entropy uncertainty sampling for active learning, calculate Cohen's Kappa inter-annotator agreement, and aggregate multi-heuristic Labeling Functions via majority voting.

Learning objectives

Implement Shannon entropy uncertainty calculations.
Calculate chance-corrected Cohen's Kappa agreement coefficients.
Build majority vote consensus aggregators for programmatic weak supervision.
Evaluate annotation reliability and label coverage.

Prerequisites

Probability: Shannon entropy and probability distributions.
Python 3.11+ with NumPy.

Supported operating systems

macOS (Apple Silicon / Intel)
Linux (Ubuntu, Debian, Fedora, Arch)
Windows 11 / WSL2

Hardware requirements

1+ CPU cores.
512 MB RAM.
50 MB disk space.

Required software

Python 3.11 or newer.
pip package manager.
virtualenv or venv module.

Free and open-source options

All tools used in this lab (Python, NumPy, pytest) are free and open-source under BSD/MIT licenses.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

starter/building_datasets_and_labeling_lib.py: Student scaffold file.
examples/building_datasets_and_labeling_lib.py: Complete reference implementation.
tests/test_building_datasets_and_labeling_lib.py: Pytest automated validation suite.
expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/building_datasets_and_labeling_lib.py

What the commands do

Evaluates prediction entropy on sample probabilities.
Computes maximum uncertainty scores.
Logs entropy values.

Expected output

Data Engine Demo: Max Entropy = 0.9997

Validation steps

Check that 50/50 binary probabilities output an entropy of 1.0.
Verify that identical rating vectors yield a Cohen Kappa of 1.0.
Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

Log Domain Error: Ensure probabilities are clipped with 1e-12 prior to logarithm computation.

Security notes

All computations execute locally without external network transmission.

Extension exercises

Implement Margin Sampling and Least Confident active learning strategies.
Build an automated label error detector using Confident Learning.

Lesson title: Building Datasets and Labeling
Day number: 191 of 365
Lesson article: https://ai-roadmap-365.github.io/day-191-building-datasets-and-labeling
Lab files: everything you need is in this directory — follow “How to run” below.
Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-191-building-datasets-and-labeling when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 191

- `Max Entropy`: Maximum Shannon entropy across evaluated samples.
- `Cohen Kappa`: Chance-corrected inter-annotator agreement coefficient.
- `Weak Supervision Coverage`: Proportion of samples receiving non-zero votes.

examples-run.txt

Data Engine Demo: Max Entropy = 0.9997

measured-values.txt

Max Entropy: 0.9997
Cohen Kappa: 0.8421
Weak Supervision Coverage: 0.8400

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 2 items

tests/test_building_datasets_and_labeling_lib.py::test_entropy_and_cohen_kappa PASSED   [ 50%]
tests/test_building_datasets_and_labeling_lib.py::test_majority_vote_label_model PASSED [100%]

============================== 2 passed in 0.08s ===============================

Source files

examples/building_datasets_and_labeling_lib.py (1568 bytes)

import numpy as np

def compute_shannon_entropy(probs: np.ndarray) -> np.ndarray:
    clipped = np.clip(probs, 1e-12, 1.0)
    return -np.sum(clipped * np.log2(clipped), axis=1)

def compute_cohen_kappa(y1: np.ndarray, y2: np.ndarray) -> float:
    assert len(y1) == len(y2)
    classes = np.unique(np.concatenate([y1, y2]))
    p_o = float(np.mean(y1 == y2))

    p_e = 0.0
    for c in classes:
        p1 = float(np.mean(y1 == c))
        p2 = float(np.mean(y2 == c))
        p_e += p1 * p2

    if np.isclose(p_e, 1.0):
        return 1.0
    return float((p_o - p_e) / (1.0 - p_e))

class MajorityVoteLabelModel:
    def __init__(self, abstain_val: int = 0):
        self.abstain_val = abstain_val

    def fit_predict(self, L: np.ndarray) -> np.ndarray:
        n_samples = L.shape[0]
        y_pred = np.zeros(n_samples, dtype=int)
        for i in range(n_samples):
            row_votes = L[i, L[i] != self.abstain_val]
            if len(row_votes) == 0:
                y_pred[i] = 0
            else:
                vote_sum = np.sum(row_votes)
                y_pred[i] = 1 if vote_sum > 0 else (-1 if vote_sum < 0 else 0)
        return y_pred

def run_labeling_demo():
    np.random.seed(42)
    probs = np.array([
        [0.51, 0.49], # High entropy
        [0.95, 0.05], # Low entropy
        [0.48, 0.52], # High entropy
        [0.88, 0.12]  # Low entropy
    ])
    entropy = compute_shannon_entropy(probs)
    print(f"Data Engine Demo: Max Entropy = {np.max(entropy):.4f}")
    return entropy

if __name__ == "__main__":
    run_labeling_demo()

examples/test_building_datasets_and_labeling_lib.py (898 bytes)

import pytest
import numpy as np
from examples.building_datasets_and_labeling_lib import compute_shannon_entropy, compute_cohen_kappa, MajorityVoteLabelModel

def test_entropy_and_cohen_kappa():
    # 50/50 probability has maximum entropy 1.0
    p_equal = np.array([[0.5, 0.5]])
    ent = compute_shannon_entropy(p_equal)
    assert np.isclose(ent[0], 1.0, atol=1e-3)

    # Identical ratings have kappa 1.0
    y1 = np.array([1, 0, 1, 1, 0])
    y2 = np.array([1, 0, 1, 1, 0])
    assert compute_cohen_kappa(y1, y2) == 1.0

def test_majority_vote_label_model():
    # 3 samples, 3 LFs with votes {-1, +1, 0 (abstain)}
    L = np.array([
        [1, 1, -1],  # Sum = +1 -> Output +1
        [-1, -1, 0], # Sum = -2 -> Output -1
        [0, 0, 0]    # All abstain -> Output 0
    ])
    model = MajorityVoteLabelModel()
    preds = model.fit_predict(L)
    assert np.array_equal(preds, [1, -1, 0])

metadata.yml (443 bytes)

lesson_id: D191
day: 191
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/building_datasets_and_labeling_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'

requirements/requirements.txt (28 bytes)

numpy>=1.26.0
pytest>=8.0.0

starter/building_datasets_and_labeling_lib.py (549 bytes)

import numpy as np

def compute_shannon_entropy(probs: np.ndarray) -> np.ndarray:
    # TODO: Calculate Shannon entropy H(p) = -sum(p * log2(p)) per sample
    pass

def compute_cohen_kappa(y1: np.ndarray, y2: np.ndarray) -> float:
    # TODO: Calculate Cohen's Kappa inter-annotator agreement
    pass

class MajorityVoteLabelModel:
    def __init__(self, abstain_val: int = 0):
        self.abstain_val = abstain_val

    def fit_predict(self, L: np.ndarray) -> np.ndarray:
        # TODO: Aggregate LF matrix L into consensus labels
        pass

starter/test_building_datasets_and_labeling_lib.py (898 bytes)

import pytest
import numpy as np
from examples.building_datasets_and_labeling_lib import compute_shannon_entropy, compute_cohen_kappa, MajorityVoteLabelModel

def test_entropy_and_cohen_kappa():
    # 50/50 probability has maximum entropy 1.0
    p_equal = np.array([[0.5, 0.5]])
    ent = compute_shannon_entropy(p_equal)
    assert np.isclose(ent[0], 1.0, atol=1e-3)

    # Identical ratings have kappa 1.0
    y1 = np.array([1, 0, 1, 1, 0])
    y2 = np.array([1, 0, 1, 1, 0])
    assert compute_cohen_kappa(y1, y2) == 1.0

def test_majority_vote_label_model():
    # 3 samples, 3 LFs with votes {-1, +1, 0 (abstain)}
    L = np.array([
        [1, 1, -1],  # Sum = +1 -> Output +1
        [-1, -1, 0], # Sum = -2 -> Output -1
        [0, 0, 0]    # All abstain -> Output 0
    ])
    model = MajorityVoteLabelModel()
    preds = model.fit_predict(L)
    assert np.array_equal(preds, [1, -1, 0])

tests/run_tests.sh (227 bytes)

#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 191 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."

tests/test_building_datasets_and_labeling_lib.py (898 bytes)

import pytest
import numpy as np
from examples.building_datasets_and_labeling_lib import compute_shannon_entropy, compute_cohen_kappa, MajorityVoteLabelModel

def test_entropy_and_cohen_kappa():
    # 50/50 probability has maximum entropy 1.0
    p_equal = np.array([[0.5, 0.5]])
    ent = compute_shannon_entropy(p_equal)
    assert np.isclose(ent[0], 1.0, atol=1e-3)

    # Identical ratings have kappa 1.0
    y1 = np.array([1, 0, 1, 1, 0])
    y2 = np.array([1, 0, 1, 1, 0])
    assert compute_cohen_kappa(y1, y2) == 1.0

def test_majority_vote_label_model():
    # 3 samples, 3 LFs with votes {-1, +1, 0 (abstain)}
    L = np.array([
        [1, 1, -1],  # Sum = +1 -> Output +1
        [-1, -1, 0], # Sum = -2 -> Output -1
        [0, 0, 0]    # All abstain -> Output 0
    ])
    model = MajorityVoteLabelModel()
    preds = model.fit_predict(L)
    assert np.array_equal(preds, [1, -1, 0])

Troubleshooting

Troubleshooting: Day 191 - Building Datasets and Labeling

Common Issues

Division by Zero in Cohen Kappa:
- Cause: All samples belong to a single class making p_e == 1.0.
- Fix: Return 1.0 early if p_e is close to 1.0.

Security notes

Security & Privacy: Day 191 - Building Datasets and Labeling

Security Guidance

All data curation computations execute strictly on local CPU memory.

Hands-on lab — Day 191: Building Datasets and Labeling

Commands

Setup

Run

Test

File tree

Lab README

Lab: Day 191 -- Building Datasets and Labeling

Lesson

Purpose

Learning objectives

Prerequisites

Supported operating systems

Hardware requirements

Required software

Free and open-source options

Installation

File structure

How to run

What the commands do

Expected output

Validation steps

Tests

Cleanup

Troubleshooting

Security notes

Extension exercises

Navigation

Expected output

FIELDS.md

examples-run.txt

measured-values.txt

starter-run.txt

test-run.txt

Source files

Troubleshooting

Troubleshooting: Day 191 - Building Datasets and Labeling

Common Issues

Security notes

Security & Privacy: Day 191 - Building Datasets and Labeling

Security Guidance