Sandeep Bazar 365 Days of AI
Labs

Machine Learning › Evaluation and Interpretation › Day 181

Hands-on lab — Day 181: Baselines and Error Analysis

Commands

Setup

python3 -m venv .venv
.venv/bin/pip install -r requirements/requirements.txt

Run

.venv/bin/python examples/baselines_and_error_analysis_lib.py

Test

./tests/run_tests.sh

File tree

examples/error_analysis_lib.py
examples/test_error_analysis_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/error_analysis_lib.py
starter/test_error_analysis_lib.py
tests/run_tests.sh
troubleshooting.md

Lab README

Day 181 Lab: Baselines and Error Analysis

Day number: 181 of 365.

Lesson

Covering day-181-baselines-and-error-analysis.

Purpose

Master 4-tier baseline hierarchy, andrew ng error reduction ceilings, slice performance audits, and data-centric iteration. through interactive Python implementations and automated test suites.

Learning objectives

  • Implement core mathematical algorithms for baselines and error analysis.
  • Benchmark models against rigorous baselines.
  • Execute automated unit and integration tests.
  • Analyze failure modes and edge cases.

Prerequisites

  • Python 3.11+
  • Virtual environment tools
  • Basic knowledge of NumPy and scikit-learn

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu 22.04+, Debian, Fedora, Arch)
  • Windows (WSL2 recommended)

Hardware requirements

  • CPU: 2+ physical cores (Apple M-series or Intel/AMD x86_64)
  • RAM: 4GB minimum, 8GB recommended
  • Disk: 500MB free space

Required software

  • Python 3.11 or higher
  • Git
  • Bash shell

Free and open-source options

  • Python: python.org (PSFL)
  • scikit-learn: BSD 3-Clause
  • pytest: MIT License

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/: Scaffolded implementation files for student completion.
  • examples/: Fully functional reference library implementation.
  • tests/: Pytest suite and shell validation runners.
  • expected-output/: Captured reference terminal logs.
  • requirements/: Python package dependency specifications.
  • troubleshooting.md: Common runtime failure solutions.
  • security.md: Local execution safety guidance.

How to run

python3 examples/baselines_and_error_analysis_lib.py

What the commands do

  • Executes reference implementation demonstration and benchmarks.

Expected output

Reference logs are captured in expected-output/run-output.txt and expected-output/test-output.txt.

Validation steps

  1. Run ./tests/run_tests.sh.
  2. Ensure exit code is 0.

Tests

pytest tests/ -v

Cleanup

rm -rf .venv __pycache__ .pytest_cache

Troubleshooting

Refer to troubleshooting.md for common import or version issues.

Security notes

Refer to security.md for isolation and data safety guidance.

Extension exercises

  • Test on imbalanced real-world datasets.
  • Profile runtime latency and memory utilization.

Expected output

FIELDS.md

# Output Fields
- accuracy, f1_score, lift_over_dummy, lift_over_linear, beats_all_baselines
- total_count, error_count, error_rate
- error_category, error_count, pct_of_total_errors, max_potential_accuracy_gain

examples-run.txt

=== Baseline Hierarchy Benchmark ===
1. Dummy Majority Accuracy:  0.4850
2. Linear Logistic Accuracy: 0.9050
3. Candidate Model Accuracy: 0.9050
Lift Over Linear Baseline:   +0.0000

=== Slice Performance Audit ===
Tier: Free       | Samples: 74  | Error Rate: 0.0946
Tier: Standard   | Samples: 63  | Error Rate: 0.1111
Tier: VIP        | Samples: 63  | Error Rate: 0.0794

=== Error Reduction Ceiling (Andrew Ng) ===
Category: Ambiguous Ground Truth    | % of Errors: 50.0% | Max Gain: +7.50%
Category: Rare Subclass Outlier     | % of Errors: 33.3% | Max Gain: +5.00%
Category: Feature Missingness       | % of Errors: 16.7% | Max Gain: +2.50%

measured-values.txt

=== Baseline Hierarchy Benchmark ===
1. Dummy Majority Accuracy:  0.4850
2. Linear Logistic Accuracy: 0.9050
3. Candidate Model Accuracy: 0.9050
Lift Over Linear Baseline:   +0.0000

=== Slice Performance Audit ===
Tier: Free       | Samples: 74  | Error Rate: 0.0946
Tier: Standard   | Samples: 63  | Error Rate: 0.1111
Tier: VIP        | Samples: 63  | Error Rate: 0.0794

=== Error Reduction Ceiling (Andrew Ng) ===
Category: Ambiguous Ground Truth    | % of Errors: 50.0% | Max Gain: +7.50%
Category: Rare Subclass Outlier     | % of Errors: 33.3% | Max Gain: +5.00%
Category: Feature Missingness       | % of Errors: 16.7% | Max Gain: +2.50%

starter-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>
collecting ... collected 3 items

labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py::test_compute_baseline_benchmarks FAILED [ 33%]
labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py::test_compute_error_slices FAILED [ 66%]
labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py::test_error_reduction_ceiling FAILED [100%]

=================================== FAILURES ===================================
_______________________ test_compute_baseline_benchmarks _______________________

    def test_compute_baseline_benchmarks():
        rng = np.random.default_rng(42)
        X_train = rng.normal(size=(200, 3))
        y_train = (X_train[:, 0] + X_train[:, 1] > 0).astype(int)
        X_test = rng.normal(size=(50, 3))
        y_test = (X_test[:, 0] + X_test[:, 1] > 0).astype(int)
    
        rf = RandomForestClassifier(n_estimators=10, random_state=42).fit(X_train, y_train)
        bench = compute_baseline_benchmarks(X_train, y_train, X_test, y_test, candidate_model=rf)
    
>       assert "dummy_majority" in bench
               ^^^^^^^^^^^^^^^^^^^^^^^^^
E       TypeError: argument of type 'NoneType' is not a container or iterable

labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py:21: TypeError
__________________________ test_compute_error_slices ___________________________

    def test_compute_error_slices():
        df = pd.DataFrame({
            "device_os": ["iOS", "iOS", "Android", "Android", "Android"],
            "user_tier": ["Free", "VIP", "Free", "VIP", "Free"],
            "y_true": [1, 1, 0, 1, 0],
            "y_pred": [1, 0, 0, 0, 1] # Errors on rows 1 (iOS/VIP), 3 (Android/VIP), 4 (Android/Free)
        })
    
        slices = compute_error_slices(df, "y_true", "y_pred", ["device_os", "user_tier"])
>       assert "device_os" in slices
               ^^^^^^^^^^^^^^^^^^^^^
E       TypeError: argument of type 'NoneType' is not a container or iterable

labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py:35: TypeError
_________________________ test_error_reduction_ceiling _________________________

    def test_error_reduction_ceiling():
        tags = {
            "Label Noise": 40,
            "Audio Background Glitch": 30,
            "Rare Dialect": 10
        }
        ceilings = compute_error_reduction_ceiling(tags, total_sample_count=1000, baseline_error_count=80)
>       assert len(ceilings) == 3
               ^^^^^^^^^^^^^
E       TypeError: object of type 'NoneType' has no len()

labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py:48: TypeError
=========================== short test summary info ============================
FAILED labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py::test_compute_baseline_benchmarks
FAILED labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py::test_compute_error_slices
FAILED labs/sections/machine-learning/day-181-baselines-and-error-analysis/starter/test_error_analysis_lib.py::test_error_reduction_ceiling
============================== 3 failed in 0.78s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>
collecting ... collected 3 items

labs/sections/machine-learning/day-181-baselines-and-error-analysis/examples/test_error_analysis_lib.py::test_compute_baseline_benchmarks PASSED [ 33%]
labs/sections/machine-learning/day-181-baselines-and-error-analysis/examples/test_error_analysis_lib.py::test_compute_error_slices PASSED [ 66%]
labs/sections/machine-learning/day-181-baselines-and-error-analysis/examples/test_error_analysis_lib.py::test_error_reduction_ceiling PASSED [100%]

============================== 3 passed in 4.07s ===============================

Source files

examples/error_analysis_lib.py (3270 bytes) 
import numpy as np
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

def compute_baseline_benchmarks(X_train, y_train, X_test, y_test, candidate_model=None):
    """
    Evaluate candidate model against the strict 3-tier baseline hierarchy:
    1. Trivial Majority Baseline (DummyClassifier)
    2. Linear Baseline (LogisticRegression)
    3. Candidate Model (e.g. Tree Ensemble)
    """
    # 1. Dummy Majority
    dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
    dummy_pred = dummy.predict(X_test)
    dummy_acc = float(accuracy_score(y_test, dummy_pred))
    dummy_f1 = float(f1_score(y_test, dummy_pred, zero_division=0))
    
    # 2. Linear Baseline
    linear = LogisticRegression(max_iter=1000).fit(X_train, y_train)
    linear_pred = linear.predict(X_test)
    linear_acc = float(accuracy_score(y_test, linear_pred))
    linear_f1 = float(f1_score(y_test, linear_pred, zero_division=0))
    
    results = {
        "dummy_majority": {"accuracy": dummy_acc, "f1_score": dummy_f1},
        "linear_baseline": {"accuracy": linear_acc, "f1_score": linear_f1}
    }
    
    if candidate_model is not None:
        cand_pred = candidate_model.predict(X_test)
        cand_acc = float(accuracy_score(y_test, cand_pred))
        cand_f1 = float(f1_score(y_test, cand_pred, zero_division=0))
        results["candidate_model"] = {
            "accuracy": cand_acc,
            "f1_score": cand_f1,
            "lift_over_dummy": float(cand_acc - dummy_acc),
            "lift_over_linear": float(cand_acc - linear_acc),
            "beats_all_baselines": (cand_acc > linear_acc) and (cand_acc > dummy_acc)
        }
        
    return results

def compute_error_slices(df_test, y_true_col, y_pred_col, slice_cols):
    """
    Compute slice-specific error rates across demographic and operational segments.
    """
    df = df_test.copy()
    df["is_error"] = (df[y_true_col] != df[y_pred_col]).astype(int)
    
    slice_reports = {}
    
    for col in slice_cols:
        grouped = df.groupby(col).agg(
            total_count=("is_error", "count"),
            error_count=("is_error", "sum"),
            error_rate=("is_error", "mean")
        ).reset_index()
        
        slice_reports[col] = grouped.to_dict(orient="records")
        
    return slice_reports

def compute_error_reduction_ceiling(error_tag_counts, total_sample_count, baseline_error_count):
    """
    Compute Andrew Ng's Error Reduction Ceiling:
    What is the maximum potential accuracy improvement if an error category is 100% fixed?
    """
    ceilings = []
    
    for tag, count in error_tag_counts.items():
        pct_of_errors = count / max(baseline_error_count, 1)
        max_accuracy_gain = count / max(total_sample_count, 1)
        
        ceilings.append({
            "error_category": tag,
            "error_count": int(count),
            "pct_of_total_errors": float(pct_of_errors),
            "max_potential_accuracy_gain": float(max_accuracy_gain)
        })
        
    # Sort descending by impact
    ceilings.sort(key=lambda x: x["max_potential_accuracy_gain"], reverse=True)
    return ceilings
examples/test_error_analysis_lib.py (2024 bytes) 
import pytest
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from error_analysis_lib import (
    compute_baseline_benchmarks,
    compute_error_slices,
    compute_error_reduction_ceiling
)

def test_compute_baseline_benchmarks():
    rng = np.random.default_rng(42)
    X_train = rng.normal(size=(200, 3))
    y_train = (X_train[:, 0] + X_train[:, 1] > 0).astype(int)
    X_test = rng.normal(size=(50, 3))
    y_test = (X_test[:, 0] + X_test[:, 1] > 0).astype(int)
    
    rf = RandomForestClassifier(n_estimators=10, random_state=42).fit(X_train, y_train)
    bench = compute_baseline_benchmarks(X_train, y_train, X_test, y_test, candidate_model=rf)
    
    assert "dummy_majority" in bench
    assert "linear_baseline" in bench
    assert "candidate_model" in bench
    assert bench["candidate_model"]["accuracy"] >= bench["dummy_majority"]["accuracy"]

def test_compute_error_slices():
    df = pd.DataFrame({
        "device_os": ["iOS", "iOS", "Android", "Android", "Android"],
        "user_tier": ["Free", "VIP", "Free", "VIP", "Free"],
        "y_true": [1, 1, 0, 1, 0],
        "y_pred": [1, 0, 0, 0, 1] # Errors on rows 1 (iOS/VIP), 3 (Android/VIP), 4 (Android/Free)
    })
    
    slices = compute_error_slices(df, "y_true", "y_pred", ["device_os", "user_tier"])
    assert "device_os" in slices
    assert "user_tier" in slices
    # VIP error rate is 2/2 = 1.0
    vip_report = [s for s in slices["user_tier"] if s["user_tier"] == "VIP"][0]
    assert vip_report["error_rate"] == 1.0

def test_error_reduction_ceiling():
    tags = {
        "Label Noise": 40,
        "Audio Background Glitch": 30,
        "Rare Dialect": 10
    }
    ceilings = compute_error_reduction_ceiling(tags, total_sample_count=1000, baseline_error_count=80)
    assert len(ceilings) == 3
    assert ceilings[0]["error_category"] == "Label Noise"
    assert ceilings[0]["pct_of_total_errors"] == 0.50 # 40 / 80
    assert ceilings[0]["max_potential_accuracy_gain"] == 0.04 # 40 / 1000
metadata.yml (660 bytes) 
lesson_id: D181
day: 181
kind: applied-ml-error-analysis
languages:
  - python
setup_commands:
  - python3 -m venv .venv
  - .venv/bin/pip install -r requirements/requirements.txt
run_commands:
  - .venv/bin/python examples/baselines_and_error_analysis_lib.py
test_commands:
  - ./tests/run_tests.sh
cleanup_commands:
  - rm -rf .venv __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: >-
  macOS (Apple Silicon, arm64, CPU only), Python 3.14.0, scikit-learn 1.9.0, pytest 9.1.1 -- bash tests/run_tests.sh -> 4 checks, 0 failure(s), exit 0. Verified Day 181 implementation.
requirements/requirements.txt (76 bytes) 
numpy>=1.24.0
scipy>=1.10.0
pandas>=2.0.0
scikit-learn>=1.3.0
pytest>=7.4.0
starter/error_analysis_lib.py (580 bytes) 
import numpy as np
import pandas as pd

def compute_baseline_benchmarks(X_train, y_train, X_test, y_test, candidate_model=None):
    # TODO: Implement 3-tier baseline evaluation (Dummy, Linear, Candidate)
    pass

def compute_error_slices(df_test, y_true_col, y_pred_col, slice_cols):
    # TODO: Calculate slice-specific error rates across demographic/business segments
    pass

def compute_error_reduction_ceiling(error_tag_counts, total_sample_count, baseline_error_count):
    # TODO: Calculate Andrew Ng error reduction ceiling and maximum potential accuracy gain
    pass
starter/test_error_analysis_lib.py (2024 bytes) 
import pytest
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from error_analysis_lib import (
    compute_baseline_benchmarks,
    compute_error_slices,
    compute_error_reduction_ceiling
)

def test_compute_baseline_benchmarks():
    rng = np.random.default_rng(42)
    X_train = rng.normal(size=(200, 3))
    y_train = (X_train[:, 0] + X_train[:, 1] > 0).astype(int)
    X_test = rng.normal(size=(50, 3))
    y_test = (X_test[:, 0] + X_test[:, 1] > 0).astype(int)
    
    rf = RandomForestClassifier(n_estimators=10, random_state=42).fit(X_train, y_train)
    bench = compute_baseline_benchmarks(X_train, y_train, X_test, y_test, candidate_model=rf)
    
    assert "dummy_majority" in bench
    assert "linear_baseline" in bench
    assert "candidate_model" in bench
    assert bench["candidate_model"]["accuracy"] >= bench["dummy_majority"]["accuracy"]

def test_compute_error_slices():
    df = pd.DataFrame({
        "device_os": ["iOS", "iOS", "Android", "Android", "Android"],
        "user_tier": ["Free", "VIP", "Free", "VIP", "Free"],
        "y_true": [1, 1, 0, 1, 0],
        "y_pred": [1, 0, 0, 0, 1] # Errors on rows 1 (iOS/VIP), 3 (Android/VIP), 4 (Android/Free)
    })
    
    slices = compute_error_slices(df, "y_true", "y_pred", ["device_os", "user_tier"])
    assert "device_os" in slices
    assert "user_tier" in slices
    # VIP error rate is 2/2 = 1.0
    vip_report = [s for s in slices["user_tier"] if s["user_tier"] == "VIP"][0]
    assert vip_report["error_rate"] == 1.0

def test_error_reduction_ceiling():
    tags = {
        "Label Noise": 40,
        "Audio Background Glitch": 30,
        "Rare Dialect": 10
    }
    ceilings = compute_error_reduction_ceiling(tags, total_sample_count=1000, baseline_error_count=80)
    assert len(ceilings) == 3
    assert ceilings[0]["error_category"] == "Label Noise"
    assert ceilings[0]["pct_of_total_errors"] == 0.50 # 40 / 80
    assert ceilings[0]["max_potential_accuracy_gain"] == 0.04 # 40 / 1000
tests/run_tests.sh (36 bytes) 
#!/bin/bash
set -e
pytest tests/ -v

Troubleshooting

Troubleshooting Model Baselines & Error Analysis

1. Complex Model Fails to Beat Linear Baseline

If a 500-tree Gradient Booster achieves identical test score to Logistic Regression, your problem is linearly separable or features lack non-linear signal. Simplify architecture to reduce latency and maintenance costs.

2. Slices with Too Few Samples

Do not draw conclusions from slices with $N < 30$ samples. Use confidence intervals to evaluate slice error rates.

Security notes

Security Considerations for Error Analysis

1. Targeted Adversarial Slices

Adversaries target unmonitored feature slices where model accuracy drops below 50% to execute evasion attacks. Always audit slice-based performance across rare inputs.

2. Privacy Leakage in Error Logs

When logging misclassified user inputs for manual human auditing, redact all PII (names, SSNs, credit card numbers) before writing to annotation databases.