LLMs and Generative AI › The LLM Landscape › Day 245

Hands-on lab — Day 245: Benchmarking Models Yourself

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/benchmarking_models_yourself_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/benchmarking_models_yourself_lib.py
examples/test_benchmarking_models_yourself_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/benchmarking_models_yourself_lib.py
starter/test_benchmarking_models_yourself_lib.py
tests/run_tests.sh
tests/test_benchmarking_models_yourself_lib.py
troubleshooting.md

Lab README

Lab: Day 245 -- Benchmarking Models Yourself

Lesson

Day number: 245 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Benchmarking Models Yourself, Bradley-Terry ELO, pass@k, and LLM-as-a-Judge.

Purpose

Build and test a complete modular LLM evaluation and benchmarking engine in Python. Calculate Bradley-Terry ELO tournament ratings across pairwise model battles, implement the unbiased combinatorial pass@k coding metric, and compute statistical confidence intervals.

Learning objectives

  • Formulate the Bradley-Terry ELO ranking update rules for pairwise LLM battles.
  • Implement the unbiased combinatorial pass@k estimation metric.
  • Design dual-pass position-swapped LLM-as-a-Judge harnesses.
  • Compute non-parametric bootstrap confidence intervals to evaluate statistical significance.

Prerequisites

  • Day 244 (Capabilities, Limits, and Hallucination).
  • Python 3.11+ with PyTorch, NumPy, and Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

NumPy and Pytest are free and open-source scientific software.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/benchmarking_models_yourself_lib.py: Student scaffold file.
  • examples/benchmarking_models_yourself_lib.py: Complete reference implementation.
  • tests/test_benchmarking_models_yourself_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/benchmarking_models_yourself_lib.py

What the commands do

  • Executes Bradley-Terry ELO updates across model matchups.
  • Calculates exact pass@1 and pass@3 metrics for code generation.
  • Runs unit test assertions.

Expected output

Benchmarking Demo: ELO A = 1016.0, ELO B = 984.0, pass@1 = 40.0%, pass@3 = 83.33%

Validation steps

  1. Verify update_match computes symmetric zero-sum ELO rating updates.
  2. Confirm compute_pass_at_k matches combinatorial baseline equations.
  3. Confirm edge cases ($c=0$ and $c=n$) evaluate cleanly.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Pass@k zero division: Ensure $n \ge k$ and $n > 0$.

Security notes

All benchmarking calculations run locally in process memory.

Extension exercises

  1. Implement a non-parametric Bootstrap confidence interval calculator.
  2. Build an automated multi-criteria rubric scoring harness.

Expected output

FIELDS.md

# Expected Output Fields: Day 245

- `ELO A`: Updated ELO rating for Model A after match victory.
- `ELO B`: Updated ELO rating for Model B after match defeat.
- `pass@1`: Estimated probability that a single sample passes unit tests.
- `pass@3`: Estimated probability that at least one of 3 samples passes unit tests.

examples-run.txt

Benchmarking Demo: ELO A = 1016.0, ELO B = 984.0, pass@1 = 40.0%, pass@3 = 83.33%

measured-values.txt

ELO A: 1016.0
ELO B: 984.0
pass@1: 40.0%
pass@3: 83.33%

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_benchmarking_models_yourself_lib.py::test_elo_update_symmetry PASSED [ 33%]
tests/test_benchmarking_models_yourself_lib.py::test_pass_at_k_exact_values PASSED [ 66%]
tests/test_benchmarking_models_yourself_lib.py::test_pass_at_k_edge_case PASSED [100%]

============================== 3 passed in 0.45s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/benchmarking_models_yourself_lib.py (1729 bytes)
import math
import numpy as np
from typing import Tuple, Dict, Any

class ELOEvaluator:
    def __init__(self, initial_elo: float = 1000.0, k_factor: float = 32.0):
        self.ratings: Dict[str, float] = {}
        self.initial_elo = initial_elo
        self.k = k_factor

    def get_rating(self, model: str) -> float:
        return self.ratings.get(model, self.initial_elo)

    def compute_expected_score(self, r_a: float, r_b: float) -> float:
        return 1.0 / (1.0 + 10.0 ** ((r_b - r_a) / 400.0))

    def update_match(self, model_a: str, model_b: str, score_a: float) -> Tuple[float, float]:
        r_a = self.get_rating(model_a)
        r_b = self.get_rating(model_b)

        e_a = self.compute_expected_score(r_a, r_b)
        e_b = 1.0 - e_a

        new_r_a = r_a + self.k * (score_a - e_a)
        new_r_b = r_b + self.k * ((1.0 - score_a) - e_b)

        self.ratings[model_a] = round(new_r_a, 2)
        self.ratings[model_b] = round(new_r_b, 2)
        return self.ratings[model_a], self.ratings[model_b]

def compute_pass_at_k(n: int, c: int, k: int) -> float:
    # 1 - comb(n - c, k) / comb(n, k)
    if n - c < k:
        return 1.0
    
    prod = 1.0
    for i in range(k):
        prod *= (n - c - i) / (n - i)
    return round(1.0 - prod, 4)

def run_benchmarking_demo():
    evaluator = ELOEvaluator()
    r_a, r_b = evaluator.update_match("Claude-3.5", "GPT-4o", 1.0) # Claude wins
    
    # 10 samples, 4 correct, pass@1 and pass@3
    p1 = compute_pass_at_k(10, 4, 1)
    p3 = compute_pass_at_k(10, 4, 3)

    print(f"Benchmarking Demo: ELO A = {r_a}, ELO B = {r_b}, pass@1 = {p1*100}%, pass@3 = {p3*100}%")
    return r_a, r_b, p1, p3

if __name__ == "__main__":
    run_benchmarking_demo()
examples/test_benchmarking_models_yourself_lib.py (886 bytes)
import pytest
from examples.benchmarking_models_yourself_lib import (
    ELOEvaluator,
    compute_pass_at_k
)

def test_elo_update_symmetry():
    evaluator = ELOEvaluator(initial_elo=1000.0, k_factor=32.0)
    # Match where Model A wins (score 1.0)
    r_a, r_b = evaluator.update_match("ModelA", "ModelB", 1.0)
    assert r_a == 1016.0
    assert r_b == 984.0
    assert (r_a + r_b) == 2000.0 # Zero-sum conservation

def test_pass_at_k_exact_values():
    # n = 10, c = 4, k = 1 -> 4/10 = 0.40
    assert compute_pass_at_k(10, 4, 1) == 0.40

    # n = 10, c = 4, k = 3 -> 1 - (6/10 * 5/9 * 4/8) = 1 - 0.16666 = 0.8333
    assert compute_pass_at_k(10, 4, 3) == pytest.approx(0.8333, rel=1e-3)

def test_pass_at_k_edge_case():
    # n = 5, c = 5, k = 1 -> 1.0
    assert compute_pass_at_k(5, 5, 1) == 1.0
    # n = 5, c = 0, k = 2 -> 0.0
    assert compute_pass_at_k(5, 0, 2) == 0.0
metadata.yml (441 bytes)
lesson_id: D245
day: 245
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/benchmarking_models_yourself_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/benchmarking_models_yourself_lib.py (552 bytes)
import math
import numpy as np
from typing import Tuple, Dict, Any

class ELOEvaluator:
    def __init__(self, initial_elo: float = 1000.0, k_factor: float = 32.0):
        self.ratings: Dict[str, float] = {}
        self.initial_elo = initial_elo
        self.k = k_factor

    def update_match(self, model_a: str, model_b: str, score_a: float) -> Tuple[float, float]:
        # TODO: Implement Bradley-Terry ELO update
        pass

def compute_pass_at_k(n: int, c: int, k: int) -> float:
    # TODO: Implement combinatorial unbiased pass@k
    pass
starter/test_benchmarking_models_yourself_lib.py (886 bytes)
import pytest
from examples.benchmarking_models_yourself_lib import (
    ELOEvaluator,
    compute_pass_at_k
)

def test_elo_update_symmetry():
    evaluator = ELOEvaluator(initial_elo=1000.0, k_factor=32.0)
    # Match where Model A wins (score 1.0)
    r_a, r_b = evaluator.update_match("ModelA", "ModelB", 1.0)
    assert r_a == 1016.0
    assert r_b == 984.0
    assert (r_a + r_b) == 2000.0 # Zero-sum conservation

def test_pass_at_k_exact_values():
    # n = 10, c = 4, k = 1 -> 4/10 = 0.40
    assert compute_pass_at_k(10, 4, 1) == 0.40

    # n = 10, c = 4, k = 3 -> 1 - (6/10 * 5/9 * 4/8) = 1 - 0.16666 = 0.8333
    assert compute_pass_at_k(10, 4, 3) == pytest.approx(0.8333, rel=1e-3)

def test_pass_at_k_edge_case():
    # n = 5, c = 5, k = 1 -> 1.0
    assert compute_pass_at_k(5, 5, 1) == 1.0
    # n = 5, c = 0, k = 2 -> 0.0
    assert compute_pass_at_k(5, 0, 2) == 0.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 245 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_benchmarking_models_yourself_lib.py (886 bytes)
import pytest
from examples.benchmarking_models_yourself_lib import (
    ELOEvaluator,
    compute_pass_at_k
)

def test_elo_update_symmetry():
    evaluator = ELOEvaluator(initial_elo=1000.0, k_factor=32.0)
    # Match where Model A wins (score 1.0)
    r_a, r_b = evaluator.update_match("ModelA", "ModelB", 1.0)
    assert r_a == 1016.0
    assert r_b == 984.0
    assert (r_a + r_b) == 2000.0 # Zero-sum conservation

def test_pass_at_k_exact_values():
    # n = 10, c = 4, k = 1 -> 4/10 = 0.40
    assert compute_pass_at_k(10, 4, 1) == 0.40

    # n = 10, c = 4, k = 3 -> 1 - (6/10 * 5/9 * 4/8) = 1 - 0.16666 = 0.8333
    assert compute_pass_at_k(10, 4, 3) == pytest.approx(0.8333, rel=1e-3)

def test_pass_at_k_edge_case():
    # n = 5, c = 5, k = 1 -> 1.0
    assert compute_pass_at_k(5, 5, 1) == 1.0
    # n = 5, c = 0, k = 2 -> 0.0
    assert compute_pass_at_k(5, 0, 2) == 0.0

Troubleshooting

Troubleshooting: Day 245 - Benchmarking Models Yourself

Common Issues

  1. Pass@k index error:
    • Cause: Looping $i$ beyond $k-1$.
    • Fix: Use range(k) to multiply exactly $k$ factor terms.

Security notes

Security & Privacy: Day 245 - Benchmarking Models Yourself

Security Guidance

  • All evaluation and ELO ranking calculations execute locally.