LLMs and Generative AI › The LLM Landscape › Day 245
Hands-on lab — Day 245: Benchmarking Models Yourself
- ← Back to the Day 245 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-245-benchmarking-models-yourself/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/benchmarking_models_yourself_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/benchmarking_models_yourself_lib.py examples/test_benchmarking_models_yourself_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/benchmarking_models_yourself_lib.py starter/test_benchmarking_models_yourself_lib.py tests/run_tests.sh tests/test_benchmarking_models_yourself_lib.py troubleshooting.md
Lab README
Lab: Day 245 -- Benchmarking Models Yourself
Lesson
Day number: 245 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Benchmarking Models Yourself, Bradley-Terry ELO, pass@k, and LLM-as-a-Judge.
Purpose
Build and test a complete modular LLM evaluation and benchmarking engine in Python. Calculate Bradley-Terry ELO tournament ratings across pairwise model battles, implement the unbiased combinatorial pass@k coding metric, and compute statistical confidence intervals.
Learning objectives
- Formulate the Bradley-Terry ELO ranking update rules for pairwise LLM battles.
- Implement the unbiased combinatorial pass@k estimation metric.
- Design dual-pass position-swapped LLM-as-a-Judge harnesses.
- Compute non-parametric bootstrap confidence intervals to evaluate statistical significance.
Prerequisites
- Day 244 (Capabilities, Limits, and Hallucination).
- Python 3.11+ with PyTorch, NumPy, and Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
NumPy and Pytest are free and open-source scientific software.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/benchmarking_models_yourself_lib.py: Student scaffold file.examples/benchmarking_models_yourself_lib.py: Complete reference implementation.tests/test_benchmarking_models_yourself_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/benchmarking_models_yourself_lib.py
What the commands do
- Executes Bradley-Terry ELO updates across model matchups.
- Calculates exact pass@1 and pass@3 metrics for code generation.
- Runs unit test assertions.
Expected output
Benchmarking Demo: ELO A = 1016.0, ELO B = 984.0, pass@1 = 40.0%, pass@3 = 83.33%
Validation steps
- Verify
update_matchcomputes symmetric zero-sum ELO rating updates. - Confirm
compute_pass_at_kmatches combinatorial baseline equations. - Confirm edge cases ($c=0$ and $c=n$) evaluate cleanly.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Pass@k zero division: Ensure $n \ge k$ and $n > 0$.
Security notes
All benchmarking calculations run locally in process memory.
Extension exercises
- Implement a non-parametric Bootstrap confidence interval calculator.
- Build an automated multi-criteria rubric scoring harness.
Navigation
- Lesson title: Benchmarking Models Yourself
- Day number: 245 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-245-benchmarking-models-yourself
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-245-benchmarking-models-yourselfwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 245
- `ELO A`: Updated ELO rating for Model A after match victory.
- `ELO B`: Updated ELO rating for Model B after match defeat.
- `pass@1`: Estimated probability that a single sample passes unit tests.
- `pass@3`: Estimated probability that at least one of 3 samples passes unit tests.
examples-run.txt
Benchmarking Demo: ELO A = 1016.0, ELO B = 984.0, pass@1 = 40.0%, pass@3 = 83.33%
measured-values.txt
ELO A: 1016.0
ELO B: 984.0
pass@1: 40.0%
pass@3: 83.33%
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_benchmarking_models_yourself_lib.py::test_elo_update_symmetry PASSED [ 33%]
tests/test_benchmarking_models_yourself_lib.py::test_pass_at_k_exact_values PASSED [ 66%]
tests/test_benchmarking_models_yourself_lib.py::test_pass_at_k_edge_case PASSED [100%]
============================== 3 passed in 0.45s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/benchmarking_models_yourself_lib.py (1729 bytes)
import math
import numpy as np
from typing import Tuple, Dict, Any
class ELOEvaluator:
def __init__(self, initial_elo: float = 1000.0, k_factor: float = 32.0):
self.ratings: Dict[str, float] = {}
self.initial_elo = initial_elo
self.k = k_factor
def get_rating(self, model: str) -> float:
return self.ratings.get(model, self.initial_elo)
def compute_expected_score(self, r_a: float, r_b: float) -> float:
return 1.0 / (1.0 + 10.0 ** ((r_b - r_a) / 400.0))
def update_match(self, model_a: str, model_b: str, score_a: float) -> Tuple[float, float]:
r_a = self.get_rating(model_a)
r_b = self.get_rating(model_b)
e_a = self.compute_expected_score(r_a, r_b)
e_b = 1.0 - e_a
new_r_a = r_a + self.k * (score_a - e_a)
new_r_b = r_b + self.k * ((1.0 - score_a) - e_b)
self.ratings[model_a] = round(new_r_a, 2)
self.ratings[model_b] = round(new_r_b, 2)
return self.ratings[model_a], self.ratings[model_b]
def compute_pass_at_k(n: int, c: int, k: int) -> float:
# 1 - comb(n - c, k) / comb(n, k)
if n - c < k:
return 1.0
prod = 1.0
for i in range(k):
prod *= (n - c - i) / (n - i)
return round(1.0 - prod, 4)
def run_benchmarking_demo():
evaluator = ELOEvaluator()
r_a, r_b = evaluator.update_match("Claude-3.5", "GPT-4o", 1.0) # Claude wins
# 10 samples, 4 correct, pass@1 and pass@3
p1 = compute_pass_at_k(10, 4, 1)
p3 = compute_pass_at_k(10, 4, 3)
print(f"Benchmarking Demo: ELO A = {r_a}, ELO B = {r_b}, pass@1 = {p1*100}%, pass@3 = {p3*100}%")
return r_a, r_b, p1, p3
if __name__ == "__main__":
run_benchmarking_demo()
examples/test_benchmarking_models_yourself_lib.py (886 bytes)
import pytest
from examples.benchmarking_models_yourself_lib import (
ELOEvaluator,
compute_pass_at_k
)
def test_elo_update_symmetry():
evaluator = ELOEvaluator(initial_elo=1000.0, k_factor=32.0)
# Match where Model A wins (score 1.0)
r_a, r_b = evaluator.update_match("ModelA", "ModelB", 1.0)
assert r_a == 1016.0
assert r_b == 984.0
assert (r_a + r_b) == 2000.0 # Zero-sum conservation
def test_pass_at_k_exact_values():
# n = 10, c = 4, k = 1 -> 4/10 = 0.40
assert compute_pass_at_k(10, 4, 1) == 0.40
# n = 10, c = 4, k = 3 -> 1 - (6/10 * 5/9 * 4/8) = 1 - 0.16666 = 0.8333
assert compute_pass_at_k(10, 4, 3) == pytest.approx(0.8333, rel=1e-3)
def test_pass_at_k_edge_case():
# n = 5, c = 5, k = 1 -> 1.0
assert compute_pass_at_k(5, 5, 1) == 1.0
# n = 5, c = 0, k = 2 -> 0.0
assert compute_pass_at_k(5, 0, 2) == 0.0
metadata.yml (441 bytes)
lesson_id: D245
day: 245
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/benchmarking_models_yourself_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/benchmarking_models_yourself_lib.py (552 bytes)
import math
import numpy as np
from typing import Tuple, Dict, Any
class ELOEvaluator:
def __init__(self, initial_elo: float = 1000.0, k_factor: float = 32.0):
self.ratings: Dict[str, float] = {}
self.initial_elo = initial_elo
self.k = k_factor
def update_match(self, model_a: str, model_b: str, score_a: float) -> Tuple[float, float]:
# TODO: Implement Bradley-Terry ELO update
pass
def compute_pass_at_k(n: int, c: int, k: int) -> float:
# TODO: Implement combinatorial unbiased pass@k
pass
starter/test_benchmarking_models_yourself_lib.py (886 bytes)
import pytest
from examples.benchmarking_models_yourself_lib import (
ELOEvaluator,
compute_pass_at_k
)
def test_elo_update_symmetry():
evaluator = ELOEvaluator(initial_elo=1000.0, k_factor=32.0)
# Match where Model A wins (score 1.0)
r_a, r_b = evaluator.update_match("ModelA", "ModelB", 1.0)
assert r_a == 1016.0
assert r_b == 984.0
assert (r_a + r_b) == 2000.0 # Zero-sum conservation
def test_pass_at_k_exact_values():
# n = 10, c = 4, k = 1 -> 4/10 = 0.40
assert compute_pass_at_k(10, 4, 1) == 0.40
# n = 10, c = 4, k = 3 -> 1 - (6/10 * 5/9 * 4/8) = 1 - 0.16666 = 0.8333
assert compute_pass_at_k(10, 4, 3) == pytest.approx(0.8333, rel=1e-3)
def test_pass_at_k_edge_case():
# n = 5, c = 5, k = 1 -> 1.0
assert compute_pass_at_k(5, 5, 1) == 1.0
# n = 5, c = 0, k = 2 -> 0.0
assert compute_pass_at_k(5, 0, 2) == 0.0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 245 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_benchmarking_models_yourself_lib.py (886 bytes)
import pytest
from examples.benchmarking_models_yourself_lib import (
ELOEvaluator,
compute_pass_at_k
)
def test_elo_update_symmetry():
evaluator = ELOEvaluator(initial_elo=1000.0, k_factor=32.0)
# Match where Model A wins (score 1.0)
r_a, r_b = evaluator.update_match("ModelA", "ModelB", 1.0)
assert r_a == 1016.0
assert r_b == 984.0
assert (r_a + r_b) == 2000.0 # Zero-sum conservation
def test_pass_at_k_exact_values():
# n = 10, c = 4, k = 1 -> 4/10 = 0.40
assert compute_pass_at_k(10, 4, 1) == 0.40
# n = 10, c = 4, k = 3 -> 1 - (6/10 * 5/9 * 4/8) = 1 - 0.16666 = 0.8333
assert compute_pass_at_k(10, 4, 3) == pytest.approx(0.8333, rel=1e-3)
def test_pass_at_k_edge_case():
# n = 5, c = 5, k = 1 -> 1.0
assert compute_pass_at_k(5, 5, 1) == 1.0
# n = 5, c = 0, k = 2 -> 0.0
assert compute_pass_at_k(5, 0, 2) == 0.0
Troubleshooting
Troubleshooting: Day 245 - Benchmarking Models Yourself
Common Issues
- Pass@k index error:
- Cause: Looping $i$ beyond $k-1$.
- Fix: Use
range(k)to multiply exactly $k$ factor terms.
Security notes
Security & Privacy: Day 245 - Benchmarking Models Yourself
Security Guidance
- All evaluation and ELO ranking calculations execute locally.