AI Engineering: Agents and Applications › Evaluation and Reliability › Day 311
Hands-on lab — Day 311: LLM-as-Judge
- ← Back to the Day 311 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-311-llm-as-judge/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/llm_judge.py Test
bash tests/run_tests.sh File tree
examples/llm_judge.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/llm_judge.py tests/run_tests.sh tests/test_llm_judge.py troubleshooting.md
Lab README
Day 311 Lab: LLM-as-Judge
Lesson
- Lesson title: LLM-as-Judge
- Day number: 311 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-311-llm-as-judge
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-311-llm-as-judgewhen the site is running.
Purpose
Build a Rubric-Anchored LLM-as-a-Judge Evaluation Engine in Python supporting prompt formatting, JSON response parsing, and position-swapped pairwise comparison.
Learning objectives
- Format structured rubric-anchored judge prompts for Faithfulness.
- Extract and validate JSON reasoning and score payloads.
- Implement position-swapped pairwise evaluation to eliminate positional bias.
- Handle malformed responses gracefully.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/llm_judge.py: Starter implementation skeletonexamples/llm_judge.py: Verified reference implementationtests/test_llm_judge.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/llm_judge.py
What the commands do
- Executes rubric parsing and position-swapped pairwise evaluation resolution.
Expected output
Parsed Judge Output: {'score': 5, 'reasoning': 'Fully grounded in doc 1', 'passed': True}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Prompt generation with anchored criteria
- Robust JSON response parsing
- Clamping of out-of-range scores
- Position-swapped pairwise resolution (candidate, baseline, tie, inconsistent)
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure JSON responses are extracted using regex boundary search.
Security notes
Runs completely offline on local CPU.
Extension exercises
Add multi-criteria composite scoring across Faithfulness, Relevance, and Conciseness.
Navigation
Day number: 311 of 365
Expected output
FIELDS.md
# Measured Values\n- `rubrics_tested`: 3
- `judge_accuracy`: 1
examples-run.txt
Parsed Judge Output: {'score': 5, 'reasoning': 'Fully grounded in doc 1', 'passed': True}
measured-values.txt
rubrics_tested=3
judge_accuracy=1
run-output.txt
Parsed Judge Output: {'score': 5, 'reasoning': 'Fully grounded in doc 1', 'passed': True}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
pass1_winner = 'A', pass2_winner = 'B'
@staticmethod
def resolve_pairwise_swap(pass1_winner: str, pass2_winner: str) -> str:
> raise NotImplementedError('TASK 3: implement resolve_pairwise_swap.')
E NotImplementedError: TASK 3: implement resolve_pairwise_swap.
examples/llm_judge.py:24: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_llm_judge.py::test_build_rubric_prompt - NotImplementedErro...
FAILED tests/test_llm_judge.py::test_parse_judge_response_valid - NotImplemen...
FAILED tests/test_llm_judge.py::test_parse_judge_response_with_markdown_wrapper
FAILED tests/test_llm_judge.py::test_parse_judge_response_malformed - NotImpl...
FAILED tests/test_llm_judge.py::test_resolve_pairwise_swap - NotImplementedEr...
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-311-llm-as-judge
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_llm_judge.py::test_build_rubric_prompt PASSED [ 20%]
tests/test_llm_judge.py::test_parse_judge_response_valid PASSED [ 40%]
tests/test_llm_judge.py::test_parse_judge_response_with_markdown_wrapper PASSED [ 60%]
tests/test_llm_judge.py::test_parse_judge_response_malformed PASSED [ 80%]
tests/test_llm_judge.py::test_resolve_pairwise_swap PASSED [100%]
============================== 5 passed in 0.02s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-311-llm-as-judge
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_llm_judge.py::test_build_rubric_prompt PASSED [ 20%]
tests/test_llm_judge.py::test_parse_judge_response_valid PASSED [ 40%]
tests/test_llm_judge.py::test_parse_judge_response_with_markdown_wrapper PASSED [ 60%]
tests/test_llm_judge.py::test_parse_judge_response_malformed PASSED [ 80%]
tests/test_llm_judge.py::test_resolve_pairwise_swap PASSED [100%]
============================== 5 passed in 0.02s ===============================
Source files
examples/llm_judge.py (2659 bytes)
import json
import re
from typing import Dict, Any
class LLMJudgeEvaluator:
@staticmethod
def build_rubric_prompt(query: str, context: str, candidate_answer: str) -> str:
prompt_template = (
"You are an expert impartial evaluation judge. Grade the candidate response on a 1-5 scale for FAITHFULNESS.\n\n"
"Evaluation Rubric:\n"
"- Score 1: Completely ungrounded or contains severe factual hallucinations.\n"
"- Score 2: Major claims cannot be verified against the context.\n"
"- Score 3: Partially grounded, but includes minor unsupported extrapolations.\n"
"- Score 4: Mostly grounded with only trivial non-factual filler.\n"
"- Score 5: 100% faithful and fully supported by the context documents.\n\n"
f"Input Context:\n{context}\n\n"
f"User Query:\n{query}\n\n"
f"Candidate Response:\n{candidate_answer}\n\n"
"Respond ONLY with a JSON object in this exact schema:\n"
'{\n "reasoning": "<step-by-step critique>",\n "score": <integer 1 to 5>,\n "passed": <boolean>\n}'
)
return prompt_template
@staticmethod
def parse_judge_response(raw_response: str) -> Dict[str, Any]:
try:
match = re.search(r'\{.*\}', raw_response, re.DOTALL)
if match:
data = json.loads(match.group(0))
if "score" in data and "reasoning" in data:
score = int(data["score"])
clamped_score = max(1, min(5, score))
return {
"score": clamped_score,
"reasoning": str(data["reasoning"]).strip(),
"passed": clamped_score >= 4
}
except Exception:
pass
return {"score": 1, "reasoning": "Failed to parse judge output JSON", "passed": False}
@staticmethod
def resolve_pairwise_swap(pass1_winner: str, pass2_winner: str) -> str:
norm_p1 = str(pass1_winner).strip().upper()
norm_p2 = str(pass2_winner).strip().upper()
if norm_p1 == "A" and norm_p2 == "B":
return "candidate"
elif norm_p1 == "B" and norm_p2 == "A":
return "baseline"
elif norm_p1 == "TIE" or norm_p2 == "TIE":
return "tie"
else:
return "position_bias_inconsistent"
if __name__ == "__main__":
judge = LLMJudgeEvaluator()
sample_json = '```json\n{"reasoning": "Fully grounded in doc 1", "score": 5}\n```'
print("Parsed Judge Output:", judge.parse_judge_response(sample_json))
metadata.yml (396 bytes)
lesson_id: D311
day: 311
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/llm_judge.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/llm_judge.py (1091 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import json
import re
from typing import Dict, Any
class LLMJudgeEvaluator:
@staticmethod
def build_rubric_prompt(query: str, context: str, candidate_answer: str) -> str:
raise NotImplementedError('TASK 1: implement build_rubric_prompt.')
@staticmethod
def parse_judge_response(raw_response: str) -> Dict[str, Any]:
raise NotImplementedError('TASK 2: implement parse_judge_response.')
@staticmethod
def resolve_pairwise_swap(pass1_winner: str, pass2_winner: str) -> str:
raise NotImplementedError('TASK 3: implement resolve_pairwise_swap.')
if __name__ == '__main__':
judge = LLMJudgeEvaluator()
sample_json = '```json\n{"reasoning": "Fully grounded in doc 1", "score": 5}\n```'
print('Parsed Judge Output:', judge.parse_judge_response(sample_json))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_llm_judge.py (1419 bytes)
import pytest
from examples.llm_judge import LLMJudgeEvaluator
def test_build_rubric_prompt():
prompt = LLMJudgeEvaluator.build_rubric_prompt("Query", "Context", "Answer")
assert "FAITHFULNESS" in prompt
assert "Evaluation Rubric:" in prompt
assert "Query" in prompt
assert "Context" in prompt
def test_parse_judge_response_valid():
raw = '{"reasoning": "All facts match context.", "score": 5, "passed": true}'
res = LLMJudgeEvaluator.parse_judge_response(raw)
assert res["score"] == 5
assert res["passed"] is True
assert "All facts match" in res["reasoning"]
def test_parse_judge_response_with_markdown_wrapper():
raw = 'Here is the evaluation:\n```json\n{"reasoning": "Minor issue", "score": 3}\n```'
res = LLMJudgeEvaluator.parse_judge_response(raw)
assert res["score"] == 3
assert res["passed"] is False
def test_parse_judge_response_malformed():
raw = "I think the answer is okay."
res = LLMJudgeEvaluator.parse_judge_response(raw)
assert res["score"] == 1
assert res["passed"] is False
def test_resolve_pairwise_swap():
assert LLMJudgeEvaluator.resolve_pairwise_swap("A", "B") == "candidate"
assert LLMJudgeEvaluator.resolve_pairwise_swap("B", "A") == "baseline"
assert LLMJudgeEvaluator.resolve_pairwise_swap("A", "A") == "position_bias_inconsistent"
assert LLMJudgeEvaluator.resolve_pairwise_swap("TIE", "B") == "tie"