AI Engineering: Agents and Applications › Evaluation and Reliability › Day 311

Hands-on lab — Day 311: LLM-as-Judge

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/llm_judge.py

Test

bash tests/run_tests.sh

File tree

examples/llm_judge.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/llm_judge.py
tests/run_tests.sh
tests/test_llm_judge.py
troubleshooting.md

Lab README

Day 311 Lab: LLM-as-Judge

Lesson

  • Lesson title: LLM-as-Judge
  • Day number: 311 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-311-llm-as-judge
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-311-llm-as-judge when the site is running.

Purpose

Build a Rubric-Anchored LLM-as-a-Judge Evaluation Engine in Python supporting prompt formatting, JSON response parsing, and position-swapped pairwise comparison.

Learning objectives

  • Format structured rubric-anchored judge prompts for Faithfulness.
  • Extract and validate JSON reasoning and score payloads.
  • Implement position-swapped pairwise evaluation to eliminate positional bias.
  • Handle malformed responses gracefully.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/llm_judge.py: Starter implementation skeleton
  • examples/llm_judge.py: Verified reference implementation
  • tests/test_llm_judge.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/llm_judge.py

What the commands do

  • Executes rubric parsing and position-swapped pairwise evaluation resolution.

Expected output

Parsed Judge Output: {'score': 5, 'reasoning': 'Fully grounded in doc 1', 'passed': True}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Prompt generation with anchored criteria
  • Robust JSON response parsing
  • Clamping of out-of-range scores
  • Position-swapped pairwise resolution (candidate, baseline, tie, inconsistent)

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure JSON responses are extracted using regex boundary search.

Security notes

Runs completely offline on local CPU.

Extension exercises

Add multi-criteria composite scoring across Faithfulness, Relevance, and Conciseness.

Day number: 311 of 365

Expected output

FIELDS.md

# Measured Values\n- `rubrics_tested`: 3
- `judge_accuracy`: 1

examples-run.txt

Parsed Judge Output: {'score': 5, 'reasoning': 'Fully grounded in doc 1', 'passed': True}

measured-values.txt

rubrics_tested=3
judge_accuracy=1

run-output.txt

Parsed Judge Output: {'score': 5, 'reasoning': 'Fully grounded in doc 1', 'passed': True}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


pass1_winner = 'A', pass2_winner = 'B'

    @staticmethod
    def resolve_pairwise_swap(pass1_winner: str, pass2_winner: str) -> str:
>       raise NotImplementedError('TASK 3: implement resolve_pairwise_swap.')
E       NotImplementedError: TASK 3: implement resolve_pairwise_swap.

examples/llm_judge.py:24: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_llm_judge.py::test_build_rubric_prompt - NotImplementedErro...
FAILED tests/test_llm_judge.py::test_parse_judge_response_valid - NotImplemen...
FAILED tests/test_llm_judge.py::test_parse_judge_response_with_markdown_wrapper
FAILED tests/test_llm_judge.py::test_parse_judge_response_malformed - NotImpl...
FAILED tests/test_llm_judge.py::test_resolve_pairwise_swap - NotImplementedEr...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-311-llm-as-judge
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_llm_judge.py::test_build_rubric_prompt PASSED                 [ 20%]
tests/test_llm_judge.py::test_parse_judge_response_valid PASSED          [ 40%]
tests/test_llm_judge.py::test_parse_judge_response_with_markdown_wrapper PASSED [ 60%]
tests/test_llm_judge.py::test_parse_judge_response_malformed PASSED      [ 80%]
tests/test_llm_judge.py::test_resolve_pairwise_swap PASSED               [100%]

============================== 5 passed in 0.02s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-311-llm-as-judge
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_llm_judge.py::test_build_rubric_prompt PASSED                 [ 20%]
tests/test_llm_judge.py::test_parse_judge_response_valid PASSED          [ 40%]
tests/test_llm_judge.py::test_parse_judge_response_with_markdown_wrapper PASSED [ 60%]
tests/test_llm_judge.py::test_parse_judge_response_malformed PASSED      [ 80%]
tests/test_llm_judge.py::test_resolve_pairwise_swap PASSED               [100%]

============================== 5 passed in 0.02s ===============================

Source files

examples/llm_judge.py (2659 bytes)
import json
import re
from typing import Dict, Any

class LLMJudgeEvaluator:
    @staticmethod
    def build_rubric_prompt(query: str, context: str, candidate_answer: str) -> str:
        prompt_template = (
            "You are an expert impartial evaluation judge. Grade the candidate response on a 1-5 scale for FAITHFULNESS.\n\n"
            "Evaluation Rubric:\n"
            "- Score 1: Completely ungrounded or contains severe factual hallucinations.\n"
            "- Score 2: Major claims cannot be verified against the context.\n"
            "- Score 3: Partially grounded, but includes minor unsupported extrapolations.\n"
            "- Score 4: Mostly grounded with only trivial non-factual filler.\n"
            "- Score 5: 100% faithful and fully supported by the context documents.\n\n"
            f"Input Context:\n{context}\n\n"
            f"User Query:\n{query}\n\n"
            f"Candidate Response:\n{candidate_answer}\n\n"
            "Respond ONLY with a JSON object in this exact schema:\n"
            '{\n  "reasoning": "<step-by-step critique>",\n  "score": <integer 1 to 5>,\n  "passed": <boolean>\n}'
        )
        return prompt_template

    @staticmethod
    def parse_judge_response(raw_response: str) -> Dict[str, Any]:
        try:
            match = re.search(r'\{.*\}', raw_response, re.DOTALL)
            if match:
                data = json.loads(match.group(0))
                if "score" in data and "reasoning" in data:
                    score = int(data["score"])
                    clamped_score = max(1, min(5, score))
                    return {
                        "score": clamped_score,
                        "reasoning": str(data["reasoning"]).strip(),
                        "passed": clamped_score >= 4
                    }
        except Exception:
            pass
        return {"score": 1, "reasoning": "Failed to parse judge output JSON", "passed": False}

    @staticmethod
    def resolve_pairwise_swap(pass1_winner: str, pass2_winner: str) -> str:
        norm_p1 = str(pass1_winner).strip().upper()
        norm_p2 = str(pass2_winner).strip().upper()
        
        if norm_p1 == "A" and norm_p2 == "B":
            return "candidate"
        elif norm_p1 == "B" and norm_p2 == "A":
            return "baseline"
        elif norm_p1 == "TIE" or norm_p2 == "TIE":
            return "tie"
        else:
            return "position_bias_inconsistent"

if __name__ == "__main__":
    judge = LLMJudgeEvaluator()
    sample_json = '```json\n{"reasoning": "Fully grounded in doc 1", "score": 5}\n```'
    print("Parsed Judge Output:", judge.parse_judge_response(sample_json))
metadata.yml (396 bytes)
lesson_id: D311
day: 311
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/llm_judge.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/llm_judge.py (1091 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import json
import re
from typing import Dict, Any

class LLMJudgeEvaluator:

    @staticmethod
    def build_rubric_prompt(query: str, context: str, candidate_answer: str) -> str:
        raise NotImplementedError('TASK 1: implement build_rubric_prompt.')

    @staticmethod
    def parse_judge_response(raw_response: str) -> Dict[str, Any]:
        raise NotImplementedError('TASK 2: implement parse_judge_response.')

    @staticmethod
    def resolve_pairwise_swap(pass1_winner: str, pass2_winner: str) -> str:
        raise NotImplementedError('TASK 3: implement resolve_pairwise_swap.')
if __name__ == '__main__':
    judge = LLMJudgeEvaluator()
    sample_json = '```json\n{"reasoning": "Fully grounded in doc 1", "score": 5}\n```'
    print('Parsed Judge Output:', judge.parse_judge_response(sample_json))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_llm_judge.py (1419 bytes)
import pytest
from examples.llm_judge import LLMJudgeEvaluator

def test_build_rubric_prompt():
    prompt = LLMJudgeEvaluator.build_rubric_prompt("Query", "Context", "Answer")
    assert "FAITHFULNESS" in prompt
    assert "Evaluation Rubric:" in prompt
    assert "Query" in prompt
    assert "Context" in prompt

def test_parse_judge_response_valid():
    raw = '{"reasoning": "All facts match context.", "score": 5, "passed": true}'
    res = LLMJudgeEvaluator.parse_judge_response(raw)
    assert res["score"] == 5
    assert res["passed"] is True
    assert "All facts match" in res["reasoning"]

def test_parse_judge_response_with_markdown_wrapper():
    raw = 'Here is the evaluation:\n```json\n{"reasoning": "Minor issue", "score": 3}\n```'
    res = LLMJudgeEvaluator.parse_judge_response(raw)
    assert res["score"] == 3
    assert res["passed"] is False

def test_parse_judge_response_malformed():
    raw = "I think the answer is okay."
    res = LLMJudgeEvaluator.parse_judge_response(raw)
    assert res["score"] == 1
    assert res["passed"] is False

def test_resolve_pairwise_swap():
    assert LLMJudgeEvaluator.resolve_pairwise_swap("A", "B") == "candidate"
    assert LLMJudgeEvaluator.resolve_pairwise_swap("B", "A") == "baseline"
    assert LLMJudgeEvaluator.resolve_pairwise_swap("A", "A") == "position_bias_inconsistent"
    assert LLMJudgeEvaluator.resolve_pairwise_swap("TIE", "B") == "tie"

Troubleshooting

Troubleshooting - Day 311\nHandle markdown code fence wrappers when parsing JSON.\n

Security notes

Security - Day 311\nIsolate candidate output strings in prompt boundaries to prevent judge injection attacks.\n