AI Engineering: Agents and Applications › Evaluation and Reliability › Day 309

Hands-on lab — Day 309: Why Evals Are the Real Moat

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/eval_metrics.py

Test

bash tests/run_tests.sh

File tree

examples/eval_metrics.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/eval_metrics.py
tests/run_tests.sh
tests/test_eval_metrics.py
troubleshooting.md

Lab README

Day 309 Lab: Why Evals Are the Real Moat

Lesson

Purpose

Build an Automated AI Evaluation Metric Engine in Python that calculates normalized Exact Match, field-level JSON schema F1 scores, and token-level overlap F1 metrics.

Learning objectives

  • Calculate normalized exact match binary scores.
  • Parse JSON predictions and compute precision, recall, and F1 across dictionary keys.
  • Compute token-level overlap F1 metrics for free-form text.
  • Guard against malformed JSON and division-by-zero edge cases.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/eval_metrics.py: Starter implementation skeleton
  • examples/eval_metrics.py: Verified reference implementation
  • tests/test_eval_metrics.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/eval_metrics.py

What the commands do

  • Evaluates exact match, JSON field F1, and token overlap metrics across test cases.

Expected output

Exact Match: 1.0

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Exact match normalization (case, whitespace)
  • JSON field-level precision, recall, and F1 calculation
  • Malformed JSON error handling (returns 0.0)
  • Token overlap F1 calculation

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Verify inputs are valid strings and dictionaries.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement weighted Levenshtein distance metrics.

Day number: 309 of 365

Expected output

FIELDS.md

# Measured Values
- `metrics_evaluated`: 3
- `evals_passed`: 1

examples-run.txt

Exact Match: 1.0

measured-values.txt

metrics_evaluated=3
evals_passed=1

run-output.txt

Exact Match: 1.0

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

prediction = 'The capital of France is Paris.'
ground_truth = 'Paris is the capital of France.'

    @staticmethod
    def token_overlap_f1(prediction: str, ground_truth: str) -> float:
>       raise NotImplementedError('TASK 3: implement token_overlap_f1.')
E       NotImplementedError: TASK 3: implement token_overlap_f1.

examples/eval_metrics.py:24: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_eval_metrics.py::test_exact_match - NotImplementedError: TA...
FAILED tests/test_eval_metrics.py::test_json_field_f1_perfect_match - NotImpl...
FAILED tests/test_eval_metrics.py::test_json_field_f1_partial_match - NotImpl...
FAILED tests/test_eval_metrics.py::test_json_field_f1_invalid_json - NotImple...
FAILED tests/test_eval_metrics.py::test_token_overlap_f1 - NotImplementedErro...
============================== 5 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-309-why-evals-are-the-real-moat
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_eval_metrics.py::test_exact_match PASSED                      [ 20%]
tests/test_eval_metrics.py::test_json_field_f1_perfect_match PASSED      [ 40%]
tests/test_eval_metrics.py::test_json_field_f1_partial_match PASSED      [ 60%]
tests/test_eval_metrics.py::test_json_field_f1_invalid_json PASSED       [ 80%]
tests/test_eval_metrics.py::test_token_overlap_f1 PASSED                 [100%]

============================== 5 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-309-why-evals-are-the-real-moat
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_eval_metrics.py::test_exact_match PASSED                      [ 20%]
tests/test_eval_metrics.py::test_json_field_f1_perfect_match PASSED      [ 40%]
tests/test_eval_metrics.py::test_json_field_f1_partial_match PASSED      [ 60%]
tests/test_eval_metrics.py::test_json_field_f1_invalid_json PASSED       [ 80%]
tests/test_eval_metrics.py::test_token_overlap_f1 PASSED                 [100%]

============================== 5 passed in 0.01s ===============================

Source files

examples/eval_metrics.py (2276 bytes)
import json
import re
from typing import Dict, Any

class EvalMetricEngine:
    @staticmethod
    def exact_match(prediction: str, ground_truth: str) -> float:
        norm_pred = prediction.strip().lower()
        norm_gt = ground_truth.strip().lower()
        return 1.0 if norm_pred == norm_gt else 0.0
        
    @staticmethod
    def json_field_f1(predicted_json_str: str, ground_truth_dict: Dict[str, Any]) -> Dict[str, float]:
        try:
            pred_dict = json.loads(predicted_json_str)
        except Exception:
            return {"precision": 0.0, "recall": 0.0, "f1": 0.0, "valid_json": 0.0}
            
        if not isinstance(pred_dict, dict):
            return {"precision": 0.0, "recall": 0.0, "f1": 0.0, "valid_json": 0.0}
            
        correct_fields = 0
        total_pred_fields = len(pred_dict)
        total_gt_fields = len(ground_truth_dict)
        
        for k, v in pred_dict.items():
            if k in ground_truth_dict and str(ground_truth_dict[k]).strip().lower() == str(v).strip().lower():
                correct_fields += 1
                
        precision = correct_fields / total_pred_fields if total_pred_fields > 0 else 0.0
        recall = correct_fields / total_gt_fields if total_gt_fields > 0 else 0.0
        f1 = (2 * precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
        
        return {
            "valid_json": 1.0,
            "precision": round(precision, 4),
            "recall": round(recall, 4),
            "f1": round(f1, 4)
        }
        
    @staticmethod
    def token_overlap_f1(prediction: str, ground_truth: str) -> float:
        pred_tokens = set(re.findall(r'\w+', prediction.lower()))
        gt_tokens = set(re.findall(r'\w+', ground_truth.lower()))
        
        if not pred_tokens or not gt_tokens:
            return 0.0
            
        common = pred_tokens.intersection(gt_tokens)
        if not common:
            return 0.0
            
        precision = len(common) / len(pred_tokens)
        recall = len(common) / len(gt_tokens)
        return round((2 * precision * recall) / (precision + recall), 4)

if __name__ == "__main__":
    engine = EvalMetricEngine()
    print("Exact Match:", engine.exact_match("Paris", "paris"))
metadata.yml (399 bytes)
lesson_id: D309
day: 309
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/eval_metrics.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/eval_metrics.py (984 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import json
import re
from typing import Dict, Any

class EvalMetricEngine:

    @staticmethod
    def exact_match(prediction: str, ground_truth: str) -> float:
        raise NotImplementedError('TASK 1: implement exact_match.')

    @staticmethod
    def json_field_f1(predicted_json_str: str, ground_truth_dict: Dict[str, Any]) -> Dict[str, float]:
        raise NotImplementedError('TASK 2: implement json_field_f1.')

    @staticmethod
    def token_overlap_f1(prediction: str, ground_truth: str) -> float:
        raise NotImplementedError('TASK 3: implement token_overlap_f1.')
if __name__ == '__main__':
    engine = EvalMetricEngine()
    print('Exact Match:', engine.exact_match('Paris', 'paris'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_eval_metrics.py (1406 bytes)
import pytest
from examples.eval_metrics import EvalMetricEngine

def test_exact_match():
    assert EvalMetricEngine.exact_match("San Francisco", "san francisco  ") == 1.0
    assert EvalMetricEngine.exact_match("New York", "Boston") == 0.0

def test_json_field_f1_perfect_match():
    pred = '{"name": "Alice", "role": "Admin"}'
    gt = {"name": "Alice", "role": "Admin"}
    res = EvalMetricEngine.json_field_f1(pred, gt)
    assert res["valid_json"] == 1.0
    assert res["precision"] == 1.0
    assert res["recall"] == 1.0
    assert res["f1"] == 1.0

def test_json_field_f1_partial_match():
    pred = '{"name": "Alice", "role": "User"}'
    gt = {"name": "Alice", "role": "Admin"}
    res = EvalMetricEngine.json_field_f1(pred, gt)
    assert res["valid_json"] == 1.0
    assert res["precision"] == 0.5
    assert res["recall"] == 0.5
    assert res["f1"] == 0.5

def test_json_field_f1_invalid_json():
    pred = '{"name": "Alice", BROKEN JSON'
    gt = {"name": "Alice"}
    res = EvalMetricEngine.json_field_f1(pred, gt)
    assert res["valid_json"] == 0.0
    assert res["f1"] == 0.0

def test_token_overlap_f1():
    pred = "The capital of France is Paris."
    gt = "Paris is the capital of France."
    f1 = EvalMetricEngine.token_overlap_f1(pred, gt)
    assert f1 == 1.0
    
    partial = EvalMetricEngine.token_overlap_f1("Paris is nice", "Tokyo is nice")
    assert 0.0 < partial < 1.0

Troubleshooting

Troubleshooting - Day 309

Handle JSON parsing exceptions with standard try/except blocks.

Security notes

Security - Day 309

Evaluation metrics execute strictly local string computations.