AI Engineering: Agents and Applications › Evaluation and Reliability › Day 309
Hands-on lab — Day 309: Why Evals Are the Real Moat
- ← Back to the Day 309 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-309-why-evals-are-the-real-moat/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/eval_metrics.py Test
bash tests/run_tests.sh File tree
examples/eval_metrics.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/eval_metrics.py tests/run_tests.sh tests/test_eval_metrics.py troubleshooting.md
Lab README
Day 309 Lab: Why Evals Are the Real Moat
Lesson
- Lesson title: Why Evals Are the Real Moat
- Day number: 309 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-309-why-evals-are-the-real-moat
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-309-why-evals-are-the-real-moatwhen the site is running.
Purpose
Build an Automated AI Evaluation Metric Engine in Python that calculates normalized Exact Match, field-level JSON schema F1 scores, and token-level overlap F1 metrics.
Learning objectives
- Calculate normalized exact match binary scores.
- Parse JSON predictions and compute precision, recall, and F1 across dictionary keys.
- Compute token-level overlap F1 metrics for free-form text.
- Guard against malformed JSON and division-by-zero edge cases.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/eval_metrics.py: Starter implementation skeletonexamples/eval_metrics.py: Verified reference implementationtests/test_eval_metrics.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/eval_metrics.py
What the commands do
- Evaluates exact match, JSON field F1, and token overlap metrics across test cases.
Expected output
Exact Match: 1.0
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Exact match normalization (case, whitespace)
- JSON field-level precision, recall, and F1 calculation
- Malformed JSON error handling (returns 0.0)
- Token overlap F1 calculation
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Verify inputs are valid strings and dictionaries.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement weighted Levenshtein distance metrics.
Navigation
Day number: 309 of 365
Expected output
FIELDS.md
# Measured Values
- `metrics_evaluated`: 3
- `evals_passed`: 1
examples-run.txt
Exact Match: 1.0
measured-values.txt
metrics_evaluated=3
evals_passed=1
run-output.txt
Exact Match: 1.0
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
prediction = 'The capital of France is Paris.'
ground_truth = 'Paris is the capital of France.'
@staticmethod
def token_overlap_f1(prediction: str, ground_truth: str) -> float:
> raise NotImplementedError('TASK 3: implement token_overlap_f1.')
E NotImplementedError: TASK 3: implement token_overlap_f1.
examples/eval_metrics.py:24: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_eval_metrics.py::test_exact_match - NotImplementedError: TA...
FAILED tests/test_eval_metrics.py::test_json_field_f1_perfect_match - NotImpl...
FAILED tests/test_eval_metrics.py::test_json_field_f1_partial_match - NotImpl...
FAILED tests/test_eval_metrics.py::test_json_field_f1_invalid_json - NotImple...
FAILED tests/test_eval_metrics.py::test_token_overlap_f1 - NotImplementedErro...
============================== 5 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-309-why-evals-are-the-real-moat
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_eval_metrics.py::test_exact_match PASSED [ 20%]
tests/test_eval_metrics.py::test_json_field_f1_perfect_match PASSED [ 40%]
tests/test_eval_metrics.py::test_json_field_f1_partial_match PASSED [ 60%]
tests/test_eval_metrics.py::test_json_field_f1_invalid_json PASSED [ 80%]
tests/test_eval_metrics.py::test_token_overlap_f1 PASSED [100%]
============================== 5 passed in 0.01s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-309-why-evals-are-the-real-moat
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_eval_metrics.py::test_exact_match PASSED [ 20%]
tests/test_eval_metrics.py::test_json_field_f1_perfect_match PASSED [ 40%]
tests/test_eval_metrics.py::test_json_field_f1_partial_match PASSED [ 60%]
tests/test_eval_metrics.py::test_json_field_f1_invalid_json PASSED [ 80%]
tests/test_eval_metrics.py::test_token_overlap_f1 PASSED [100%]
============================== 5 passed in 0.01s ===============================
Source files
examples/eval_metrics.py (2276 bytes)
import json
import re
from typing import Dict, Any
class EvalMetricEngine:
@staticmethod
def exact_match(prediction: str, ground_truth: str) -> float:
norm_pred = prediction.strip().lower()
norm_gt = ground_truth.strip().lower()
return 1.0 if norm_pred == norm_gt else 0.0
@staticmethod
def json_field_f1(predicted_json_str: str, ground_truth_dict: Dict[str, Any]) -> Dict[str, float]:
try:
pred_dict = json.loads(predicted_json_str)
except Exception:
return {"precision": 0.0, "recall": 0.0, "f1": 0.0, "valid_json": 0.0}
if not isinstance(pred_dict, dict):
return {"precision": 0.0, "recall": 0.0, "f1": 0.0, "valid_json": 0.0}
correct_fields = 0
total_pred_fields = len(pred_dict)
total_gt_fields = len(ground_truth_dict)
for k, v in pred_dict.items():
if k in ground_truth_dict and str(ground_truth_dict[k]).strip().lower() == str(v).strip().lower():
correct_fields += 1
precision = correct_fields / total_pred_fields if total_pred_fields > 0 else 0.0
recall = correct_fields / total_gt_fields if total_gt_fields > 0 else 0.0
f1 = (2 * precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
return {
"valid_json": 1.0,
"precision": round(precision, 4),
"recall": round(recall, 4),
"f1": round(f1, 4)
}
@staticmethod
def token_overlap_f1(prediction: str, ground_truth: str) -> float:
pred_tokens = set(re.findall(r'\w+', prediction.lower()))
gt_tokens = set(re.findall(r'\w+', ground_truth.lower()))
if not pred_tokens or not gt_tokens:
return 0.0
common = pred_tokens.intersection(gt_tokens)
if not common:
return 0.0
precision = len(common) / len(pred_tokens)
recall = len(common) / len(gt_tokens)
return round((2 * precision * recall) / (precision + recall), 4)
if __name__ == "__main__":
engine = EvalMetricEngine()
print("Exact Match:", engine.exact_match("Paris", "paris"))
metadata.yml (399 bytes)
lesson_id: D309
day: 309
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/eval_metrics.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/eval_metrics.py (984 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import json
import re
from typing import Dict, Any
class EvalMetricEngine:
@staticmethod
def exact_match(prediction: str, ground_truth: str) -> float:
raise NotImplementedError('TASK 1: implement exact_match.')
@staticmethod
def json_field_f1(predicted_json_str: str, ground_truth_dict: Dict[str, Any]) -> Dict[str, float]:
raise NotImplementedError('TASK 2: implement json_field_f1.')
@staticmethod
def token_overlap_f1(prediction: str, ground_truth: str) -> float:
raise NotImplementedError('TASK 3: implement token_overlap_f1.')
if __name__ == '__main__':
engine = EvalMetricEngine()
print('Exact Match:', engine.exact_match('Paris', 'paris'))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_eval_metrics.py (1406 bytes)
import pytest
from examples.eval_metrics import EvalMetricEngine
def test_exact_match():
assert EvalMetricEngine.exact_match("San Francisco", "san francisco ") == 1.0
assert EvalMetricEngine.exact_match("New York", "Boston") == 0.0
def test_json_field_f1_perfect_match():
pred = '{"name": "Alice", "role": "Admin"}'
gt = {"name": "Alice", "role": "Admin"}
res = EvalMetricEngine.json_field_f1(pred, gt)
assert res["valid_json"] == 1.0
assert res["precision"] == 1.0
assert res["recall"] == 1.0
assert res["f1"] == 1.0
def test_json_field_f1_partial_match():
pred = '{"name": "Alice", "role": "User"}'
gt = {"name": "Alice", "role": "Admin"}
res = EvalMetricEngine.json_field_f1(pred, gt)
assert res["valid_json"] == 1.0
assert res["precision"] == 0.5
assert res["recall"] == 0.5
assert res["f1"] == 0.5
def test_json_field_f1_invalid_json():
pred = '{"name": "Alice", BROKEN JSON'
gt = {"name": "Alice"}
res = EvalMetricEngine.json_field_f1(pred, gt)
assert res["valid_json"] == 0.0
assert res["f1"] == 0.0
def test_token_overlap_f1():
pred = "The capital of France is Paris."
gt = "Paris is the capital of France."
f1 = EvalMetricEngine.token_overlap_f1(pred, gt)
assert f1 == 1.0
partial = EvalMetricEngine.token_overlap_f1("Paris is nice", "Tokyo is nice")
assert 0.0 < partial < 1.0
Troubleshooting
Troubleshooting - Day 309
Handle JSON parsing exceptions with standard try/except blocks.
Security notes
Security - Day 309
Evaluation metrics execute strictly local string computations.