AI Engineering: Agents and Applications › Evaluation and Reliability › Day 314
Hands-on lab — Day 314: Observability and Tracing for AI
- ← Back to the Day 314 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/tracing_engine.py Test
bash tests/run_tests.sh File tree
examples/tracing_engine.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/tracing_engine.py tests/run_tests.sh tests/test_tracing_engine.py troubleshooting.md
Lab README
Day 314 Lab: Observability and Tracing for AI
Lesson
- Lesson title: Observability and Tracing for AI
- Day number: 314 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-314-observability-and-tracing-for-ai
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-314-observability-and-tracing-for-aiwhen the site is running.
Purpose
Build an OpenInference-Compatible AI Tracing and Telemetry Engine in Python supporting hierarchical span trees, duration timing, token accounting, and JSON trace export.
Learning objectives
- Implement hierarchical span management with parent-child relationships.
- Measure elapsed execution duration for operations.
- Calculate token usage and financial cost attribution.
- Export nested JSON trace trees for observability backends.
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library, Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/tracing_engine.py: Starter implementation skeletonexamples/tracing_engine.py: Verified reference implementationtests/test_tracing_engine.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/tracing_engine.py
What the commands do
- Executes span creation, token cost calculations, and trace exports.
Expected output
Trace Root: {'span_id': '09571464', 'parent_id': None, 'name': 'Test_Trace', 'kind': 'ROOT', 'duration_ms': 0.0, 'attributes': {}, 'children': []}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Root span creation and termination
- Nested child span relationships and attributes
- Duration timing (> 0.0 ms)
- Accurate token cost calculations
- JSON serialization of the full span tree
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Remember to invoke .finish() on spans to record durations.
Security notes
Runs completely offline on local CPU.
Extension exercises
Add Time to First Token (TTFT) tracking attributes.
Navigation
Day number: 314 of 365
Expected output
FIELDS.md
# Measured Values
- `spans_created`: 4
- `traces_completed`: 1
examples-run.txt
# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
Trace Root: {'span_id': '09571464', 'parent_id': None, 'name': 'Test_Trace', 'kind': 'ROOT', 'duration_ms': 0.0, 'attributes': {}, 'children': []}
measured-values.txt
spans_created=4
traces_completed=1
run-output.txt
# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
Trace Root: {'span_id': '09571464', 'parent_id': None, 'name': 'Test_Trace', 'kind': 'ROOT', 'duration_ms': 0.0, 'attributes': {}, 'children': []}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
self = <examples.tracing_engine.TraceTelemetryEngine object at 0x107260a50>
name = 'Main_Pipeline'
def start_trace(self, name: str) -> Span:
> raise NotImplementedError('TASK 4: implement start_trace.')
E NotImplementedError: TASK 4: implement start_trace.
examples/tracing_engine.py:41: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_tracing_engine.py::test_start_root_trace - NotImplementedEr...
FAILED tests/test_tracing_engine.py::test_create_child_span_and_hierarchy - N...
FAILED tests/test_tracing_engine.py::test_span_timing_and_attributes - NotImp...
FAILED tests/test_tracing_engine.py::test_calculate_cost - NotImplementedErro...
FAILED tests/test_tracing_engine.py::test_trace_to_dict_serialization - NotIm...
============================== 5 failed in 0.04s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_tracing_engine.py::test_start_root_trace PASSED [ 20%]
tests/test_tracing_engine.py::test_create_child_span_and_hierarchy PASSED [ 40%]
tests/test_tracing_engine.py::test_span_timing_and_attributes PASSED [ 60%]
tests/test_tracing_engine.py::test_calculate_cost PASSED [ 80%]
tests/test_tracing_engine.py::test_trace_to_dict_serialization PASSED [100%]
============================== 5 passed in 0.03s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items
tests/test_tracing_engine.py::test_start_root_trace PASSED [ 20%]
tests/test_tracing_engine.py::test_create_child_span_and_hierarchy PASSED [ 40%]
tests/test_tracing_engine.py::test_span_timing_and_attributes PASSED [ 60%]
tests/test_tracing_engine.py::test_calculate_cost PASSED [ 80%]
tests/test_tracing_engine.py::test_trace_to_dict_serialization PASSED [100%]
============================== 5 passed in 0.03s ===============================
Source files
examples/tracing_engine.py (2048 bytes)
import time
import uuid
import json
from typing import Dict, Any, List, Optional
class Span:
def __init__(self, name: str, kind: str, parent_id: Optional[str] = None):
self.span_id = str(uuid.uuid4())[:8]
self.parent_id = parent_id
self.name = name
self.kind = kind
self.start_time = time.time()
self.end_time: Optional[float] = None
self.duration_ms: float = 0.0
self.attributes: Dict[str, Any] = {}
self.children: List['Span'] = []
def set_attribute(self, key: str, value: Any):
self.attributes[key] = value
def finish(self):
self.end_time = time.time()
self.duration_ms = round((self.end_time - self.start_time) * 1000, 2)
def to_dict(self) -> Dict[str, Any]:
return {
"span_id": self.span_id,
"parent_id": self.parent_id,
"name": self.name,
"kind": self.kind,
"duration_ms": self.duration_ms,
"attributes": self.attributes,
"children": [c.to_dict() for c in self.children]
}
class TraceTelemetryEngine:
def __init__(self):
self.traces: Dict[str, Span] = {}
def start_trace(self, name: str) -> Span:
root_span = Span(name=name, kind="ROOT")
self.traces[root_span.span_id] = root_span
return root_span
def create_child_span(self, parent_span: Span, name: str, kind: str) -> Span:
child = Span(name=name, kind=kind, parent_id=parent_span.span_id)
parent_span.children.append(child)
return child
@staticmethod
def calculate_cost(prompt_tokens: int, completion_tokens: int, model: str = "claude-3-5-sonnet") -> float:
cost = (prompt_tokens * 3.0 / 1_000_000) + (completion_tokens * 15.0 / 1_000_000)
return round(cost, 6)
if __name__ == "__main__":
engine = TraceTelemetryEngine()
root = engine.start_trace("Test_Trace")
root.finish()
print("Trace Root:", root.to_dict())
metadata.yml (401 bytes)
lesson_id: D314
day: 314
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/tracing_engine.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/tracing_engine.py (1834 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import time
import uuid
import json
from typing import Dict, Any, List, Optional
class Span:
def __init__(self, name: str, kind: str, parent_id: Optional[str]=None):
self.span_id = str(uuid.uuid4())[:8]
self.parent_id = parent_id
self.name = name
self.kind = kind
self.start_time = time.time()
self.end_time: Optional[float] = None
self.duration_ms: float = 0.0
self.attributes: Dict[str, Any] = {}
self.children: List['Span'] = []
def set_attribute(self, key: str, value: Any):
raise NotImplementedError('TASK 1: implement set_attribute.')
def finish(self):
raise NotImplementedError('TASK 2: implement finish.')
def to_dict(self) -> Dict[str, Any]:
raise NotImplementedError('TASK 3: implement to_dict.')
class TraceTelemetryEngine:
def __init__(self):
self.traces: Dict[str, Span] = {}
def start_trace(self, name: str) -> Span:
raise NotImplementedError('TASK 4: implement start_trace.')
def create_child_span(self, parent_span: Span, name: str, kind: str) -> Span:
raise NotImplementedError('TASK 5: implement create_child_span.')
@staticmethod
def calculate_cost(prompt_tokens: int, completion_tokens: int, model: str='claude-3-5-sonnet') -> float:
raise NotImplementedError('TASK 6: implement calculate_cost.')
if __name__ == '__main__':
engine = TraceTelemetryEngine()
root = engine.start_trace('Test_Trace')
root.finish()
print('Trace Root:', root.to_dict())
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_tracing_engine.py (1493 bytes)
import pytest
import time
from examples.tracing_engine import TraceTelemetryEngine, Span
def test_start_root_trace():
engine = TraceTelemetryEngine()
root = engine.start_trace("Query_Root")
assert root.kind == "ROOT"
assert root.parent_id is None
assert root.span_id in engine.traces
def test_create_child_span_and_hierarchy():
engine = TraceTelemetryEngine()
root = engine.start_trace("Root")
child = engine.create_child_span(root, "LLM_Call", "LLM")
assert child.parent_id == root.span_id
assert len(root.children) == 1
assert root.children[0].name == "LLM_Call"
def test_span_timing_and_attributes():
span = Span(name="TestSpan", kind="TOOL")
span.set_attribute("tool.name", "weather_api")
time.sleep(0.01)
span.finish()
assert span.duration_ms >= 5.0
assert span.attributes["tool.name"] == "weather_api"
def test_calculate_cost():
cost = TraceTelemetryEngine.calculate_cost(prompt_tokens=1000, completion_tokens=500)
# (1000 * 3 / 1M) + (500 * 15 / 1M) = 0.003 + 0.0075 = 0.0105
assert cost == 0.0105
def test_trace_to_dict_serialization():
engine = TraceTelemetryEngine()
root = engine.start_trace("Main_Pipeline")
child = engine.create_child_span(root, "Vector_Search", "RETRIEVER")
child.finish()
root.finish()
data = root.to_dict()
assert data["name"] == "Main_Pipeline"
assert len(data["children"]) == 1
assert data["children"][0]["name"] == "Vector_Search"