AI Engineering: Agents and Applications › Evaluation and Reliability › Day 314

Hands-on lab — Day 314: Observability and Tracing for AI

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/tracing_engine.py

Test

bash tests/run_tests.sh

File tree

examples/tracing_engine.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/tracing_engine.py
tests/run_tests.sh
tests/test_tracing_engine.py
troubleshooting.md

Lab README

Day 314 Lab: Observability and Tracing for AI

Lesson

Purpose

Build an OpenInference-Compatible AI Tracing and Telemetry Engine in Python supporting hierarchical span trees, duration timing, token accounting, and JSON trace export.

Learning objectives

  • Implement hierarchical span management with parent-child relationships.
  • Measure elapsed execution duration for operations.
  • Calculate token usage and financial cost attribution.
  • Export nested JSON trace trees for observability backends.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/tracing_engine.py: Starter implementation skeleton
  • examples/tracing_engine.py: Verified reference implementation
  • tests/test_tracing_engine.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/tracing_engine.py

What the commands do

  • Executes span creation, token cost calculations, and trace exports.

Expected output

Trace Root: {'span_id': '09571464', 'parent_id': None, 'name': 'Test_Trace', 'kind': 'ROOT', 'duration_ms': 0.0, 'attributes': {}, 'children': []}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Root span creation and termination
  • Nested child span relationships and attributes
  • Duration timing (> 0.0 ms)
  • Accurate token cost calculations
  • JSON serialization of the full span tree

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Remember to invoke .finish() on spans to record durations.

Security notes

Runs completely offline on local CPU.

Extension exercises

Add Time to First Token (TTFT) tracking attributes.

Day number: 314 of 365

Expected output

FIELDS.md

# Measured Values
- `spans_created`: 4
- `traces_completed`: 1

examples-run.txt

# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
Trace Root: {'span_id': '09571464', 'parent_id': None, 'name': 'Test_Trace', 'kind': 'ROOT', 'duration_ms': 0.0, 'attributes': {}, 'children': []}

measured-values.txt

spans_created=4
traces_completed=1

run-output.txt

# Captured from a real run. Identifiers and timings below vary between
# runs -- compare the SHAPE of the output, not the exact values.
Trace Root: {'span_id': '09571464', 'parent_id': None, 'name': 'Test_Trace', 'kind': 'ROOT', 'duration_ms': 0.0, 'attributes': {}, 'children': []}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


self = <examples.tracing_engine.TraceTelemetryEngine object at 0x107260a50>
name = 'Main_Pipeline'

    def start_trace(self, name: str) -> Span:
>       raise NotImplementedError('TASK 4: implement start_trace.')
E       NotImplementedError: TASK 4: implement start_trace.

examples/tracing_engine.py:41: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_tracing_engine.py::test_start_root_trace - NotImplementedEr...
FAILED tests/test_tracing_engine.py::test_create_child_span_and_hierarchy - N...
FAILED tests/test_tracing_engine.py::test_span_timing_and_attributes - NotImp...
FAILED tests/test_tracing_engine.py::test_calculate_cost - NotImplementedErro...
FAILED tests/test_tracing_engine.py::test_trace_to_dict_serialization - NotIm...
============================== 5 failed in 0.04s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_tracing_engine.py::test_start_root_trace PASSED               [ 20%]
tests/test_tracing_engine.py::test_create_child_span_and_hierarchy PASSED [ 40%]
tests/test_tracing_engine.py::test_span_timing_and_attributes PASSED     [ 60%]
tests/test_tracing_engine.py::test_calculate_cost PASSED                 [ 80%]
tests/test_tracing_engine.py::test_trace_to_dict_serialization PASSED    [100%]

============================== 5 passed in 0.03s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 5 items

tests/test_tracing_engine.py::test_start_root_trace PASSED               [ 20%]
tests/test_tracing_engine.py::test_create_child_span_and_hierarchy PASSED [ 40%]
tests/test_tracing_engine.py::test_span_timing_and_attributes PASSED     [ 60%]
tests/test_tracing_engine.py::test_calculate_cost PASSED                 [ 80%]
tests/test_tracing_engine.py::test_trace_to_dict_serialization PASSED    [100%]

============================== 5 passed in 0.03s ===============================

Source files

examples/tracing_engine.py (2048 bytes)
import time
import uuid
import json
from typing import Dict, Any, List, Optional

class Span:
    def __init__(self, name: str, kind: str, parent_id: Optional[str] = None):
        self.span_id = str(uuid.uuid4())[:8]
        self.parent_id = parent_id
        self.name = name
        self.kind = kind
        self.start_time = time.time()
        self.end_time: Optional[float] = None
        self.duration_ms: float = 0.0
        self.attributes: Dict[str, Any] = {}
        self.children: List['Span'] = []
        
    def set_attribute(self, key: str, value: Any):
        self.attributes[key] = value
        
    def finish(self):
        self.end_time = time.time()
        self.duration_ms = round((self.end_time - self.start_time) * 1000, 2)
        
    def to_dict(self) -> Dict[str, Any]:
        return {
            "span_id": self.span_id,
            "parent_id": self.parent_id,
            "name": self.name,
            "kind": self.kind,
            "duration_ms": self.duration_ms,
            "attributes": self.attributes,
            "children": [c.to_dict() for c in self.children]
        }

class TraceTelemetryEngine:
    def __init__(self):
        self.traces: Dict[str, Span] = {}
        
    def start_trace(self, name: str) -> Span:
        root_span = Span(name=name, kind="ROOT")
        self.traces[root_span.span_id] = root_span
        return root_span
        
    def create_child_span(self, parent_span: Span, name: str, kind: str) -> Span:
        child = Span(name=name, kind=kind, parent_id=parent_span.span_id)
        parent_span.children.append(child)
        return child
        
    @staticmethod
    def calculate_cost(prompt_tokens: int, completion_tokens: int, model: str = "claude-3-5-sonnet") -> float:
        cost = (prompt_tokens * 3.0 / 1_000_000) + (completion_tokens * 15.0 / 1_000_000)
        return round(cost, 6)

if __name__ == "__main__":
    engine = TraceTelemetryEngine()
    root = engine.start_trace("Test_Trace")
    root.finish()
    print("Trace Root:", root.to_dict())
metadata.yml (401 bytes)
lesson_id: D314
day: 314
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/tracing_engine.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/tracing_engine.py (1834 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import time
import uuid
import json
from typing import Dict, Any, List, Optional

class Span:

    def __init__(self, name: str, kind: str, parent_id: Optional[str]=None):
        self.span_id = str(uuid.uuid4())[:8]
        self.parent_id = parent_id
        self.name = name
        self.kind = kind
        self.start_time = time.time()
        self.end_time: Optional[float] = None
        self.duration_ms: float = 0.0
        self.attributes: Dict[str, Any] = {}
        self.children: List['Span'] = []

    def set_attribute(self, key: str, value: Any):
        raise NotImplementedError('TASK 1: implement set_attribute.')

    def finish(self):
        raise NotImplementedError('TASK 2: implement finish.')

    def to_dict(self) -> Dict[str, Any]:
        raise NotImplementedError('TASK 3: implement to_dict.')

class TraceTelemetryEngine:

    def __init__(self):
        self.traces: Dict[str, Span] = {}

    def start_trace(self, name: str) -> Span:
        raise NotImplementedError('TASK 4: implement start_trace.')

    def create_child_span(self, parent_span: Span, name: str, kind: str) -> Span:
        raise NotImplementedError('TASK 5: implement create_child_span.')

    @staticmethod
    def calculate_cost(prompt_tokens: int, completion_tokens: int, model: str='claude-3-5-sonnet') -> float:
        raise NotImplementedError('TASK 6: implement calculate_cost.')
if __name__ == '__main__':
    engine = TraceTelemetryEngine()
    root = engine.start_trace('Test_Trace')
    root.finish()
    print('Trace Root:', root.to_dict())
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_tracing_engine.py (1493 bytes)
import pytest
import time
from examples.tracing_engine import TraceTelemetryEngine, Span

def test_start_root_trace():
    engine = TraceTelemetryEngine()
    root = engine.start_trace("Query_Root")
    assert root.kind == "ROOT"
    assert root.parent_id is None
    assert root.span_id in engine.traces

def test_create_child_span_and_hierarchy():
    engine = TraceTelemetryEngine()
    root = engine.start_trace("Root")
    child = engine.create_child_span(root, "LLM_Call", "LLM")
    assert child.parent_id == root.span_id
    assert len(root.children) == 1
    assert root.children[0].name == "LLM_Call"

def test_span_timing_and_attributes():
    span = Span(name="TestSpan", kind="TOOL")
    span.set_attribute("tool.name", "weather_api")
    time.sleep(0.01)
    span.finish()
    assert span.duration_ms >= 5.0
    assert span.attributes["tool.name"] == "weather_api"

def test_calculate_cost():
    cost = TraceTelemetryEngine.calculate_cost(prompt_tokens=1000, completion_tokens=500)
    # (1000 * 3 / 1M) + (500 * 15 / 1M) = 0.003 + 0.0075 = 0.0105
    assert cost == 0.0105

def test_trace_to_dict_serialization():
    engine = TraceTelemetryEngine()
    root = engine.start_trace("Main_Pipeline")
    child = engine.create_child_span(root, "Vector_Search", "RETRIEVER")
    child.finish()
    root.finish()
    
    data = root.to_dict()
    assert data["name"] == "Main_Pipeline"
    assert len(data["children"]) == 1
    assert data["children"][0]["name"] == "Vector_Search"

Troubleshooting

Troubleshooting - Day 314\nCall finish() on all spans to record accurate duration metrics.\n

Security notes

Security - Day 314\nDo not persist unmasked customer PII or raw authentication tokens in trace spans.\n