AI Engineering: Agents and Applications › Evaluation and Reliability › Day 314

Day 314: Observability and Tracing for AI

Day 314 of 365 — Observability and Tracing for AI

Master production observability and distributed tracing for LLM applications: instrument span trees with OpenTelemetry, track token economics and p99 latencies, capture user feedback loops, and debug multi-agent execution traces.

Course
AI Engineering: Agents and Applications
Category
Evaluation and Reliability
Reading time
≈ 35 min
Practical time
≈ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Deploying a generative AI application without observability is like flying a supersonic jet at night with all cockpit instruments disconnected. When an end user complains that “the assistant gave the wrong answer and was really slow”:

AI Observability and Distributed Tracing provides complete, granular visibility into every step of a compound AI system: tracking nested span trees, recording prompt versions, auditing token economics, and capturing user feedback to fuel continuous evaluation datasets.

Distributed AI Span Tree Architecture showing Root Trace, Child Spans for Query Embedding, Vector DB Search, Prompt Construction, LLM Inference, and Guardrail Scan

The idea in plain language

Diagram: unstructured logging compared with distributed tracing for a failed request

Diagram: the contents of a span in an AI system trace

Think of AI observability like the flight data recorder (black box) and live telemetry telemetry stream on a spacecraft:

Historical background

Diagram: the progression from RPC tracing to generative AI observability

Distributed tracing and APM evolved through three generations:

1. Traditional APM and Dapper (2010–2018)

Google’s Dapper paper established distributed RPC tracing (trace_id, span_id). Tools like Datadog, Jaeger, and Zipkin visualized HTTP service-to-service calls.

2. The OpenTelemetry Standard (2019–2022)

The Cloud Native Computing Foundation (CNCF) merged OpenTracing and OpenCensus into OpenTelemetry (OTel), creating a universal, vendor-neutral telemetry standard.

3. OpenInference and Generative AI Observability (2023–Present)

Generative AI required domain-specific semantic conventions: tracking prompt templates, temperature, token counts (input/cached/output), retrieved vector documents, and LLM-as-a-Judge evaluations (Arize Phoenix, Langfuse, LangSmith).

What it is — and what it is not

Let us define the core components of production AI observability:

What AI Observability Is:

What AI Observability Is Not:

+-------------------------------------------------------------------------------+
|                       OPENINFERENCE SPAN ATTRIBUTES                           |
+---------------------------+-----------------------+---------------------------+
| Semantic Attribute        | Data Type             | Example Value             |
+---------------------------+-----------------------+---------------------------+
| `openinference.span.kind` | string                | `LLM`, `RETRIEVER`, `TOOL`|
| `llm.model_name`          | string                | `claude-3-5-sonnet-20241022`|
| `llm.token_count.prompt`  | integer               | `1420`                    |
| `llm.token_count.completion`| integer             | `215`                     |
| `llm.token_count.cached`  | integer               | `1024`                    |
| `llm.cost_usd`            | float                 | `0.0031`                  |
| `retrieval.documents`     | JSON list             | `[{"doc_id": "d1", "score": 0.88}]`|
+---------------------------+-----------------------+---------------------------+

Why it was created and what problems it solves

AI observability solves the four primary operational challenges of production AI systems:

1. Root-Cause Debugging in Compound AI Pipelines

When a RAG assistant fails, distributed traces reveal whether the failure was caused by empty vector search results, truncated prompt context, or model reasoning errors.

2. Controlling Runaway Token Costs

Real-time token telemetry tracks token usage per user and organization, immediately identifying infinite tool loops or prompt inflation regressions.

3. Continuous Improvement via User Feedback

When a user clicks “thumbs down”, the UI sends the rating attached to the trace_id. Engineers can review the exact trace, identify the failure mode, and add it to the evaluation dataset.

How it works

Diagram: the three operational capabilities distributed AI tracing provides

Let us inspect the complete Production Observability to Evaluation Flywheel:

[1. Live Production User Request]
  - User submits query with `trace_id = trace_abc123`
                         │
                         ▼
[2. Distributed Span Tree Execution]
  - Span: Embedding Search (45ms)
  - Span: LLM Generation (Claude 3.5 Sonnet, 1,200 tokens, 680ms)
  - Span: Output Guardrail Validation (4ms)
                         │
                         ▼
[3. User Feedback Collection]
  - User clicks Thumbs Down: "The answer was outdated."
  - Telemetry logs feedback linked to `trace_abc123`
                         │
                         ▼
[4. Automated Failure Triage & Clustering]
  - Semantic clustering tags issue: `category/outdated_context`
  - Annotator adds correct ground truth response
                         │
                         ▼
[5. Append to Golden Eval Dataset]
  - New test case added to `golden_eval.jsonl`
  - CI regression gate guarantees this bug will never recur

Production Feedback to Eval Flywheel showing Live Production Traffic, Trace Logging, User Thumbs Down, Triage & Clustering, and Eval Dataset Append

An everyday analogy

Diagram: parcel tracking as an analogy for distributed tracing

Think of an AI distributed tracer like the parcel tracking barcode on an Amazon delivery:

Examples in practice

Let us build a complete Python Trace & Telemetry Engine implementing nested span creation, duration timing, token accounting, cost calculation, and JSON trace export:

import time
import uuid
import json
from typing import Dict, Any, List, Optional

class Span:
    def __init__(self, name: str, kind: str, parent_id: Optional[str] = None):
        self.span_id = str(uuid.uuid4())[:8]
        self.parent_id = parent_id
        self.name = name
        self.kind = kind
        self.start_time = time.time()
        self.end_time: Optional[float] = None
        self.duration_ms: float = 0.0
        self.attributes: Dict[str, Any] = {}
        self.children: List['Span'] = []
        
    def set_attribute(self, key: str, value: Any):
        self.attributes[key] = value
        
    def finish(self):
        self.end_time = time.time()
        self.duration_ms = round((self.end_time - self.start_time) * 1000, 2)
        
    def to_dict(self) -> Dict[str, Any]:
        return {
            "span_id": self.span_id,
            "parent_id": self.parent_id,
            "name": self.name,
            "kind": self.kind,
            "duration_ms": self.duration_ms,
            "attributes": self.attributes,
            "children": [c.to_dict() for c in self.children]
        }

class TraceTelemetryEngine:
    def __init__(self):
        self.traces: Dict[str, Span] = {}
        
    def start_trace(self, name: str) -> Span:
        root_span = Span(name=name, kind="ROOT")
        self.traces[root_span.span_id] = root_span
        return root_span
        
    def create_child_span(self, parent_span: Span, name: str, kind: str) -> Span:
        child = Span(name=name, kind=kind, parent_id=parent_span.span_id)
        parent_span.children.append(child)
        return child
        
    @staticmethod
    def calculate_cost(prompt_tokens: int, completion_tokens: int, model: str = "claude-3-5-sonnet") -> float:
        # Example pricing: $3.00/M prompt, $15.00/M completion
        cost = (prompt_tokens * 3.0 / 1_000_000) + (completion_tokens * 15.0 / 1_000_000)
        return round(cost, 6)

if __name__ == "__main__":
    engine = TraceTelemetryEngine()
    root = engine.start_trace("RAG_Query")
    llm_span = engine.create_child_span(root, "LLM_Inference", "LLM")
    llm_span.set_attribute("llm.model", "claude-3-5-sonnet")
    llm_span.set_attribute("llm.prompt_tokens", 1200)
    llm_span.set_attribute("llm.completion_tokens", 250)
    llm_span.set_attribute("llm.cost_usd", engine.calculate_cost(1200, 250))
    llm_span.finish()
    root.finish()
    print("Trace Export:", json.dumps(root.to_dict(), indent=2))

Implications: security, privacy, performance, scalability, and cost

1. Privacy Scrubbing in Traces

Traces capture raw input prompts and model completions. Always apply PII scrubbing before writing spans to trace storage so customer credentials and personal data are never exposed in observability dashboards.

2. Telemetry Sampling at Scale

At millions of requests per day, storing 100% of LLM traces can generate massive database bills. Implement Tail-Based Sampling: store 100% of failed traces, high-latency traces (> 2000ms), and user-rated sessions, while sampling only 5% of routine happy-path requests.

Diagram: tail-based trace sampling by outcome

Alternatives: free, open source, and commercial

Tool / FrameworkTypeBest Suited ForKey Capabilities
LangfuseOpen SourceFull-stack AI observabilitySelf-hostable, web UI, eval integrations
Arize PhoenixOpen SourceLocal tracing & evalsZero-dependency local notebook tracing
LangSmithCommercialLangChain / LangGraph tracingInteractive prompt playground
OpenLLMetryOpen SourceOpenTelemetry auto-instrumentationDrop-in OTel SDK for all LLM providers
+-----------------------+-----------------------+-----------------------+
| Dimension             | Standard APM Logs     | AI Observability      |
+-----------------------+-----------------------+-----------------------+
| Trace Payloads        | HTTP Headers & Status | Full Prompts & Context|
| Cost Accounting       | CPU / Memory metrics  | Token-level economics |
| Quality Feedback      | HTTP 500 error rates  | User thumbs & LLM eval|
| Semantic Analysis     | None (Flat text logs) | RAG chunk attribution |
+-----------------------+-----------------------+-----------------------+

When to use it — and when not to

Diagram: when to implement AI observability and what to avoid

Implement AI Tracing & Observability for:

Do NOT require distributed tracing for:

Advanced Topic: OpenInference Semantic Conventions and Distributed Trace Spans

To achieve vendor-neutral observability across different LLM providers and frameworks, production systems adopt OpenInference Semantic Conventions:

# OpenInference Semantic Span Attributes
span.set_attribute("openinference.span.kind", "LLM")
span.set_attribute("llm.model_name", "claude-3-5-sonnet-20241022")
span.set_attribute("llm.invocation_parameters", json.dumps({"temperature": 0.0, "max_tokens": 1024}))
span.set_attribute("llm.token_count.prompt", 1420)
span.set_attribute("llm.token_count.completion", 215)
span.set_attribute("llm.token_count.cached", 1024)
span.set_attribute("llm.cost_usd", 0.0031)

Architectural Blueprint: High-Throughput Real-Time Telemetry Pipeline

In enterprise AI applications serving millions of queries per day, tracing telemetry is decoupled from the main request path:

  1. Non-Blocking Asynchronous Spans: Spans are buffered in an in-memory ring buffer and dispatched in background batches via OpenTelemetry OTLP exporters.
  2. Kafka / Redpanda Ingestion Queue: Telemetry spans are ingested into high-throughput streaming topics.
  3. ClickHouse Analytical Store: Traces and token metrics are written to columnar storage, enabling sub-second analytical queries across billions of spans.
  4. Grafana / Datadog Dashboards: Real-time dashboards monitor p95 latency, Time to First Token (TTFT), token consumption rate, and cost attribution per tenant.

Diagram: the stages of a high-throughput telemetry pipeline

Production Checklist: AI Observability and Telemetry Readiness

Ensure your production monitoring pipeline implements the following Observability Best Practices:

  1. Distributed Trace Propagation: Does every user request carry a unique trace_id propagated through all downstream spans?
  2. Token Economics Tracking: Are prompt tokens, completion tokens, cached tokens, and financial costs recorded per request?
  3. Latency Milestone Timing: Are Time to First Token (TTFT), inter-token latency (ITL), and total duration tracked?
  4. Tail-Based Sampling: Are 100% of errors, high-latency traces, and user feedback sessions persisted in analytical storage?
  5. User Feedback Integration: Can end-user ratings (thumbs up/down) be tied directly to execution trace spans for triage?

Deep Dive: Real-Time Token Economics, Caching Optimization, and Cost Attribution

Managing cloud GPU expenditures requires granular visibility into token consumption:

Diagram: the token economics signals worth monitoring

Advanced Topic: Correlating User Feedback with Automated Evaluation Metrics

To validate that your offline evaluation metrics predict real-world customer satisfaction:

Diagram: correlating offline evaluation scores with live user satisfaction

Practical Implementation Tips for Distributed AI Tracing

When instrumenting complex AI microservices with OpenTelemetry:

Real-World Case Study: Debugging a High-Latency RAG Outage

An enterprise search engine experienced intermittent 5-second latency spikes in production:

Diagram: the span breakdown of a latency investigation

Observability Anti-Patterns to Avoid in Production AI Systems

When designing telemetry systems for enterprise LLM workloads, avoid these common engineering pitfalls:

The AI connection

Knowledge check

  1. What is a span tree in AI tracing? A hierarchical representation of nested operations (root request -> vector search -> prompt building -> LLM generation -> guardrail validation).
  2. Why is token telemetry crucial for cost attribution? Because it tracks exact prompt, completion, and cached token consumption, allowing accurate billing per tenant.
  3. How does user feedback close the continuous evaluation loop? By capturing low-rated production interactions and automatically converting them into labeled test cases in the golden evaluation dataset.

Hands-on exercise

In this lab, you will build an OpenInference-Compatible AI Tracing Engine in Python. Your implementation will:

  1. Create and manage hierarchical span trees with unique IDs and parent relationships.
  2. Measure microsecond execution durations for child spans.
  3. Record model attributes, token counts, and dollar costs.
  4. Export valid JSON trace trees for observability backends.

Expected output

============================= test session starts ==============================
collected 5 items

tests/test_tracing_engine.py .....                                       [100%]

============================== 5 passed in 0.02s ===============================
[TRACING ENGINE] Root trace initialized with 3 nested child spans.
[TELEMETRY] Token cost accurately computed: $0.00735 for 1,450 tokens.
[STATUS] Distributed AI tracing engine verified with 100% test pass rate.

Validate your work

Run the pytest test suite:

pytest tests/run_tests.sh

Troubleshooting

  1. Duration zero: Ensure .finish() is invoked on spans to compute the elapsed duration.
  2. Hierarchy broken: Verify child spans append correctly to the parent’s children list.

Common mistakes

Practice assignment

Extend the tracing engine to support Streaming Telemetry Metrics measuring Time to First Token (TTFT) and Inter-Token Latency (ITL).

Extension challenge

Implement a Tail-Based Trace Sampler that saves 100% of traces with duration $> 1000\text{ms}$ or error status while randomly sampling 10% of successful traces.

Quiz

Q1. Why is traditional microservice APM (like standard HTTP request tracing) insufficient for diagnosing complex LLM and RAG applications?

  1. APM sees HTTP codes, not prompts, chunks, or tokens
  2. Because APM tools do not support Python
  3. Because LLMs do not use HTTP requests
  4. Because tracing makes language models stop working
Show answer

Answer: A. APM sees HTTP codes, not prompts, chunks, or tokens

AI observability requires capturing nested spans containing prompts, completions, token counts, and retrieved context chunks.

Q2. In OpenTelemetry and OpenInference standards, what is a 'Span Tree' in an AI workflow?

  1. A binary search tree for sorting strings
  2. A nested graph of every stage, with latency and payloads
  3. A network cabling topology
  4. A database index
Show answer

Answer: B. A nested graph of every stage, with latency and payloads

Span trees represent the hierarchical decomposition of a complex compound AI workflow.

Q3. What is the primary benefit of tracking 'Token Telemetry' (Prompt Tokens, Completion Tokens, Cached Tokens) in production?

  1. It makes the computer run faster
  2. It replaces database backups
  3. It attributes cost and catches token inflation
  4. It encrypts user passwords
Show answer

Answer: C. It attributes cost and catches token inflation

Token telemetry allows financial accounting, rate limit governance, and caching optimization.

Q4. How does closing the 'Production User Feedback Loop' accelerate model improvement?

  1. By deleting user accounts who give negative feedback
  2. By restarting the server
  3. By sending marketing emails to users
  4. Turn low-rated sessions into regression cases
Show answer

Answer: D. Turn low-rated sessions into regression cases

Negative user feedback is the highest-signal source for discovering real-world edge cases.

Q5. Which open-source telemetry framework is widely adopted for vendor-neutral tracing in AI engineering?

  1. OpenTelemetry
  2. Flash Player
  3. DirectX 12
  4. Apache Subversion
Show answer

Answer: A. OpenTelemetry

OpenTelemetry and OpenInference provide standardized, vendor-neutral telemetry instrumentation.

Glossary

AI Observability
Monitoring, tracing, evaluating, and debugging the runtime behavior and economics of production AI systems.
Distributed Trace
An end-to-end record of an entire user request through all downstream services, LLMs, and databases.
Span
A single contiguous unit of work within a trace (e.g. vector search, LLM completion, guardrail check).
OpenInference
An open standard extending OpenTelemetry semantic conventions specifically for AI systems.
Token Telemetry
Tracking input tokens, cached tokens, output tokens, and financial costs per request.
Feedback Loop
Capturing user ratings (thumbs up/down, corrections) and feeding failed interactions into evaluation datasets.
Time to First Token (TTFT)
The latency in milliseconds from sending a prompt until receiving the first streamed token.
Inter-Token Latency (ITL)
The average time in milliseconds between consecutive streamed tokens.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.