AI Engineering: Agents and Applications › Evaluation and Reliability › Day 314
Day 314: Observability and Tracing for AI
Master production observability and distributed tracing for LLM applications: instrument span trees with OpenTelemetry, track token economics and p99 latencies, capture user feedback loops, and debug multi-agent execution traces.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
cd labs/sections/ai-engineering/day-314-observability-and-tracing-for-ai - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Architect distributed tracing pipelines capturing nested span trees for retrieval, prompt formatting, LLM generation, tool calls, and post-processing.
- Instrument OpenTelemetry and OpenInference semantic conventions for generative AI applications.
- Track real-time token economics (input tokens, cached tokens, output tokens, financial costs per user/tenant).
- Implement a production Trace & Telemetry Engine in Python supporting contextual span creation, duration timing, token accounting, and JSON trace export.
- Integrate production user feedback loops (thumbs-up/down, ratings, escalations) into continuous evaluation datasets.
Prerequisites
Why this matters
Deploying a generative AI application without observability is like flying a supersonic jet at night with all cockpit instruments disconnected. When an end user complains that “the assistant gave the wrong answer and was really slow”:
- Was the failure caused by poor retrieval ranking in the vector database?
- Did the system prompt fail to instruct the model properly?
- Did the LLM provider experience an infrastructure latency spike (
> 4000ms)? - Did the output guardrail redact legitimate content by mistake?
AI Observability and Distributed Tracing provides complete, granular visibility into every step of a compound AI system: tracking nested span trees, recording prompt versions, auditing token economics, and capturing user feedback to fuel continuous evaluation datasets.
The idea in plain language
Think of AI observability like the flight data recorder (black box) and live telemetry telemetry stream on a spacecraft:
- No Observability (Console Logs Only): The spacecraft experiences a thruster malfunction. The engineer searches a flat text log and sees
Error: 500. No one knows whether the fuel valve stuck, the guidance computer calculated the wrong trajectory, or a sensor disconnected (Complete Blindness). - Distributed AI Tracing:
- The entire mission is assigned a global
trace_id(The Root Trace). - Every operation (sensor read, navigation calculation, thruster firing) is recorded as a nested child span with microsecond start and end timestamps (Span Hierarchy).
- Fuel consumption and sensor payloads are logged at every node (Token & Cost Telemetry).
- The flight engineer clicks the failed span and sees the exact input pressure, computed delta-V, and error code in seconds.
- The entire mission is assigned a global
Historical background
Distributed tracing and APM evolved through three generations:
1. Traditional APM and Dapper (2010–2018)
Google’s Dapper paper established distributed RPC tracing (trace_id, span_id). Tools like Datadog, Jaeger, and Zipkin visualized HTTP service-to-service calls.
2. The OpenTelemetry Standard (2019–2022)
The Cloud Native Computing Foundation (CNCF) merged OpenTracing and OpenCensus into OpenTelemetry (OTel), creating a universal, vendor-neutral telemetry standard.
3. OpenInference and Generative AI Observability (2023–Present)
Generative AI required domain-specific semantic conventions: tracking prompt templates, temperature, token counts (input/cached/output), retrieved vector documents, and LLM-as-a-Judge evaluations (Arize Phoenix, Langfuse, LangSmith).
What it is — and what it is not
Let us define the core components of production AI observability:
What AI Observability Is:
- A structured hierarchy of nested spans capturing every retrieval, prompt formatting, model invocation, and guardrail step.
- Real-time accounting of token usage, prompt caching efficiency, and dollar costs per tenant.
- An integrated feedback collection system linking user ratings (thumbs up/down) to specific execution trace IDs.
What AI Observability Is Not:
- Unstructured
print()statements scattered across Python code. - Storing unencrypted customer passwords or sensitive PII in trace databases.
+-------------------------------------------------------------------------------+
| OPENINFERENCE SPAN ATTRIBUTES |
+---------------------------+-----------------------+---------------------------+
| Semantic Attribute | Data Type | Example Value |
+---------------------------+-----------------------+---------------------------+
| `openinference.span.kind` | string | `LLM`, `RETRIEVER`, `TOOL`|
| `llm.model_name` | string | `claude-3-5-sonnet-20241022`|
| `llm.token_count.prompt` | integer | `1420` |
| `llm.token_count.completion`| integer | `215` |
| `llm.token_count.cached` | integer | `1024` |
| `llm.cost_usd` | float | `0.0031` |
| `retrieval.documents` | JSON list | `[{"doc_id": "d1", "score": 0.88}]`|
+---------------------------+-----------------------+---------------------------+
Why it was created and what problems it solves
AI observability solves the four primary operational challenges of production AI systems:
1. Root-Cause Debugging in Compound AI Pipelines
When a RAG assistant fails, distributed traces reveal whether the failure was caused by empty vector search results, truncated prompt context, or model reasoning errors.
2. Controlling Runaway Token Costs
Real-time token telemetry tracks token usage per user and organization, immediately identifying infinite tool loops or prompt inflation regressions.
3. Continuous Improvement via User Feedback
When a user clicks “thumbs down”, the UI sends the rating attached to the trace_id. Engineers can review the exact trace, identify the failure mode, and add it to the evaluation dataset.
How it works
Let us inspect the complete Production Observability to Evaluation Flywheel:
[1. Live Production User Request]
- User submits query with `trace_id = trace_abc123`
│
▼
[2. Distributed Span Tree Execution]
- Span: Embedding Search (45ms)
- Span: LLM Generation (Claude 3.5 Sonnet, 1,200 tokens, 680ms)
- Span: Output Guardrail Validation (4ms)
│
▼
[3. User Feedback Collection]
- User clicks Thumbs Down: "The answer was outdated."
- Telemetry logs feedback linked to `trace_abc123`
│
▼
[4. Automated Failure Triage & Clustering]
- Semantic clustering tags issue: `category/outdated_context`
- Annotator adds correct ground truth response
│
▼
[5. Append to Golden Eval Dataset]
- New test case added to `golden_eval.jsonl`
- CI regression gate guarantees this bug will never recur
An everyday analogy
Think of an AI distributed tracer like the parcel tracking barcode on an Amazon delivery:
- When you order a package, you do not just wait 5 days in silence (Unobservable System).
- A barcode is scanned at the fulfillment center (Span 1: Order Packed).
- It is scanned entering the sorting hub (Span 2: In Transit).
- It is scanned onto the local delivery van (Span 3: Out for Delivery).
- If your package is delayed, you can see the exact sorting hub where it stopped.
Examples in practice
Let us build a complete Python Trace & Telemetry Engine implementing nested span creation, duration timing, token accounting, cost calculation, and JSON trace export:
import time
import uuid
import json
from typing import Dict, Any, List, Optional
class Span:
def __init__(self, name: str, kind: str, parent_id: Optional[str] = None):
self.span_id = str(uuid.uuid4())[:8]
self.parent_id = parent_id
self.name = name
self.kind = kind
self.start_time = time.time()
self.end_time: Optional[float] = None
self.duration_ms: float = 0.0
self.attributes: Dict[str, Any] = {}
self.children: List['Span'] = []
def set_attribute(self, key: str, value: Any):
self.attributes[key] = value
def finish(self):
self.end_time = time.time()
self.duration_ms = round((self.end_time - self.start_time) * 1000, 2)
def to_dict(self) -> Dict[str, Any]:
return {
"span_id": self.span_id,
"parent_id": self.parent_id,
"name": self.name,
"kind": self.kind,
"duration_ms": self.duration_ms,
"attributes": self.attributes,
"children": [c.to_dict() for c in self.children]
}
class TraceTelemetryEngine:
def __init__(self):
self.traces: Dict[str, Span] = {}
def start_trace(self, name: str) -> Span:
root_span = Span(name=name, kind="ROOT")
self.traces[root_span.span_id] = root_span
return root_span
def create_child_span(self, parent_span: Span, name: str, kind: str) -> Span:
child = Span(name=name, kind=kind, parent_id=parent_span.span_id)
parent_span.children.append(child)
return child
@staticmethod
def calculate_cost(prompt_tokens: int, completion_tokens: int, model: str = "claude-3-5-sonnet") -> float:
# Example pricing: $3.00/M prompt, $15.00/M completion
cost = (prompt_tokens * 3.0 / 1_000_000) + (completion_tokens * 15.0 / 1_000_000)
return round(cost, 6)
if __name__ == "__main__":
engine = TraceTelemetryEngine()
root = engine.start_trace("RAG_Query")
llm_span = engine.create_child_span(root, "LLM_Inference", "LLM")
llm_span.set_attribute("llm.model", "claude-3-5-sonnet")
llm_span.set_attribute("llm.prompt_tokens", 1200)
llm_span.set_attribute("llm.completion_tokens", 250)
llm_span.set_attribute("llm.cost_usd", engine.calculate_cost(1200, 250))
llm_span.finish()
root.finish()
print("Trace Export:", json.dumps(root.to_dict(), indent=2))
Implications: security, privacy, performance, scalability, and cost
1. Privacy Scrubbing in Traces
Traces capture raw input prompts and model completions. Always apply PII scrubbing before writing spans to trace storage so customer credentials and personal data are never exposed in observability dashboards.
2. Telemetry Sampling at Scale
At millions of requests per day, storing 100% of LLM traces can generate massive database bills. Implement Tail-Based Sampling: store 100% of failed traces, high-latency traces (> 2000ms), and user-rated sessions, while sampling only 5% of routine happy-path requests.
Alternatives: free, open source, and commercial
| Tool / Framework | Type | Best Suited For | Key Capabilities |
|---|---|---|---|
| Langfuse | Open Source | Full-stack AI observability | Self-hostable, web UI, eval integrations |
| Arize Phoenix | Open Source | Local tracing & evals | Zero-dependency local notebook tracing |
| LangSmith | Commercial | LangChain / LangGraph tracing | Interactive prompt playground |
| OpenLLMetry | Open Source | OpenTelemetry auto-instrumentation | Drop-in OTel SDK for all LLM providers |
Comparison with related concepts
+-----------------------+-----------------------+-----------------------+
| Dimension | Standard APM Logs | AI Observability |
+-----------------------+-----------------------+-----------------------+
| Trace Payloads | HTTP Headers & Status | Full Prompts & Context|
| Cost Accounting | CPU / Memory metrics | Token-level economics |
| Quality Feedback | HTTP 500 error rates | User thumbs & LLM eval|
| Semantic Analysis | None (Flat text logs) | RAG chunk attribution |
+-----------------------+-----------------------+-----------------------+
When to use it — and when not to
Implement AI Tracing & Observability for:
- All production customer-facing AI applications, multi-step RAG pipelines, and autonomous multi-agent systems.
Do NOT require distributed tracing for:
- Standalone offline data processing scripts running in single batches.
Advanced Topic: OpenInference Semantic Conventions and Distributed Trace Spans
To achieve vendor-neutral observability across different LLM providers and frameworks, production systems adopt OpenInference Semantic Conventions:
# OpenInference Semantic Span Attributes
span.set_attribute("openinference.span.kind", "LLM")
span.set_attribute("llm.model_name", "claude-3-5-sonnet-20241022")
span.set_attribute("llm.invocation_parameters", json.dumps({"temperature": 0.0, "max_tokens": 1024}))
span.set_attribute("llm.token_count.prompt", 1420)
span.set_attribute("llm.token_count.completion", 215)
span.set_attribute("llm.token_count.cached", 1024)
span.set_attribute("llm.cost_usd", 0.0031)
Architectural Blueprint: High-Throughput Real-Time Telemetry Pipeline
In enterprise AI applications serving millions of queries per day, tracing telemetry is decoupled from the main request path:
- Non-Blocking Asynchronous Spans: Spans are buffered in an in-memory ring buffer and dispatched in background batches via OpenTelemetry OTLP exporters.
- Kafka / Redpanda Ingestion Queue: Telemetry spans are ingested into high-throughput streaming topics.
- ClickHouse Analytical Store: Traces and token metrics are written to columnar storage, enabling sub-second analytical queries across billions of spans.
- Grafana / Datadog Dashboards: Real-time dashboards monitor p95 latency, Time to First Token (TTFT), token consumption rate, and cost attribution per tenant.
Production Checklist: AI Observability and Telemetry Readiness
Ensure your production monitoring pipeline implements the following Observability Best Practices:
- Distributed Trace Propagation: Does every user request carry a unique
trace_idpropagated through all downstream spans? - Token Economics Tracking: Are prompt tokens, completion tokens, cached tokens, and financial costs recorded per request?
- Latency Milestone Timing: Are Time to First Token (TTFT), inter-token latency (ITL), and total duration tracked?
- Tail-Based Sampling: Are 100% of errors, high-latency traces, and user feedback sessions persisted in analytical storage?
- User Feedback Integration: Can end-user ratings (thumbs up/down) be tied directly to execution trace spans for triage?
Deep Dive: Real-Time Token Economics, Caching Optimization, and Cost Attribution
Managing cloud GPU expenditures requires granular visibility into token consumption:
- Prompt Caching Telemetry: Track cache hit ratios across LLM providers (e.g. Anthropic Prompt Caching or OpenAI Cached Tokens). High cache hits (> 80%) reduce input token costs by up to 90% and cut Time to First Token (TTFT) in half.
- Tenant-Level Quotas & Rate Limiting: Enforce dynamic budget limits per customer organization, preventing accidental runaway billing from autonomous agent loops.
- Anomaly Detection: Trigger automated alerts in Slack or PagerDuty when average token usage per query spikes by more than 25% over a 1-hour window.
Advanced Topic: Correlating User Feedback with Automated Evaluation Metrics
To validate that your offline evaluation metrics predict real-world customer satisfaction:
- Telemetry Correlation Analysis: Calculate the Pearson correlation between offline composite eval scores and live production user thumbs-up ratios across application features.
- Identifying Blind Spots: If a feature scores 95% on offline benchmarks but receives 20% thumbs-down ratings from live users, the benchmark suffers from dataset mismatch or missing evaluation criteria.
- Continuous Benchmark Calibration: Use production feedback discrepancies to discover unmeasured quality dimensions (such as response latency, UI formatting, or missing domain terminology) and add them to the evaluation suite.
Practical Implementation Tips for Distributed AI Tracing
When instrumenting complex AI microservices with OpenTelemetry:
- Propagate Trace Context via HTTP Headers: Use W3C Trace Context (
traceparent) headers to correlate client requests with backend vector search and model worker pods. - Asynchronous Telemetry Dispatch: Ensure all trace exports execute asynchronously without blocking the user-facing request thread or streaming response buffer.
- Visualizing Multi-Agent Graphs: Group span trees by agent role (e.g. Supervisor, Research Subagent, Code Synthesizer) to easily pinpoint latency bottlenecks across agent handoffs.
Real-World Case Study: Debugging a High-Latency RAG Outage
An enterprise search engine experienced intermittent 5-second latency spikes in production:
- The Investigation: Distributed traces revealed that while model inference took 450ms and vector search took 35ms, the metadata filtering span on the relational database was experiencing table lock contention, adding 4,500ms to the request pipeline.
- The Resolution: Adding an index on the tenant organization ID reduced the metadata query time to 3ms, cutting overall p99 latency by 90%.
Observability Anti-Patterns to Avoid in Production AI Systems
When designing telemetry systems for enterprise LLM workloads, avoid these common engineering pitfalls:
-
Logging Unmasked Sensitive Customer Data: Raw prompts often contain proprietary source code, personal financial figures, or confidential medical details. Always apply token masking before exporting spans to external tracing backends.
-
Synchronous Tracing Blocking Request Latency: Exporting telemetry spans synchronously over HTTP can add 100-300ms to user perceived latency. Always use asynchronous worker queues or background thread pool dispatchers.
-
Neglecting Cache Hit Telemetry: Failing to track cached vs uncached input tokens leads to miscalculated cost projections and missed optimization opportunities for system prompts.
-
Ignoring Distributed Clock Skew: When tracing across distributed multi-agent workers or edge containers, ensure all microservice instances synchronize timestamps via NTP (Network Time Protocol) to prevent inverted span durations in timeline visualizations.
-
Missing Request Correlation IDs: Ensure incoming frontend API gateway requests forward standard
X-Request-IDandX-Correlation-IDheaders to all downstream worker services for seamless cross-tier log linking.
The AI connection
-
A compound AI system has many places to fail — retrieval, prompt assembly, the provider, a guardrail — and only a span tree says which one it was.
-
Token telemetry is cost telemetry. Prompt, completion and cached tokens per request is the only way to attribute a bill or catch an inflating prompt.
-
User feedback attaches to a trace ID, which is what closes the loop: a thumbs-down becomes a specific trace, then a specific test case in Day 310’s dataset.
-
Traces hold the most sensitive text you store. Scrub before writing the span, not before reading the dashboard.
-
Cache hit ratio is the highest-leverage number on the dashboard: above 80% cuts input cost by most of it and halves time to first token.
Knowledge check
- What is a span tree in AI tracing? A hierarchical representation of nested operations (root request -> vector search -> prompt building -> LLM generation -> guardrail validation).
- Why is token telemetry crucial for cost attribution? Because it tracks exact prompt, completion, and cached token consumption, allowing accurate billing per tenant.
- How does user feedback close the continuous evaluation loop? By capturing low-rated production interactions and automatically converting them into labeled test cases in the golden evaluation dataset.
Hands-on exercise
In this lab, you will build an OpenInference-Compatible AI Tracing Engine in Python. Your implementation will:
- Create and manage hierarchical span trees with unique IDs and parent relationships.
- Measure microsecond execution durations for child spans.
- Record model attributes, token counts, and dollar costs.
- Export valid JSON trace trees for observability backends.
Expected output
============================= test session starts ==============================
collected 5 items
tests/test_tracing_engine.py ..... [100%]
============================== 5 passed in 0.02s ===============================
[TRACING ENGINE] Root trace initialized with 3 nested child spans.
[TELEMETRY] Token cost accurately computed: $0.00735 for 1,450 tokens.
[STATUS] Distributed AI tracing engine verified with 100% test pass rate.
Validate your work
Run the pytest test suite:
pytest tests/run_tests.sh
Troubleshooting
- Duration zero: Ensure
.finish()is invoked on spans to compute the elapsed duration. - Hierarchy broken: Verify child spans append correctly to the parent’s
childrenlist.
Common mistakes
- Calculating token cost without accounting for differences between prompt token rates and completion token rates.
Practice assignment
Extend the tracing engine to support Streaming Telemetry Metrics measuring Time to First Token (TTFT) and Inter-Token Latency (ITL).
Extension challenge
Implement a Tail-Based Trace Sampler that saves 100% of traces with duration $> 1000\text{ms}$ or error status while randomly sampling 10% of successful traces.
Quiz
Q1. Why is traditional microservice APM (like standard HTTP request tracing) insufficient for diagnosing complex LLM and RAG applications?
- APM sees HTTP codes, not prompts, chunks, or tokens
- Because APM tools do not support Python
- Because LLMs do not use HTTP requests
- Because tracing makes language models stop working
Show answer
Answer: A. APM sees HTTP codes, not prompts, chunks, or tokens
AI observability requires capturing nested spans containing prompts, completions, token counts, and retrieved context chunks.
Q2. In OpenTelemetry and OpenInference standards, what is a 'Span Tree' in an AI workflow?
- A binary search tree for sorting strings
- A nested graph of every stage, with latency and payloads
- A network cabling topology
- A database index
Show answer
Answer: B. A nested graph of every stage, with latency and payloads
Span trees represent the hierarchical decomposition of a complex compound AI workflow.
Q3. What is the primary benefit of tracking 'Token Telemetry' (Prompt Tokens, Completion Tokens, Cached Tokens) in production?
- It makes the computer run faster
- It replaces database backups
- It attributes cost and catches token inflation
- It encrypts user passwords
Show answer
Answer: C. It attributes cost and catches token inflation
Token telemetry allows financial accounting, rate limit governance, and caching optimization.
Q4. How does closing the 'Production User Feedback Loop' accelerate model improvement?
- By deleting user accounts who give negative feedback
- By restarting the server
- By sending marketing emails to users
- Turn low-rated sessions into regression cases
Show answer
Answer: D. Turn low-rated sessions into regression cases
Negative user feedback is the highest-signal source for discovering real-world edge cases.
Q5. Which open-source telemetry framework is widely adopted for vendor-neutral tracing in AI engineering?
- OpenTelemetry
- Flash Player
- DirectX 12
- Apache Subversion
Show answer
Answer: A. OpenTelemetry
OpenTelemetry and OpenInference provide standardized, vendor-neutral telemetry instrumentation.
Glossary
- AI Observability
- Monitoring, tracing, evaluating, and debugging the runtime behavior and economics of production AI systems.
- Distributed Trace
- An end-to-end record of an entire user request through all downstream services, LLMs, and databases.
- Span
- A single contiguous unit of work within a trace (e.g. vector search, LLM completion, guardrail check).
- OpenInference
- An open standard extending OpenTelemetry semantic conventions specifically for AI systems.
- Token Telemetry
- Tracking input tokens, cached tokens, output tokens, and financial costs per request.
- Feedback Loop
- Capturing user ratings (thumbs up/down, corrections) and feeding failed interactions into evaluation datasets.
- Time to First Token (TTFT)
- The latency in milliseconds from sending a prompt until receiving the first streamed token.
- Inter-Token Latency (ITL)
- The average time in milliseconds between consecutive streamed tokens.
Sources and further reading
- OpenInference Semantic Conventions for Generative AI (Arize AI, 2024) — GitHub (accessed 2026-08-30)
- OpenTelemetry Documentation and Specification (CNCF, 2024) — Cloud Native Computing Foundation (accessed 2026-08-30)
- Langfuse: Open Source LLM Engineering Platform (Langfuse, 2024) — GitHub (accessed 2026-08-30)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.