What you will learn
How to record each stage of an AI request as a trace using OpenTelemetry, which details to attach to model calls, and what to keep out of telemetry.
Why tracing matters for AI systems
When a user says "the answer was slow and wrong," you need to know which stage failed: authentication, retrieval, the model call, validation, or a downstream API. A trace is a tree of timed spans for one request. Each span records a stage, its duration, its status, and a few attributes. Logs tell you what happened; traces show where the time and failures went.
OpenTelemetry is the vendor-neutral standard for this. You instrument your code once and export to whatever backend you use — an open-source tracing UI, a cloud provider's monitoring, or an LLM-observability tool that accepts OpenTelemetry data.
Spans for a knowledge-assistant request
ask (request_id, tenant scope, outcome)
├── retrieve (candidate count, index version)
├── generate (model, token usage, finish status)
└── validate (passed / rejected reason)
OpenTelemetry publishes semantic conventions for generative AI, with attribute names such as gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens. These conventions are still evolving, so pin the version you follow and check the current specification before relying on a name in dashboards.
Instrument the request
Install the checked version with python -m pip install "opentelemetry-sdk==1.45.2" in your environment. This example exports spans to memory so you can test them; in a real service you configure an exporter instead.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import Status, StatusCode
exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("knowledge-service")
def fake_model(question, evidence):
return {"answer": evidence[0], "usage": {"input": 42, "output": 9}}
def ask(request_id, question, model="fixture-model"):
with tracer.start_as_current_span("ask") as root:
root.set_attribute("app.request_id", request_id)
with tracer.start_as_current_span("retrieve") as span:
evidence = ["Reset your password in Settings."] if "reset" in question else []
span.set_attribute("app.retrieval.candidates", len(evidence))
if not evidence:
root.set_attribute("app.outcome", "unsupported")
return None
with tracer.start_as_current_span("generate") as span:
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.request.model", model)
try:
result = fake_model(question, evidence)
except Exception as error:
span.set_status(Status(StatusCode.ERROR, type(error).__name__))
raise
span.set_attribute("gen_ai.usage.input_tokens", result["usage"]["input"])
span.set_attribute("gen_ai.usage.output_tokens", result["usage"]["output"])
root.set_attribute("app.outcome", "answered")
return result["answer"]
ask("r-1", "reset password")
spans = {span.name: span for span in exporter.get_finished_spans()}
assert set(spans) == {"ask", "retrieve", "generate"}
assert spans["generate"].attributes["gen_ai.usage.input_tokens"] == 42
assert spans["generate"].parent.span_id == spans["ask"].context.span_id
assert "question" not in str(dict(spans["ask"].attributes))
Notice what is recorded: stage names, counts, model, token usage, outcome. Notice what is not: the question text, the evidence, and the answer.
Keep sensitive data out
Prompts, retrieved documents, and answers often contain personal or confidential data. Do not attach them to spans by default. If you need examples for debugging or evaluation, store them in a separately access-controlled dataset with retention rules, and link to them by ID. Never record credentials or API keys — check that your HTTP client instrumentation does not capture authorization headers.
Use traces to answer real questions
- Where does the latency go? Compare retrieval and generation span durations at p50 and p95.
- What does a request cost? Sum token usage per outcome, including failed and retried attempts.
- Did the release break something? Filter spans by application and prompt version before and after the change.
- Which tenant is affected? Use a tenant scope identifier that your access rules allow you to see — not a customer name in plain text.
Exercise
Add spans to your knowledge service for retrieve, generate, and validate. Force a provider timeout and an invalid model output, and confirm each produces a span with an error status and a clear reason. Export to a local tracing UI if you have one; otherwise use the in-memory exporter in tests.
Check: one request produces one trace with correctly nested spans; failures are visible without reading logs; no prompt, document, answer, or credential appears in any attribute.