Tracing & spans

A trace is the hierarchical view of a single run (agent, step, tool call, LLM call), with a duration and token count on each node. It renders as the Trace tab on the run detail page (shown whenever a run has spans).

A span is one timed node in that tree: a name, a start/end, a parent, and attributes (model, tokens, …). Unlike a flat event, a span has a duration and a place in the hierarchy, so you can see where the time and tokens went.

Telemetry reaches AgentOS three ways. You don't have to pick one.

---

1. Hosted agents — automatic

Agents that run on AgentOS are traced for you. Every run records a root agent span with an LLM span per model call and a tool span per tool call (with durations and token usage). Open the run → Trace. Nothing to configure. A script's run is traced the same way from a root script span (Script run, 0→0 tok: a script calls no model), and a tool call that started a child run shows the child as an agent or a script.

---

2. Your own agents — the SDK span API

If you run your agent yourself and report runs via the SDK, wrap work in spans. Spans are timed and nestable; call .end() to record one.

import { AgentOS } from "@agentos-sdk/core";

const aos = new AgentOS(); // reads AGENTOS_API_KEY and AGENTOS_AGENT_ID

await aos.withRun(async (run) => {
  const step = run.startSpan("classify email", { kind: "step" });
  const llm = step.child("gpt-4.1", { kind: "llm" });
  // ... call the model ...
  llm.end({ input_tokens: 1240, output_tokens: 90 }); // tokens show on the span
  step.end();
});

from agentos import AgentOS

aos = AgentOS()  # reads AGENTOS_API_KEY and AGENTOS_AGENT_ID

with aos.start_run() as run:
    step = run.start_span("classify email", kind="step")
    llm = step.child("gpt-4.1", kind="llm")
    # ... call the model ...
    llm.end({"input_tokens": 1240, "output_tokens": 90})
    step.end()

Span kinds (agent, step, llm, tool, retrieval) drive the chip color and grouping in the trace view. Kind is open text — any value is accepted; unknown kinds render neutrally. Put input_tokens / output_tokens in the .end() payload to get the per-span token label.

---

3. OpenTelemetry — point your exporter at AgentOS

If your agent is already instrumented with OpenTelemetry (LangChain, the OpenAI Agents SDK, Vercel AI SDK, LlamaIndex, and others emit GenAI spans natively), you don't need any AgentOS-specific code — just send your traces here.

Endpoint (OTLP/HTTP, JSON encoding):

POST https://agentos-ai.dev/api/ingest/v1/traces
Authorization: Bearer <your AgentOS key>
Content-Type: application/json

Set the standard OTLP env vars on your app:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://agentos-ai.dev/api/ingest/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer <your AgentOS key>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/json"

Which agent does a trace belong to?

• With an agent-scoped key (aos_agent_…), every trace is attributed to that agent — nothing else to set.

• With a workspace key (aos_ws_…), set the agent on the OTel resource:

  agentos.agent_id = <your agent uuid>
  

How it maps:

OpenTelemetry	AgentOS
one trace (trace_id)	one run
span (span_id / parent_span_id)	a node in the trace tree
gen_ai.request.model	the LLM span's name
gen_ai.usage.input_tokens / output_tokens	per-span + run token totals
gen_ai.operation.name = execute_tool / gen_ai.tool.name	a tool span
root span status = ERROR	the run is marked failed

Notes:

• JSON only — set OTEL_EXPORTER_OTLP_PROTOCOL=http/json (protobuf isn't accepted).
• Spans for one trace can arrive across multiple exports; they attach to the same run.
• Limits: up to 1,000 spans per request; the same per-minute rate limit as events (Authentication & API keys).

---

Where traces show up

• Run detail → Trace tab — the tree, with per-span durations, a relative duration bar, and token counts.
• Token usage rolls up to the run (and into cost/usage dashboards) regardless of source.

Spans are additive — a run with no spans simply has no Trace tab, and the flat event stream is unaffected.