Tracing & spans

A trace is the hierarchical view of a single run — agent → step → tool call → LLM call — with a duration and token count on each node. It renders as the Trace tab on the run detail page (shown whenever a run has spans).

A span is one timed node in that tree: a name, a start/end, a parent, and attributes (model, tokens, …). Unlike a flat event, a span has a duration and a place in the hierarchy, so you can see where the time and tokens went.

Telemetry reaches AgentOS three ways. You don't have to pick one.

1. Hosted agents — automatic

Agents that run on AgentOS are traced for you. Every run records a root agent span with an LLM span per model call and a tool span per tool call (with durations and token usage). Open the run → Trace. Nothing to configure.

2. Your own agents — the SDK span API

If you run your agent yourself and report runs via the SDK, wrap work in spans. Spans are timed and nestable; call .end() to record one.

import { AgentOS } from "@agentos-sdk/core";

const aos = new AgentOS(); // reads AGENTOS_API_KEY and AGENTOS_AGENT_ID

await aos.withRun(async (run) => {
  const step = run.startSpan("classify email", { kind: "step" });
  const llm = step.child("gpt-4.1", { kind: "llm" });
  // ... call the model ...
  llm.end({ input_tokens: 1240, output_tokens: 90 }); // tokens show on the span
  step.end();
});

from agentos import AgentOS

aos = AgentOS()  # reads AGENTOS_API_KEY and AGENTOS_AGENT_ID

with aos.start_run() as run:
    step = run.start_span("classify email", kind="step")
    llm = step.child("gpt-4.1", kind="llm")
    # ... call the model ...
    llm.end({"input_tokens": 1240, "output_tokens": 90})
    step.end()

Span kinds (agent, step, llm, tool, retrieval) drive the chip color and grouping in the trace view. Kind is open text — any value is accepted; unknown kinds render neutrally. Put input_tokens / output_tokens in the .end() payload to get the per-span token label.

3. OpenTelemetry — point your exporter at AgentOS

If your agent is already instrumented with OpenTelemetry (LangChain, the OpenAI Agents SDK, Vercel AI SDK, LlamaIndex, and others emit GenAI spans natively), you don't need any AgentOS-specific code — just send your traces here.

Endpoint (OTLP/HTTP, JSON encoding):

POST https://agentos-ai.dev/api/ingest/v1/traces
Authorization: Bearer <your AgentOS key>
Content-Type: application/json

Set the standard OTLP env vars on your app:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://agentos-ai.dev/api/ingest/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer <your AgentOS key>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/json"

Which agent does a trace belong to?

With an agent-scoped key (aos_agent_…), every trace is attributed to that agent — nothing else to set.
With a workspace key (aos_ws_…), set the agent on the OTel resource:
```
agentos.agent_id = <your agent uuid>
```

How it maps:

OpenTelemetry	AgentOS
one trace (`trace_id`)	one run
span (`span_id` / `parent_span_id`)	a node in the trace tree
`gen_ai.request.model`	the LLM span's name
`gen_ai.usage.input_tokens` / `output_tokens`	per-span + run token totals
`gen_ai.operation.name = execute_tool` / `gen_ai.tool.name`	a tool span
span `status = ERROR`	the run is marked failed

Notes:

JSON only — set OTEL_EXPORTER_OTLP_PROTOCOL=http/json (protobuf isn't accepted).
Spans for one trace can arrive across multiple exports; they attach to the same run.
Limits: up to 1,000 spans per request; the same per-minute rate limit as events (Authentication & API keys).

Where traces show up

Run detail → Trace tab — the tree, with per-span durations, a relative duration bar, and token counts.
Token usage rolls up to the run (and into cost/usage dashboards) regardless of source.

Spans are additive — a run with no spans simply has no Trace tab, and the flat event stream is unaffected.