> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://www.comet.com/docs/opik/llms.txt.

# OpenTelemetry

Opik provides native support for OpenTelemetry (OTel), allowing you to instrument
your ML/AI applications with distributed tracing. This guide will show you how
to directly integrate OpenTelemetry SDKs with Opik.

> **Note**
>
> OpenTelemetry integration in Opik currently supports HTTP transport. We're actively working on expanding the feature
> set - stay tuned for updates!

## OpenTelemetry Endpoint Configuration

### Base Endpoint

To start sending traces to Opik, configure your OpenTelemetry exporter with one of these endpoints:

#### Opik Cloud

```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT="https://www.comet.com/opik/api/v1/private/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=<your-api-key>,projectName=<your-project-name>,Comet-Workspace=<your-workspace-name>"
```

#### Self-hosted deployment

```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:5173/api/v1/private/otel"
```

#### Enterprise deployment

```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<comet deployment url>/opik/api/v1/private/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=<your-api-key>,projectName=<your-project-name>,Comet-Workspace=<your-workspace-name>"
```

### Signal-Specific Endpoint

If your OpenTelemetry setup requires signal-specific configuration, you can use
the traces endpoint. This is particularly useful when different signals (traces,
metrics, logs) need to be sent to different endpoints:

```bash wordWrap
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://<YOUR-OPIK-INSTANCE>/api/v1/private/otel/v1/traces"
```

## Custom via OpenTelemetry SDKs

You can use any OpenTelemetry SDK to send traces directly to Opik. OpenTelemetry
provides SDKs for many languages (C++, .NET, Erlang/Elixir, Go, Java, JavaScript,
PHP, Python, Ruby, Rust, Swift). This extends Opik's language support beyond the
official SDKs (Python and TypeScript). For more instructions, visit the
[OpenTelemetry documentation](https://opentelemetry.io/docs/languages/).

Here's a Python example showing how to set up OpenTelemetry with Opik:

```python wordWrap
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
    OTLPSpanExporter
)
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# For Comet-hosted installations
OPIK_ENDPOINT = "https://<COMET-SERVER>/api/v1/private/otel/v1/traces"
API_KEY = "<your-api-key>"
PROJECT_NAME = "<your-project-name>"
WORKSPACE_NAME = "<your-workspace-name>"

# Initialize the trace provider
provider = TracerProvider()
processor = BatchSpanProcessor(
    OTLPSpanExporter(
        endpoint=OPIK_ENDPOINT,
        headers={
            "Authorization": API_KEY,
            "projectName": PROJECT_NAME,
            "Comet-Workspace": WORKSPACE_NAME
        }
    )
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
```

> **Tip**
>
> In order to track OpenAI calls, you need to use the OpenTelemetry instrumentations
> for OpenAI:
>
> ```bash wordWrap
> pip install opentelemetry-instrumentation-openai
> ```
>
> And then instrument your OpenAI client:
>
> ```python wordWrap
> from opentelemetry.instrumentation.openai import OpenAIInstrumentor
>
> OpenAIInstrumentor().instrument()
> ```

> **Warning**
>
> Make sure to import the `http` trace exporter (`opentelemetry.exporter.otlp.proto.http.trace_exporter`), if you use
> the GRPC exporter you will face errors.

## How Opik builds traces from spans

OpenTelemetry sends spans only. Opik creates a trace from the root span of each OpenTelemetry trace: the span that has
no parent span. The name, input, output, metadata, tags and errors of the root span become the values of the trace.

Opik treats a span as a root span when one of these conditions is true:

* The `parentSpanId` of the span is empty.
* The `parentSpanId` of the span is equal to its `traceId`.

> **Warning**
>
> Every trace must have one root span. If all the spans of a trace have a parent, Opik stores the spans but does not
> create the trace. The spans then show in the **Spans** tab, but the **Traces** tab stays empty and the span details do
> not load. See [Troubleshooting](#troubleshooting) for how to fix it.

> **Note**
>
> A span with the `opik.trace_id` attribute attaches to an Opik trace that already exists, so Opik does not create a
> trace for it. See [Distributed traces](/docs/opik/tracing/advanced/log_distributed_traces).

## Supported attribute conventions

Opik reads the attributes of each span and maps them to the fields of the Opik span. Opik supports the
[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), the
[OpenInference conventions](https://github.com/Arize-ai/openinference/tree/main/spec) and a set of general Opik
attributes. Opik also reads the attributes of some framework instrumentations, such as Logfire, Pydantic AI, LiteLLM,
smolagents, LiveKit and Langfuse.

| Opik field  | GenAI                                                                                                                                                             | OpenInference       | Opik general         |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | -------------------- |
| Input       | `gen_ai.prompt`, `gen_ai.input.*`, `gen_ai.system_instructions`, `gen_ai.request.*`, `gen_ai.tool.call.arguments`                                                 | `input.value`       | `input`, `input.*`   |
| Output      | `gen_ai.completion`, `gen_ai.output.*`, `gen_ai.response.*`, `gen_ai.tool.call.result`                                                                            | `output.value`      | `output`, `output.*` |
| Model       | `gen_ai.request.model`, `gen_ai.response.model`                                                                                                                   | `llm.model_name`    |                      |
| Provider    | `gen_ai.system`, `gen_ai.provider.name`                                                                                                                           |                     |                      |
| Token usage | `gen_ai.usage.*`                                                                                                                                                  | `llm.token_count.*` |                      |
| Cost        | `gen_ai.usage.cost`                                                                                                                                               |                     |                      |
| Metadata    | `gen_ai.request.temperature`, `gen_ai.response.id`, `gen_ai.response.finish_reasons`, `gen_ai.operation.name`, `gen_ai.agent.*`, `gen_ai.tool.*`, `gen_ai.cost.*` |                     | `opik.metadata.*`    |
| Tags        |                                                                                                                                                                   |                     | `opik.tags`          |
| Thread ID   | `gen_ai.conversation.id`                                                                                                                                          |                     | `thread_id`          |

Some other rules apply:

* A more specific attribute has priority over a prefix. For example, `gen_ai.request.model` sets the model, and the
  other `gen_ai.request.*` attributes go to the input.
* Opik renames the `input_tokens` and `output_tokens` usage keys to `prompt_tokens` and `completion_tokens`. If the span
  has no `total_tokens`, Opik calculates it.
* Opik sets the span type to `llm` when the span has a GenAI or OpenInference model or token usage attribute. Opik sets
  the span type to `tool` when the span has `gen_ai.tool.call.arguments` or `gen_ai.tool.call.result`.
* When the value of `input`, `output` or `opik.metadata` is a JSON object in a string, Opik adds the keys of the object
  to the field.
* Opik adds the span events to the metadata of the span. An `exception` event, or a span status of `ERROR`, becomes the
  error of the span.

> **Note**
>
> Opik puts each attribute that it does not recognize in the **Input** field of the span, with the attribute name as
> the key. If the input of your spans shows only attribute names from your framework, set `input.value` and
> `output.value` (or the GenAI attributes) on the spans that you want to read in Opik.

## Opik-specific span attributes

Opik reads a small set of span attributes and maps them to Opik fields. Set them with
the standard OpenTelemetry API on any span.

| Attribute                                              | Effect                                                                                                                                                                                                       |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `opik.project_name`                                    | Sends the span to this project instead of the project in the `projectName` header. Set it as a resource attribute, so that every span has it. See [Routing spans to a project](#routing-spans-to-a-project). |
| `opik.tags`                                            | Adds tags to the span. On the root span, Opik also adds the tags to the trace, so you can filter your traces by tag.                                                                                         |
| `opik.metadata.<key>`                                  | Adds `<key>` to the metadata of the span. The metadata of the root span also becomes the metadata of the trace.                                                                                              |
| `thread_id`                                            | Groups traces into one conversational thread. See [Multi-turn conversations](/docs/opik/evaluation/evaluate_threads).                                                                                                  |
| `opik.trace_id`, `opik.parent_span_id`, `opik.span_id` | Attaches the span to an Opik trace and parent span that already exist. See [Distributed traces](/docs/opik/tracing/advanced/log_distributed_traces).                                                                   |

### Tagging traces and spans

Opik accepts three formats for the `opik.tags` attribute:

* A list of strings: `["production", "chatbot"]`
* A JSON array in a string: `'["production", "chatbot"]'`
* A comma-separated string: `"production,chatbot"`

```python wordWrap
with tracer.start_as_current_span("chatbot_conversation") as conversation_span:
    # The root span carries the tags, so the trace carries them too
    conversation_span.set_attribute("opik.tags", ["production", "chatbot"])
    conversation_span.set_attribute("opik.metadata.environment", "staging")

    with tracer.start_as_current_span("llm_completion") as llm_span:
        # A child span carries the tags on the span only
        llm_span.set_attribute("opik.tags", ["llm-call"])
```

> **Note**
>
> Set `opik.tags` on the root span if you want to filter traces by tag. Tags on a child span apply to that span only.

### Routing spans to a project

Some OpenTelemetry Collector processors, such as `tail_sampling`, do not forward the HTTP headers of the incoming
request. In that case, set the project with the `opik.project_name` attribute instead of the `projectName` header.

Opik picks the project of each span on its own, in this order:

1. The `opik.project_name` attribute of the span
2. The `opik.project_name` attribute of the resource of the span
3. The `projectName` header
4. The `Default Project`

Opik ignores a value longer than 150 characters.

> **Warning**
>
> Every span must have the `opik.project_name` attribute. Opik does not copy the project from the root span to the
> other spans: OpenTelemetry exports the spans of a long trace in more than one batch, and the root span usually comes
> last. If you set the attribute on the root span only, the other spans go to the project in the header, and they do
> not show under any trace. Set the project as a resource attribute, so that every span has it.

#### Set the project on every span

A resource attribute applies to every span that the process sends. Use one of these methods:

#### Environment variable

The OpenTelemetry SDK reads the resource attributes from `OTEL_RESOURCE_ATTRIBUTES`:

```bash
export OTEL_RESOURCE_ATTRIBUTES="opik.project_name=<your-project-name>"
```

#### Python SDK

Add the attribute to the resource of the tracer provider:

```python
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider(
    resource=Resource.create({"opik.project_name": "<your-project-name>"})
)
```

#### OpenTelemetry Collector

Set the attribute with the `transform` processor. This example copies `service.name`, so that each service sends to
its own project:

```yaml
processors:
  transform/opik_project:
    trace_statements:
      - context: resource
        statements:
          - set(attributes["opik.project_name"], attributes["service.name"])

service:
  pipelines:
    traces:
      processors: [transform/opik_project, tail_sampling]
```

To send all the spans to one project, use a fixed value such as `"<your-project-name>"` instead of
`attributes["service.name"]`.

> **Note**
>
> All the spans of a trace must have the same project. If one trace goes through services that send to different
> projects, the spans that are not in the project of the root span do not show under any trace.

## Send spans to Opik and to other backends

One OpenTelemetry `TracerProvider` can send each span to more than one backend. Add one `BatchSpanProcessor` for each
exporter. The frameworks that use OpenTelemetry (such as Strands Agents, LangGraph, LlamaIndex and CrewAI) send their
spans to the global tracer provider, so one setup covers all of them.

#### New tracer provider

Create the tracer provider, and add one exporter for Opik and one for your other backend:

```python wordWrap
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()

# Opik
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://www.comet.com/opik/api/v1/private/otel/v1/traces",
            headers={
                "Authorization": "<your-api-key>",
                "Comet-Workspace": "<your-workspace-name>",
                "projectName": "<your-project-name>",
            },
        )
    )
)

# Other backend
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="https://<other-backend>/v1/traces"))
)

trace.set_tracer_provider(provider)
```

#### Existing tracer provider

Some platforms set up the tracer provider before your code starts. For example, Amazon Bedrock AgentCore runs your
agent with the AWS Distro for OpenTelemetry (ADOT), which sends the spans to CloudWatch. Add the Opik exporter to that
tracer provider:

```python wordWrap
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = trace.get_tracer_provider()
if not isinstance(provider, TracerProvider):
    # No SDK tracer provider is set yet, so create one
    provider = TracerProvider()
    trace.set_tracer_provider(provider)

provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://www.comet.com/opik/api/v1/private/otel/v1/traces",
            headers={
                "Authorization": "<your-api-key>",
                "Comet-Workspace": "<your-workspace-name>",
                "projectName": "<your-project-name>",
            },
        )
    )
)
```

Run this code once, before the agent starts its first span.

> **Warning**
>
> Give the Opik endpoint and headers to the exporter in code. Do not use the `OTEL_EXPORTER_OTLP_ENDPOINT` and
> `OTEL_EXPORTER_OTLP_HEADERS` environment variables for Opik: all the OTLP exporters in the process read them, so the
> other backend also gets the Opik endpoint and your Opik API key.

When the platform continues an incoming trace, the first span of your agent has a parent. In this case, add a root span
for each invocation, as shown in [Troubleshooting](#troubleshooting).

## Troubleshooting

#### The exporter gets a 401 or 403 error

* Make sure that the `Authorization` header contains your Opik API key, without the `Bearer ` prefix.
* Make sure that `Comet-Workspace` is the name of a workspace that your API key can access.

#### Spans show in the Spans tab, but the Traces tab is empty

No span of the trace is a root span, so Opik did not create the trace.

This problem occurs when the instrumentation continues a trace that started in a different process. For example, a
runtime such as Amazon Bedrock AgentCore or an API gateway sends a trace context with each request. The first span
of your agent is then a child of a span that Opik never receives.

To find out if you have this problem, print your spans with the `ConsoleSpanExporter` and look at the `parent_id`
field. At least one span of each trace must have `"parent_id": null`.

To fix the problem, start a new root span at the start of each agent invocation. Pass an empty `Context` so that
the span does not use the incoming trace context:

```python wordWrap
import json

from opentelemetry import trace
from opentelemetry.context import Context
from opentelemetry.trace import Link

tracer = trace.get_tracer("my-agent")


def invoke(payload: dict) -> str:
    # Keep a link to the incoming trace, so that other backends can still connect the two traces
    incoming = trace.get_current_span().get_span_context()
    links = [Link(incoming)] if incoming.is_valid else []

    with tracer.start_as_current_span("agent_invocation", context=Context(), links=links) as root_span:
        root_span.set_attribute("input.value", json.dumps(payload))
        result = str(agent(payload["prompt"]))
        root_span.set_attribute("output.value", result)
        return result
```

All the spans that the agent creates in the `with` block are children of `agent_invocation`, so Opik creates one
trace for each invocation.

#### Spans show under a different trace, or under no trace

All the spans of a trace must go to the same project. If you set `opik.project_name`, set it as a resource
attribute. See [Routing spans to a project](#routing-spans-to-a-project).

#### The input of a span shows framework attributes and no model or token usage

Your instrumentation uses attribute names that Opik does not recognize. Opik puts these attributes in the input.
Set the attributes of one of the [supported conventions](#supported-attribute-conventions) on your spans.

#### The exporter fails with a protocol error

Opik accepts OTLP over HTTP only. Use the `http` exporter
(`opentelemetry.exporter.otlp.proto.http.trace_exporter`), not the gRPC exporter.