Observability for Pixeltable with Opik

Pixeltable is declarative data infrastructure for multimodal AI applications. See the Pixeltable documentation for the data model.

Each table operation is one trace. With span_level set to debug, that trace includes a span per row and per UDF call, and OpenInference LLM spans nest under the UDF that made the call.

Getting started

To use the Pixeltable integration with Opik, install Pixeltable’s OpenTelemetry extra, the OpenAI SDK, and the OpenInference OpenAI instrumentor. Pixeltable requires Python 3.11 or newer.

pip install 'pixeltable[otel]' openai openinference-instrumentation-openai

The pixeltable[otel] extra installs opentelemetry-instrumentation-pixeltable (0.1.0 on PyPI) together with the OpenTelemetry SDK and the OTLP exporters that init() needs.

Environment configuration

Configure your environment variables based on your Opik deployment. init() reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS when you leave endpoint and headers unset. Pass endpoint and headers to init() instead, as the example below does.

Give the Opik endpoint and headers to the exporter in code. Do not use the OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS environment variables for Opik: all the OTLP exporters in the process read them, so the other backend also gets the Opik endpoint and your Opik API key.

init() uses the OTLP/HTTP exporter by default and appends /v1/traces to the endpoint. Pass Opik’s base URL. The path is not appended a second time when the URL already ends in /v1/traces.

Opik Cloud

Enterprise deployment

Self-hosted instance

If you are using Opik Cloud, you can set the following environment variables:

export OPIK_API_KEY='<your-api-key>'
export OPIK_WORKSPACE='default'
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=<your-api-key>,Comet-Workspace=default'

To log the traces to a specific project, you can add the projectName parameter to the OTEL_EXPORTER_OTLP_HEADERS environment variable:

export OTEL_EXPORTER_OTLP_HEADERS='Authorization=<your-api-key>,Comet-Workspace=default,projectName=<your-project-name>'

You can also update the Comet-Workspace parameter to a different value if you would like to log the data to a different workspace.

The Authorization header is the raw Opik API key. Do not add a Bearer prefix.

Using Opik with Pixeltable

Call init() once, before the first Pixeltable operation. span_level='debug' adds the per-row and per-UDF spans. The default 'info' keeps the operation span plus its catalog, store, and SQL spans, and omits the row and UDF spans. Instrument OpenAI after init() so its spans use the tracer provider Pixeltable configured and nest under the UDF span.

Opik’s OTLP endpoint accepts traces. Pass metrics=False so init() does not also export metrics.

Set OPENAI_API_KEY before you run the script. Pixeltable reads that variable when the OpenAI UDF runs. Export OPIK_API_KEY and OPIK_WORKSPACE as shown above. The script sends OPIK_API_KEY as Authorization and OPIK_WORKSPACE as Comet-Workspace.

pixeltable_opik.py

import os
import opentelemetry.instrumentation.pixeltable as pxt_otel
import pixeltable as pxt
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry import trace
from pixeltable.functions.openai import chat_completions
pxt_otel.init(
    endpoint='https://www.comet.com/opik/api/v1/private/otel',
    headers=(
        f"Authorization={os.environ['OPIK_API_KEY']},"
        f"Comet-Workspace={os.environ['OPIK_WORKSPACE']},"
        'projectName=pixeltable-demo'
    ),
    service_name='pixeltable-app',
    span_level='debug',
    metrics=False,
)
OpenAIInstrumentor().instrument()
pxt.create_dir('opik_demo', if_exists='replace_force')
prompts = pxt.create_table(
    'opik_demo.prompts',
    {'prompt': pxt.String},
    if_exists='replace_force',
)
prompts.add_computed_column(
    response=chat_completions(
        messages=[{'role': 'user', 'content': prompts.prompt}],
        model='gpt-4o-mini',
    )
)
prompts.insert([{'prompt': 'Name one fact about Paris.'}])
trace.get_tracer_provider().force_flush()

init() installs a batch span processor. force_flush() exports the queued spans before a short-lived script exits.

For a self-hosted Opik server, use the same script with the local base URL and no auth headers:

pxt_otel.init(
    endpoint='http://localhost:5173/api/v1/private/otel',
    service_name='pixeltable-app',
    span_level='debug',
    metrics=False,
)

To choose a project on that server, pass headers='projectName=<your-project-name>'. If you exported the variables in the section above, omit endpoint and headers and init() reads them from the environment.

What gets traced

With this setup, Pixeltable exports:

Table operations: one trace per operation, rooted at a span such as pixeltable.insert or pixeltable.add_computed_column
Rows and UDFs (span_level='debug'): a pixeltable.row span for each row, with a nested pixeltable.udf.<name> span for each UDF call
LLM calls: OpenInference OpenAI spans nested under the UDF that called the model

Span names and the rest of the tree are documented in the Pixeltable observability guide.

Further improvements

If you have any questions or suggestions for improving the Pixeltable integration, please open an issue on our GitHub repository.