Observability for TrueFoundry with Opik

TrueFoundry AI Gateway is the proxy layer that sits between your applications and the LLM providers and MCP Servers. It is an enterprise-grade platform that enables users to access 1000+ LLMs using a unified interface while taking care of observability and governance.

The gateway exports OpenTelemetry traces, and Opik ingests them at its native OTLP endpoint. You get one trace for every request that passes through the gateway, without a change to your application code.

Gateway Overview

The TrueFoundry AI Gateway gives you these features:

• Unified OpenAI-compatible endpoint: One endpoint routes to any supported model, from a commercial provider or from your own deployment.
• Native MCP support: Connect enterprise tools to your agents, and apply OAuth2, RBAC, and metadata policies to each tool call.
• Routing and reliability: Load balancing by weight, latency, or priority, with automatic fallback chains and retries.
• Governance controls: Rate limits, quotas per user and per team, budget alerts, spend caps, and scoped API keys with RBAC.
• Observability: Token, latency, cost, and error metrics for each request, with OpenTelemetry export to a platform such as Opik.
• Data sovereignty: VPC and on-premises deployment options for compliance and data privacy.

To learn more about the gateway, see the TrueFoundry AI Gateway documentation.

Account Setup

Comet provides a hosted version of the Opik platform. Simply create an account and grab your API Key.

You can also run the Opik platform locally, see the installation guide for more information.

Integration Options

You can connect TrueFoundry to Opik in two ways. The two options work together, so you can also use both.

Gateway Trace Export

The gateway exports OpenTelemetry traces over OTLP. Opik accepts these traces at its native OTLP endpoint. TrueFoundry also documents this setup, with a screenshot of each field, on the Comet Opik page in the TrueFoundry documentation.

Prerequisites

• A TrueFoundry account with access to the AI Gateway. See the TrueFoundry quick start.
• An Opik account, on Opik Cloud or on your own deployment.
• Your Opik API key, your Opik workspace name, and the name of the Opik project for the traces.

Trace Endpoint

The Opik OTLP base endpoint is /api/v1/private/otel. TrueFoundry sends traces to the signal-specific endpoint, which is the base endpoint with /v1/traces at the end. TrueFoundry does not add this path for you, so enter the full path in the Endpoint field.

https://www.comet.com/opik/api/v1/private/otel/v1/traces

Configuring the Exporter

Configuration Reference

Troubleshooting

Application Tracing

The gateway exposes an OpenAI-compatible API, so you can use the Opik OpenAI SDK wrapper to log gateway calls as generations in Opik. Use this option when you want to group several LLM calls into one trace.

Installation

First, ensure you have both opik and openai packages installed:

$
pip install opik openai

Configuring Opik

Configure the Opik Python SDK for your deployment type. See the Python SDK Configuration guide for detailed instructions on:

• CLI configuration: opik configure
• Code configuration: opik.configure()
• Self-hosted vs Cloud vs Enterprise setup
• Configuration files and environment variables

Configuring TrueFoundry

You need a TrueFoundry API key and the base URL of your gateway.

• Base URL: https://gateway.truefoundry.ai for TrueFoundry SaaS. For a self-hosted gateway, get the base URL from the Code Snippet tab of the TrueFoundry playground.
• API key: Create a Personal Access Token or a Virtual Account Token in the Access section of the TrueFoundry platform. See Generating TrueFoundry API keys.

Set your configuration as environment variables:

$
export TRUEFOUNDRY_API_KEY="<your-truefoundry-api-key>"
$
export TRUEFOUNDRY_BASE_URL="<your-truefoundry-base-url>"

Or set them programmatically:

1
import os
2
import getpass
3
4
if "TRUEFOUNDRY_API_KEY" not in os.environ:
5
    os.environ["TRUEFOUNDRY_API_KEY"] = getpass.getpass("Enter your TrueFoundry API key: ")
6
7
if "TRUEFOUNDRY_BASE_URL" not in os.environ:
8
    os.environ["TRUEFOUNDRY_BASE_URL"] = input("Enter your TrueFoundry base URL: ")

Simple LLM Call

1
import os
2
from opik.integrations.openai import track_openai
3
from openai import OpenAI
4
5
# Create an OpenAI client with TrueFoundry's base URL
6
client = OpenAI(
7
    api_key=os.environ["TRUEFOUNDRY_API_KEY"],
8
    base_url=os.environ["TRUEFOUNDRY_BASE_URL"]
9
)
10
11
# Wrap the client with Opik tracking
12
client = track_openai(client, project_name="truefoundry-integration-demo")
13
14
# Make a chat completion request
15
response = client.chat.completions.create(
16
    model="openai-main/gpt-4o",
17
    messages=[
18
        {"role": "system", "content": "You are a knowledgeable AI assistant."},
19
        {"role": "user", "content": "What is the largest city in France?"}
20
    ]
21
)
22
23
# Print the assistant's reply
24
print(response.choices[0].message.content)

Multi-Step Traces

If you have multiple steps in your LLM pipeline, you can use the @track decorator to log the traces for each step. If TrueFoundry is called within one of these steps, the LLM call will be associated with that corresponding step:

1
import os
2
from opik import track
3
from opik.integrations.openai import track_openai
4
from openai import OpenAI
5
6
# Create and wrap the OpenAI client with TrueFoundry's base URL
7
client = OpenAI(
8
    api_key=os.environ["TRUEFOUNDRY_API_KEY"],
9
    base_url=os.environ["TRUEFOUNDRY_BASE_URL"]
10
)
11
client = track_openai(client)
12
13
@track
14
def generate_response(prompt: str):
15
    response = client.chat.completions.create(
16
        model="openai-main/gpt-4o",
17
        messages=[
18
            {"role": "system", "content": "You are a knowledgeable AI assistant."},
19
            {"role": "user", "content": prompt}
20
        ]
21
    )
22
    return response.choices[0].message.content
23
24
@track
25
def refine_response(initial_response: str):
26
    response = client.chat.completions.create(
27
        model="openai-main/gpt-4o",
28
        messages=[
29
            {"role": "system", "content": "You enhance and polish text responses."},
30
            {"role": "user", "content": f"Please improve this response: {initial_response}"}
31
        ]
32
    )
33
    return response.choices[0].message.content
34
35
@track(project_name="truefoundry-integration-demo")
36
def generate_and_refine(prompt: str):
37
    # First LLM call: Generate initial response
38
    initial = generate_response(prompt)
39
40
    # Second LLM call: Refine the response
41
    refined = refine_response(initial)
42
43
    return refined
44
45
# Example usage
46
result = generate_and_refine("Explain quantum computing in simple terms.")

The trace will show nested LLM calls with hierarchical spans.

TrueFoundry Documentation

• Comet Opik integration — the TrueFoundry side of the OpenTelemetry export, with a screenshot of each field.
• AI Gateway quick start — create an account, add models, and get your base URL, API key, and model ID.
• Chat Completions API — the unified request format for all supported models.
• Authentication — Personal Access Tokens and Virtual Account Tokens.

Next steps

• Log traces — add tags, metadata, and feedback scores to the traces from your gateway.
• Evaluate your LLM application — build datasets from gateway traffic and score it.
• Gateway integrations — the other LLM gateways that Opik supports.

Further Improvements

If you have suggestions for improving the TrueFoundry integration, please let us know by opening an issue on GitHub.