Observability for Together AI with Opik

Together AI provides fast inference for leading open-source models including Llama, Mistral, Qwen, and many others.

Wrap your Together client with track_together and every chat completion is logged to Opik with its messages, response, model and token usage. Streaming and the async client work out of the box.

Together’s SDK is OpenAI-shaped but defines its own client and stream classes, so track_openai cannot patch it — use track_together instead.

Account Setup

Comet provides a hosted version of the Opik platform, simply create an account and grab your API Key.

You can also run the Opik platform locally, see the installation guide for more information.

Getting Started

Installation

pip install opik together

Configuring Opik

Run opik configure and follow the prompts. See the Python SDK Configuration guide for other options, such as configuring Opik in code or with environment variables.

Configuring Together AI

Set your Together AI API key as an environment variable:

export TOGETHER_API_KEY="YOUR_API_KEY"

Logging LLM calls

Create a Together client, wrap it with track_together, and use it as usual:

import together
from opik.integrations.together import track_together
client = track_together(together.Together())
response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(response.choices[0].message.content)

Advanced Usage

Using with the @track decorator

Calls made with a tracked client inside a function decorated with @track are nested under that function’s trace:

import together
from opik import track
from opik.integrations.together import track_together
client = track_together(together.Together())
@track
def generate_story(prompt: str) -> str:
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content
generate_story("Write a short story about Opik.")

Streaming

Pass stream=True and iterate as usual. The span is ended once the stream is fully consumed, and the chunks are aggregated into a single logged response:

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Async client

track_together works with together.AsyncTogether as well, including streaming:

import asyncio
import together
from opik.integrations.together import track_together
async_client = track_together(together.AsyncTogether())
async def main():
    response = await async_client.chat.completions.create(
        model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
        messages=[{"role": "user", "content": "Why is the sky blue?"}],
    )
    print(response.choices[0].message.content)
asyncio.run(main())

Cost Tracking

Opik records the token usage Together returns on each call. Cost is not estimated for Together spans yet, so they show token counts without a price.

Supported Methods

track_together logs chat.completions.create() on both together.Together and together.AsyncTogether, with and without streaming. Other methods aren’t logged; to trace them, call them inside a function decorated with @track.

Using Together AI through LiteLLM

If you call Together AI through LiteLLM rather than the Together SDK, use the LiteLLM integration instead of track_together.

Configuring Opik and LiteLLM

Configuring Opik

To start tracking your Together AI calls, you’ll need to have both opik and litellm installed. You can install them using pip:

pip install opik litellm

In addition, you can configure Opik using the opik configure command which will prompt you for the correct local server address or if you are using the Cloud platform your API key:

opik configure

Configuring Together AI

You’ll need to set your Together AI API key as an environment variable:

export TOGETHER_API_KEY="YOUR_API_KEY"

Logging LLM calls through LiteLLM

In order to log the LLM calls to Opik, you will need to create the OpikLogger callback. Once the OpikLogger callback is created and added to LiteLLM, you can make calls to LiteLLM as you normally would:

from litellm.integrations.opik.opik import OpikLogger
import litellm
opik_logger = OpikLogger()
litellm.callbacks = [opik_logger]
response = litellm.completion(
    model="together_ai/meta-llama/Llama-3.2-3B-Instruct-Turbo",
    messages=[
        {"role": "user", "content": "Why is tracking and evaluation of LLMs important?"}
    ]
)

Logging LiteLLM calls within a tracked function

If you are using LiteLLM within a function tracked with the @track decorator, you will need to pass the current_span_data as metadata to the litellm.completion call:

from opik import track, opik_context
import litellm
@track
def generate_story(prompt):
    response = litellm.completion(
        model="together_ai/meta-llama/Llama-3.2-3B-Instruct-Turbo",
        messages=[{"role": "user", "content": prompt}],
        metadata={
            "opik": {
                "current_span_data": opik_context.get_current_span_data(),
            },
        },
    )
    return response.choices[0].message.content
@track
def generate_topic():
    prompt = "Generate a topic for a story about Opik."
    response = litellm.completion(
        model="together_ai/meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo",
        messages=[{"role": "user", "content": prompt}],
        metadata={
            "opik": {
                "current_span_data": opik_context.get_current_span_data(),
            },
        },
    )
    return response.choices[0].message.content
@track
def generate_opik_story():
    topic = generate_topic()
    story = generate_story(topic)
    return story
generate_opik_story()