ObservabilityIntegrations

Together AI

Together AI provides fast inference for leading open-source models including Llama, Mistral, Qwen, and many others.

This guide explains how to integrate Opik with Together AI via LiteLLM. By using the LiteLLM integration provided by Opik, you can easily track and evaluate your Together AI calls within your Opik projects as Opik will automatically log the input prompt, model used, token usage, and response generated.

Getting Started

Configuring Opik

To start tracking your Together AI calls, you’ll need to have both opik and litellm installed. You can install them using pip:

$pip install opik litellm

In addition, you can configure Opik using the opik configure command which will prompt you for the correct local server address or if you are using the Cloud platform your API key:

$opik configure

Configuring Together AI

You’ll need to set your Together AI API key as an environment variable:

{pytest_codeblocks_skip=true}
$export TOGETHER_API_KEY="YOUR_API_KEY"

Logging LLM calls

In order to log the LLM calls to Opik, you will need to create the OpikLogger callback. Once the OpikLogger callback is created and added to LiteLLM, you can make calls to LiteLLM as you normally would:

1from litellm.integrations.opik.opik import OpikLogger
2import litellm
3
4opik_logger = OpikLogger()
5litellm.callbacks = [opik_logger]
6
7response = litellm.completion(
8    model="together_ai/meta-llama/Llama-3.2-3B-Instruct-Turbo",
9    messages=[
10        {"role": "user", "content": "Why is tracking and evaluation of LLMs important?"}
11    ]
12)

Logging LLM calls within a tracked function

If you are using LiteLLM within a function tracked with the @track decorator, you will need to pass the current_span_data as metadata to the litellm.completion call:

1from opik import track, opik_context
2import litellm
3
4@track
5def generate_story(prompt):
6    response = litellm.completion(
7        model="together_ai/meta-llama/Llama-3.2-3B-Instruct-Turbo",
8        messages=[{"role": "user", "content": prompt}],
9        metadata={
10            "opik": {
11                "current_span_data": opik_context.get_current_span_data(),
12            },
13        },
14    )
15    return response.choices[0].message.content
16
17
18@track
19def generate_topic():
20    prompt = "Generate a topic for a story about Opik."
21    response = litellm.completion(
22        model="together_ai/meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo",
23        messages=[{"role": "user", "content": prompt}],
24        metadata={
25            "opik": {
26                "current_span_data": opik_context.get_current_span_data(),
27            },
28        },
29    )
30    return response.choices[0].message.content
31
32
33@track
34def generate_opik_story():
35    topic = generate_topic()
36    story = generate_story(topic)
37    return story
38
39
40generate_opik_story()