Skip to navigation
PythonModel Providers

Observability for Cohere with Opik

View as Markdown

Cohere provides state-of-the-art large language models that excel at text generation, summarization, classification, and retrieval-augmented generation.

Wrap your Cohere client with track_cohere and every chat() and chat_stream() call is logged to Opik with its messages, response, model and token usage. The async client works out of the box.

Cohere’s native API is not OpenAI-shaped — chat() takes keyword-only arguments and streaming is a separate chat_stream() method returning a union of events — so if your code uses the Cohere SDK, use track_cohere rather than track_openai against the compatibility endpoint.

Account Setup

Comet provides a hosted version of the Opik platform, simply create an account and grab your API Key.

You can also run the Opik platform locally, see the installation guide for more information.

Getting Started

Installation

pip install opik cohere

Configuring Opik

Run opik configure and follow the prompts. See the Python SDK Configuration guide for other options, such as configuring Opik in code or with environment variables.

Configuring Cohere

Set your Cohere API key as an environment variable:

export CO_API_KEY="YOUR_API_KEY"

Logging LLM calls

Create a Cohere client, wrap it with track_cohere, and use it as usual:

import cohere
from opik.integrations.cohere import track_cohere
client = track_cohere(cohere.ClientV2())
response = client.chat(
    model="command-r-plus",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(response.message.content[0].text)

Advanced Usage

Using with the @track decorator

Calls made with a tracked client inside a function decorated with @track are nested under that function’s trace:

import cohere
from opik import track
from opik.integrations.cohere import track_cohere
client = track_cohere(cohere.ClientV2())
@track
def generate_story(prompt: str) -> str:
    response = client.chat(
        model="command-r-plus",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.message.content[0].text
generate_story("Write a short story about Opik.")

Streaming

Cohere streams through a separate chat_stream() method. The span is ended once the stream is fully consumed, and the events are aggregated into a single logged response:

stream = client.chat_stream(
    model="command-r-plus",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
for event in stream:
    if event.type == "content-delta":
        print(event.delta.message.content.text, end="")

Async client

track_cohere works with cohere.AsyncClientV2 as well, for both chat() and chat_stream():

import asyncio
import cohere
from opik.integrations.cohere import track_cohere
async_client = track_cohere(cohere.AsyncClientV2())
async def main():
    response = await async_client.chat(
        model="command-r-plus",
        messages=[{"role": "user", "content": "Why is the sky blue?"}],
    )
    print(response.message.content[0].text)
asyncio.run(main())

Cost Tracking

Opik records the token usage Cohere returns on each call. Cohere reports tokens.input_tokens and tokens.output_tokens, which the integration maps onto Opik’s prompt_tokens / completion_tokens names.

Supported Methods

track_cohere logs chat() and chat_stream() on both cohere.ClientV2 and cohere.AsyncClientV2. Other methods, such as embed() and rerank(), aren’t logged; to trace them, call them inside a function decorated with @track.

Using Cohere through the OpenAI compatibility API

If your code calls Cohere through the OpenAI SDK against Cohere’s compatibility endpoint, keep that code and use track_openai as described below.

Getting started with the compatibility API

Configuring Opik

To start tracking your Cohere LLM calls, you’ll need to have both opik and openai packages installed. You can install them using pip:

pip install opik openai

In addition, you can configure Opik using the opik configure command which will prompt you for the correct local server address or if you are using the Cloud platform your API key:

opik configure

Configuring Cohere

You’ll need to set your Cohere API key as an environment variable:

export COHERE_API_KEY="YOUR_API_KEY"

Tracking Cohere API calls through the compatibility API

Leverage the OpenAI Compatibility API by replacing the base URL with Cohere’s endpoint when initializing the client:

import os
from opik.integrations.openai import track_openai
from openai import OpenAI
client = OpenAI(
    api_key=os.environ.get("COHERE_API_KEY"),
    base_url="https://api.cohere.ai/compatibility/v1"  # Cohere Compatibility API endpoint
)
client = track_openai(client)
response = client.chat.completions.create(
    model="command-r7b-12-2024",  # Replace with the desired Cohere model
    messages=[
        {"role": "system", "content": "You are an assistant."},
        {"role": "user", "content": "Why is tracking and evaluation of LLMs important?"}
    ],
    temperature=0.7,
    max_tokens=100
)
print(response.choices[0].message.content)

The track_openai will automatically track and log the API call, including the input prompt, model used, and response generated. You can view these logs in your Opik project dashboard.

Using the compatibility API within a tracked function

If you are using Cohere within a function tracked with the @track decorator, you can use the tracked client as normal:

from opik import track
from opik.integrations.openai import track_openai
from openai import OpenAI
import os
client = OpenAI(
    api_key=os.environ.get("COHERE_API_KEY"),
    base_url="https://api.cohere.ai/compatibility/v1"
)
tracked_client = track_openai(client)
@track
def generate_story(prompt):
    response = tracked_client.chat.completions.create(
        model="command-r7b-12-2024",
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
    return response.choices[0].message.content
@track
def generate_topic():
    prompt = "Generate a topic for a story about Opik."
    response = tracked_client.chat.completions.create(
        model="command-r7b-12-2024",
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
    return response.choices[0].message.content
@track
def generate_opik_story():
    topic = generate_topic()
    story = generate_story(topic)
    return story
generate_opik_story()

Supported Cohere models

The track_openai wrapper with Cohere’s compatibility API supports the following Cohere models:

  • command-r7b-12-2024 - Command R 7B model
  • command-r-plus - Command R Plus model
  • command-r - Command R model
  • command-light - Command Light model
  • command - Command model

Supported OpenAI methods

The track_openai wrapper supports the following OpenAI methods when used with Cohere:

  • client.chat.completions.create(), including support for stream=True mode
  • client.beta.chat.completions.parse()
  • client.beta.chat.completions.stream()
  • client.responses.create()

If you would like to track another OpenAI method, please let us know by opening an issue on GitHub.