Custom model

Opik provides a set of LLM as a Judge metrics that are designed to be model-agnostic and can be used with any LLM. In order to achieve this, we use the LiteLLM library to abstract the LLM calls.

By default, Opik will use the gpt-5-nano model. However, you can change this by setting the model parameter when initializing your metric to any model supported by LiteLLM:

1
from opik.evaluation.metrics import Hallucination
2
3
hallucination_metric = Hallucination(
4
    model="gpt-4o-mini"
5
)

Using a model supported by LiteLLM

In order to use many models supported by LiteLLM, you also need to pass additional parameters. For this, you can use the LiteLLMChatModel class and passing it to the metric:

1
from opik.evaluation.metrics import Hallucination
2
from opik.evaluation import models
3
4
model = models.LiteLLMChatModel(
5
    model_name="<model_name>"
6
)
7
8
hallucination_metric = Hallucination(
9
    model=model
10
)

Using OpenAI-compatible providers

Many LLM providers (such as SiliconFlow, Together AI, Groq, and others) expose APIs that are compatible with the OpenAI API format. You can use these providers with Opik’s LLM-as-a-Judge metrics by using LiteLLM’s openai/ provider prefix and setting the appropriate environment variables.

This is a simpler alternative to creating a custom model class when your provider already supports the OpenAI API format.

Set OPENAI_API_KEY to your provider’s API key and OPENAI_BASE_URL to the provider’s API endpoint, then use the openai/ prefix when specifying the model name:

1
import os
2
from opik.evaluation.metrics import Hallucination
3
4
# Configure the OpenAI-compatible provider
5
os.environ["OPENAI_API_KEY"] = "your-provider-api-key"
6
os.environ["OPENAI_BASE_URL"] = "https://api.your-provider.com/v1"
7
8
# Use the openai/ prefix with the provider's model name
9
hallucination_metric = Hallucination(
10
    model="openai/your-model-name"
11
)
12
13
score = hallucination_metric.score(
14
    input="What is the capital of France?",
15
    output="The capital of France is Paris, a city known for its iconic Eiffel Tower.",
16
    context=["Paris is the capital and most populous city of France."]
17
)
18
print(f"Hallucination score: {score.value}")

The openai/ prefix tells LiteLLM to use the OpenAI-compatible API format with the configured base URL. This approach works with any metric that accepts a model parameter, including Hallucination, Moderation, AnswerRelevance, and others.

For the full list of supported providers and configuration options, see the LiteLLM OpenAI-compatible providers documentation.

Creating Your Own Custom Model Class

Opik’s LLM-as-a-Judge metrics, such as Hallucination, are designed to work with various language models. While Opik supports many models out-of-the-box via LiteLLM, you can integrate any LLM by creating a custom model class. This involves subclassing opik.evaluation.models.OpikBaseModel and implementing its required methods.

The OpikBaseModel Interface

OpikBaseModel is an abstract base class that defines the interface Opik metrics use to interact with LLMs. To create a compatible custom model, you must implement the following methods:

1. __init__(self, model_name: str): Initializes the base model with a given model name.
2. generate_string(self, input: str, **kwargs: Any) -> str: Simplified interface to generate a string output from the model.
3. generate_provider_response(self, **kwargs: Any) -> Any: Generate a provider-specific response. Can be used to interface with the underlying model provider (e.g., OpenAI, Anthropic) and get raw output.

Implementing a Custom Model for an OpenAI-like API

Here’s an example of a custom model class that interacts with an LLM service exposing an OpenAI-compatible API endpoint.

1
import requests
2
from typing import Any
3
4
from opik.evaluation.models import OpikBaseModel
5
6
class CustomOpenAICompatibleModel(OpikBaseModel):
7
    def __init__(self, model_name: str, api_key: str, base_url: str):
8
        super().__init__(model_name)
9
        self.api_key = api_key
10
        self.base_url = base_url # e.g., "https://api.openai.com/v1/chat/completions"
11
        self.headers = {
12
            "Authorization": f"Bearer {self.api_key}",
13
            "Content-Type": "application/json"
14
        }
15
16
    def generate_string(self, input: str, **kwargs: Any) -> str:
17
        """
18
        This method is used as part of LLM as a Judge metrics to take a string prompt, pass it to
19
        the model as a user message and return the model's response as a string.
20
        """
21
        conversation = [
22
            {
23
                "content": input,
24
                "role": "user",
25
            },
26
        ]
27
28
        provider_response = self.generate_provider_response(messages=conversation, **kwargs)
29
        return provider_response["choices"][0]["message"]["content"]
30
31
    def generate_provider_response(self, messages: list[dict[str, Any]], **kwargs: Any) -> Any:
32
        """
33
        This method is used as part of LLM as a Judge metrics to take a list of AI messages, pass it to
34
        the model and return the full model response.
35
        """
36
        payload = {
37
            "model": self.model_name,
38
            "messages": messages,
39
        }
40
41
        response = requests.post(self.base_url, headers=self.headers, json=payload)
42
43
        response.raise_for_status()
44
        return response.json()

Key considerations for the implementation:

• API Endpoint and Payload: Adjust base_url and the JSON payload to match your specific LLM provider’s requirements if they deviate from the common OpenAI structure.
• Model Name: The model_name passed to __init__ is used as the model parameter in the API call. Ensure this matches an available model on your LLM service.

Using the Custom Model with the Hallucination Metric

In order to run an evaluation using your Custom Model with the Hallucination metric, you will first need to instantiate our CustomOpenAICompatibleModel class and pass it to the Hallucination class. The evaluation can then be kicked off by calling the Hallucination.score()` method.

1
from opik.evaluation.metrics import Hallucination
2
3
# Ensure these are set securely, e.g., via environment variables
4
API_KEY = os.getenv("MY_CUSTOM_LLM_API_KEY")
5
BASE_URL = "YOUR_LLM_CHAT_COMPLETIONS_ENDPOINT" # e.g., "https://api.openai.com/v1/chat/completions"
6
MODEL_NAME = "your-model-name" # e.g., "gpt-3.5-turbo"
7
8
# Initialize your custom model
9
my_custom_model = CustomOpenAICompatibleModel(
10
    model_name=MODEL_NAME,
11
    api_key=API_KEY,
12
    base_url=BASE_URL
13
)
14
15
# Initialize the Hallucination metric with the custom model
16
hallucination_metric = Hallucination(
17
    model=my_custom_model
18
)
19
20
# Example usage:
21
evaluation = hallucination_metric.score(
22
    input="What is the capital of Mars?",
23
    output="The capital of Mars is Ares City, a bustling metropolis.",
24
    context=["Mars is a planet in our solar system. It does not currently have any established cities or a designated capital."]
25
)
26
print(f"Hallucination Score: {evaluation.value}") # Expected: 1.0 (hallucination detected)
27
print(f"Reason: {evaluation.reason}")

Key considerations for the implementation:

• ScoreResult Output: Hallucination.score() returns a ScoreResult object containing the metric name (name), score value (value), optional explanation (reason), metadata (metadata), and a failure flag (scoring_failed).

TypeScript: Using Vercel AI SDK Models

The TypeScript SDK integrates seamlessly with the Vercel AI SDK, allowing you to use language models directly with Opik’s evaluation metrics. For comprehensive model configuration including supported providers, generation parameters, and advanced settings, see the Models Reference.

Creating Custom Models with OpikBaseModel

For unsupported LLM providers, implement the OpikBaseModel interface:

1
import { OpikBaseModel, OpikMessage } from "opik/evaluation/models";
2
3
class CustomProviderModel extends OpikBaseModel {
4
  private apiKey: string;
5
  private baseUrl: string;
6
7
  constructor(modelName: string, apiKey: string, baseUrl: string) {
8
    super(modelName);
9
    this.apiKey = apiKey;
10
    this.baseUrl = baseUrl;
11
  }
12
13
  async generateString(input: string): Promise<string> {
14
    // Convert string input to message format
15
    const messages: OpikMessage[] = [
16
      {
17
        role: "user",
18
        content: input,
19
      },
20
    ];
21
22
    // Call provider API
23
    const response = await this.generateProviderResponse(messages);
24
25
    // Extract text from response
26
    return response.choices[0].message.content;
27
  }
28
29
  async generateProviderResponse(messages: OpikMessage[]): Promise<unknown> {
30
    // Make API call to your custom provider
31
    const response = await fetch(`${this.baseUrl}/chat/completions`, {
32
      method: "POST",
33
      headers: {
34
        Authorization: `Bearer ${this.apiKey}`,
35
        "Content-Type": "application/json",
36
      },
37
      body: JSON.stringify({
38
        model: this.modelName,
39
        messages: messages,
40
      }),
41
    });
42
43
    if (!response.ok) {
44
      throw new Error(`API request failed: ${response.statusText}`);
45
    }
46
47
    return response.json();
48
  }
49
}

Using Custom Models

Once implemented, use your custom model like any other:

1
import { Hallucination } from "opik";
2
import { evaluatePrompt } from "opik";
3
4
// Initialize custom model
5
const customModel = new CustomProviderModel(
6
  "custom-model-v1",
7
  process.env.CUSTOM_API_KEY!,
8
  "https://api.custom-provider.com"
9
);
10
11
// Use with metrics
12
const metric = new Hallucination({ model: customModel });
13
14
const score = await metric.score({
15
  input: "What is the capital of Mars?",
16
  output: "The capital of Mars is Ares City, a bustling metropolis.",
17
  context: [
18
    "Mars is a planet in our solar system. It does not currently have any established cities or a designated capital.",
19
  ],
20
});
21
22
console.log(`Hallucination Score: ${score.value}`); // Expected: 1.0 (hallucination detected)
23
console.log(`Reason: ${score.reason}`);
24
25
// Use with evaluatePrompt
26
await evaluatePrompt({
27
  dataset,
28
  messages: [{ role: "user", content: "{{input}}" }],
29
  model: customModel,
30
  scoringMetrics: [metric],
31
});

Best Practices

When implementing custom models:

1. Implement both required methods: Ensure your custom model implements both generateString() and generateProviderResponse() methods
2. Handle errors gracefully: Wrap API calls in try-catch blocks and provide meaningful error messages
3. Configure API keys securely: Store API keys in environment variables, never hardcode them

For standard model usage and configuration, refer to the Models Reference.