Typescript SDKEvaluation

Models

View as Markdown

In Opik 2.0, experiments are project-scoped. When using models in evaluations, specify a projectName in the evaluate() call so results are associated with the correct project.

The TypeScript SDK provides flexible model configuration through direct integration with the Vercel AI SDK. You can use models from multiple providers with a simple, unified interface.

Overview

The TypeScript SDK supports three ways to configure models for evaluation and prompt generation:

  1. Model ID strings - Simple string identifiers (e.g., "gpt-5-nano", "claude-3-5-sonnet-latest")
  2. LanguageModel instances - Pre-configured Vercel AI SDK models with custom settings
  3. OpikBaseModel implementations - Custom model integrations for unsupported providers

Quick Start

Using Model ID Strings

The simplest approach is to pass a model ID string directly:

import { evaluatePrompt } from "opik";
import { Hallucination } from "opik";
// OpenAI model
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gpt-5-nano",
  projectName: "my-project",
});
// Anthropic model
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "claude-3-5-sonnet-latest",
  projectName: "my-project",
});
// Google Gemini model
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gemini-2.0-flash",
  projectName: "my-project",
});
// Use in metrics
const metric = new Hallucination({ model: "gpt-5-nano" });

Using LanguageModel Instances

For advanced scenarios, use LanguageModel instances from Vercel AI SDK:

import { openai } from "@ai-sdk/openai";
import { evaluatePrompt } from "opik";
// Create a LanguageModel instance
const customModel = openai("gpt-5-nano");
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: customModel,
  projectName: "my-project",
});

Generation Parameters

Parameters for Metrics

All LLM Judge metrics support these generation parameters directly in the constructor:

import { Hallucination } from "opik";
const metric = new Hallucination({
  model: "gpt-5-nano",
  temperature: 0.3, // Control randomness (0.0-2.0)
  seed: 42, // For reproducible outputs
  maxTokens: 1000, // Maximum response length
});
// Use in evaluation
const score = await metric.score({
  input: "What is the capital of France?",
  output: "The capital of France is Paris.",
  context: ["France is a country in Western Europe."],
});

For advanced generation parameters, use modelSettings:

import { Hallucination } from "opik";
const metric = new Hallucination({
  model: "gpt-5-nano",
  temperature: 0.5,
  modelSettings: {
    topP: 0.9, // Nucleus sampling
    topK: 50, // Top-K sampling
    presencePenalty: 0.1, // Reduce repetition
    frequencyPenalty: 0.2, // Reduce phrase repetition
    stopSequences: ["END"], // Custom stop sequences
  },
});

Parameters for evaluatePrompt

The evaluatePrompt function supports only temperature and seed:

import { evaluatePrompt } from "opik";
import { Hallucination, AnswerRelevance } from "opik";
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gpt-5-nano",
  temperature: 0.7,
  seed: 42,
  scoringMetrics: [
    new Hallucination({
      model: "gpt-5-nano",
      temperature: 0.3, // Full parameter support in metrics
      seed: 12345,
      maxTokens: 1000,
    }),
  ],
  projectName: "my-project",
});

Note: For full control over all Vercel AI SDK parameters, create a LanguageModel instance with your desired configuration and pass it to the model parameter. See Using LanguageModel Instances below.

Supported Providers

OpenAI

OpenAI models are supported through the @ai-sdk/openai package.

Example model IDs:

"gpt-5-nano";
"gpt-5-mini";
"gpt-5";

Usage:

import { evaluatePrompt } from "opik";
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gpt-5-nano",
  projectName: "my-project",
});

For a complete list of available models, see the Vercel AI SDK OpenAI provider documentation.

Anthropic

Anthropic’s Claude models are supported through the @ai-sdk/anthropic package.

Example model IDs:

"claude-3-5-sonnet-latest";
"claude-3-5-haiku-latest";

Usage:

await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "claude-3-5-sonnet-latest",
  projectName: "my-project",
});

For a complete list of available models, see the Vercel AI SDK Anthropic provider documentation.

Google Gemini

Google’s Gemini models are supported through the @ai-sdk/google package.

Example model IDs:

"gemini-2.0-flash";
"gemini-1.5-pro";

Usage:

await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gemini-2.0-flash",
  projectName: "my-project",
});

For a complete list of available models, see the Vercel AI SDK Google provider documentation.

Using Models in Opik

Using LanguageModel Instances

For advanced scenarios requiring full Vercel AI SDK features (such as structured outputs, custom headers, or provider-specific parameters), create LanguageModel instances directly:

import { openai } from "@ai-sdk/openai";
import { anthropic } from "@ai-sdk/anthropic";
import { evaluatePrompt } from "opik";
import { Hallucination } from "opik";
// Create models with advanced configuration
const genModel = openai("gpt-5-nano", {
  structuredOutputs: true, // Provider-specific feature
});
const evalModel = anthropic("claude-3-5-sonnet-latest");
// Use different models for generation and evaluation
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: genModel,
  scoringMetrics: [new Hallucination({ model: evalModel })],
  projectName: "my-project",
});

This approach gives you full control over Vercel AI SDK parameters that aren’t exposed through Opik’s simple interface.

Using Models with Metrics

LLM Judge metrics accept model configuration:

With Model ID String

import { Hallucination, AnswerRelevance } from "opik";
// Use different models for different metrics
const hallucinationMetric = new Hallucination({ model: "gpt-5-nano" });
const relevanceMetric = new AnswerRelevance({
  model: "claude-3-5-sonnet-latest",
});
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gpt-5-nano",
  scoringMetrics: [hallucinationMetric, relevanceMetric],
  projectName: "my-project",
});

With LanguageModel Instance

import { openai } from "@ai-sdk/openai";
import { Hallucination } from "opik";
// Create model for metric evaluation
const judgeModel = openai("gpt-5-nano");
const metric = new Hallucination({ model: judgeModel });
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gpt-5-nano",
  scoringMetrics: [metric],
  projectName: "my-project",
});

Custom Model Implementation

For unsupported providers, implement the OpikBaseModel interface:

OpikBaseModel Interface

abstract class OpikBaseModel {
  constructor(public readonly modelName: string) {}
  /**
   * Generate a string response from a text prompt
   */
  abstract generateString(input: string): Promise<string>;
  /**
   * Generate a response from messages with provider-specific format
   */
  abstract generateProviderResponse(messages: OpikMessage[]): Promise<unknown>;
}

Example Implementation

import { OpikBaseModel, OpikMessage } from "opik";
class CustomProviderModel extends OpikBaseModel {
  private apiKey: string;
  private baseUrl: string;
  constructor(modelName: string, apiKey: string, baseUrl: string) {
    super(modelName);
    this.apiKey = apiKey;
    this.baseUrl = baseUrl;
  }
  async generateString(input: string): Promise<string> {
    const messages: OpikMessage[] = [
      {
        role: "user",
        content: input,
      },
    ];
    const response = await this.generateProviderResponse(messages);
    // Extract text from provider response format
    return response.choices[0].message.content;
  }
  async generateProviderResponse(messages: OpikMessage[]): Promise<unknown> {
    const response = await fetch(`${this.baseUrl}/chat/completions`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${this.apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: this.modelName,
        messages: messages,
      }),
    });
    if (!response.ok) {
      throw new Error(`API request failed: ${response.statusText}`);
    }
    return response.json();
  }
}
// Usage
const customModel = new CustomProviderModel(
  "custom-model-v1",
  process.env.CUSTOM_API_KEY,
  "https://api.custom-provider.com"
);
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: customModel,
  projectName: "my-project",
});

Model Resolution

The SDK automatically resolves models in this order:

  1. If a string is provided: Auto-detects provider and creates appropriate model
  2. If LanguageModel is provided: Uses the instance directly
  3. If OpikBaseModel is provided: Uses the custom implementation
  4. If undefined: Defaults to "gpt-5-nano"
// String → Auto-detected as OpenAI
model: "gpt-5-nano";
// LanguageModel → Used directly
import { openai } from "@ai-sdk/openai";
model: openai("gpt-5-nano");
// Custom implementation
model: new CustomProviderModel("my-model", apiKey, baseUrl);
// Undefined → Defaults to "gpt-5-nano"
model: undefined;

Best Practices

1. Use Model ID Strings for Simplicity

For most use cases, use model ID strings directly:

import { Hallucination } from "opik";
const metric = new Hallucination({ model: "gpt-5-nano" });

The Opik SDK handles model configuration internally for optimal evaluation performance.

2. Match Model Capabilities to Task

Choose models based on task requirements:

// Complex reasoning: GPT-5, Claude Sonnet
model: "gpt-5";
model: "claude-3-5-sonnet-latest";
// Fast responses: GPT-5-nano, Gemini Flash
model: "gpt-5-nano";
model: "gemini-2.0-flash";
// Long context: Claude, Gemini
model: "claude-3-5-sonnet-latest"; // 200K context
model: "gemini-1.5-pro"; // 1M context

3. Use Different Models for Tasks and Metrics

Optimize costs by using different models:

await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gpt-5-nano", // Cheaper for generation
  scoringMetrics: [
    new Hallucination({ model: "gpt-5-mini" }), // More accurate for evaluation
  ],
  projectName: "my-project",
});

4. Configure API Keys

Set up environment variables for each provider:

# OpenAI
export OPENAI_API_KEY="sk-..."
# Anthropic
export ANTHROPIC_API_KEY="sk-ant-..."
# Google
export GOOGLE_API_KEY="..."

5. Handle Rate Limits

Use appropriate worker counts to avoid rate limits:

await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gpt-5-nano",
  taskWorkers: 5, // Limit parallel requests
  projectName: "my-project",
});

Troubleshooting

API Key Not Found

// Error: API key not found for provider
// Solution: Set environment variable
process.env.OPENAI_API_KEY = "sk-...";

Model Not Supported

// Error: Unsupported model ID
// Solution: Use custom implementation
class MyModel extends OpikBaseModel {
  // ... implementation
}

Rate Limit Errors

// Error: Rate limit exceeded
// Solution: Reduce worker count
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  model: "gpt-5-nano",
  taskWorkers: 3, // Reduce from default 10
  projectName: "my-project",
});

See Also