Evaluate Prompt Function | Opik Documentation

In Opik 2.0, experiments are project-scoped. Make sure to specify a projectName when calling evaluatePrompt() so results are associated with the correct project.

The evaluatePrompt function provides a streamlined way to evaluate prompt templates against a dataset. It automatically formats message templates with dataset variables, generates LLM responses, and evaluates the results using specified metrics.

Overview

evaluatePrompt is a convenience wrapper around the evaluate function that handles:

Template formatting: Automatically formats message templates with dataset item variables
Model invocation: Generates LLM responses using your specified model
Experiment tracking: Creates experiments linked to specific prompt versions
Metric evaluation: Scores outputs using the specified metrics

This is particularly useful for prompt engineering workflows where you want to quickly test different prompt templates against a dataset.

Function Signature

function evaluatePrompt(
  options: EvaluatePromptOptions
): Promise<EvaluationResult>;

EvaluatePromptOptions

interface EvaluatePromptOptions extends Omit<EvaluateOptions, "task"> {
  // Required parameters
  dataset: Dataset;
  messages: OpikMessage[];
  // Optional parameters
  model?: SupportedModelId | LanguageModel | OpikBaseModel;
  templateType?: "mustache" | "jinja2";
  scoringMetrics?: BaseMetric[];
  experimentName?: string;
  experimentConfig?: Record<string, unknown>;
  prompts?: Prompt[];
  projectName?: string;
  nbSamples?: number;
  scoringKeyMapping?: Record<string, string>;
}

Parameters

Required Parameters

dataset

Type: Dataset
Description: The dataset to evaluate prompts against. Each dataset item will be used to format the message templates and generate responses.

const dataset = await client.getOrCreateDataset("my-dataset", "Evaluation dataset", "my-project");

messages

Type: OpikMessage[]
Description: Array of message templates with {{placeholders}} that will be formatted with dataset variables.

messages: [
  { role: "system", content: "You are a helpful assistant" },
  { role: "user", content: "Translate to {{language}}: {{text}}" },
];

Optional Parameters

model

Type: SupportedModelId | LanguageModel | OpikBaseModel
Default: "gpt-5-nano"
Description: The language model to use for generation. Can be:
- Model ID string (e.g., "gpt-5-nano", "claude-3-5-sonnet-latest", "gemini-2.0-flash")
- Pre-configured LanguageModel instance from Vercel AI SDK
- Custom OpikBaseModel implementation

// Using model ID string
model: "gpt-5-nano";
// Using LanguageModel instance
import { openai } from "@ai-sdk/openai";
const customModel = openai("gpt-5-nano");
model: customModel;

templateType

Type: "mustache" | "jinja2"
Default: "mustache"
Description: Template engine to use for variable substitution in message content.

// Mustache syntax (default)
templateType: "mustache";
messages: [{ role: "user", content: "Hello {{name}}" }];
// Jinja2 syntax
templateType: "jinja2";
messages: [{ role: "user", content: "Hello {{ name }}" }];

scoringMetrics

Type: BaseMetric[]
Description: Array of metrics to evaluate the generated outputs. Can include both heuristic and LLM Judge metrics.

import { ExactMatch, Hallucination } from "opik";
scoringMetrics: [new ExactMatch(), new Hallucination()];

experimentName

Type: string
Description: Name for the experiment. If not provided, a name will be auto-generated.

experimentName: "Prompt Evaluation - Translation Task";

experimentConfig

Type: Record<string, unknown>
Description: Additional metadata to store with the experiment. The function automatically adds prompt_template and model to this configuration.

experimentConfig: {
  temperature: 0.7,
  max_tokens: 1000,
  version: "v2",
};

prompts

Type: Prompt[]
Description: Array of Opik Prompt objects to link to this experiment. Useful for tracking which prompt versions were used.

const prompt = await client.createPrompt({
  name: "translation-prompt",
  prompt: "Translate to {{language}}: {{text}}",
});
prompts: [prompt];

projectName

Type: string
Description: Name of the Opik project to log traces to.

projectName: "prompt-engineering";

nbSamples

Type: number
Description: Maximum number of dataset items to evaluate. Useful for quick testing.

nbSamples: 10; // Only evaluate first 10 items

scoringKeyMapping

Type: Record<string, string>
Description: Maps metric parameter names to dataset/output field names when they don’t match.

scoringKeyMapping: {
  input: "question", // Map 'input' param to 'question' field
  expected: "reference_answer", // Map 'expected' param to 'reference_answer' field
};

Return Value

Returns a Promise<EvaluationResult> containing:

interface EvaluationResult {
  experimentId: string; // ID of created experiment
  experimentName: string; // Name of experiment
  testResults: TestResult[]; // Results for each dataset item
}

Examples

Basic Usage

Simple prompt evaluation with default settings:

import { Opik, evaluatePrompt } from "opik";
const client = new Opik();
const dataset = await client.getOrCreateDataset("qa-dataset", "Evaluation dataset", "my-project");
await dataset.insert([
  {
    question: "What is the capital of France?",
    expected_answer: "Paris",
  },
  {
    question: "How do you calculate the area of a circle?",
    expected_answer: "π × radius²",
  },
]);
const result = await evaluatePrompt({
  dataset,
  messages: [
    {
      role: "system",
      content:
        "You are a helpful assistant. Answer questions accurately and concisely.",
    },
    { role: "user", content: "{{question}}" },
  ],
  model: "gpt-5-nano",
  projectName: "my-project",
});
console.log(`Experiment ID: ${result.experimentId}`);
console.log(`Evaluated ${result.testResults.length} items`);

With Scoring Metrics

Evaluate prompts with automatic scoring:

import { evaluatePrompt } from "opik";
import { Hallucination, ExactMatch } from "opik";
// Create dataset with expected answers
const dataset = await client.getOrCreateDataset("geography-qa", "Geography evaluation dataset", "my-project");
await dataset.insert([
  {
    country: "France",
    expected_answer: "Paris",
  },
  {
    country: "Japan",
    expected_answer: "Tokyo",
  },
]);
await evaluatePrompt({
  dataset,
  messages: [
    {
      role: "user",
      content: "What is the capital of {{country}}?",
    },
  ],
  model: "gpt-5-nano",
  scoringMetrics: [
    new ExactMatch(), // Check exact match with expected output
    new Hallucination(), // Check for hallucinations
  ],
  experimentName: "Geography Quiz Evaluation",
  projectName: "my-project",
});

Using LanguageModel Instances

Use LanguageModel instances for provider-specific features:

import { openai } from "@ai-sdk/openai";
import { evaluatePrompt } from "opik";
// Create model instance
const customModel = openai("gpt-5-nano");
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "Summarize: {{text}}" }],
  model: customModel,
  experimentConfig: {
    model_provider: "openai",
    model_name: "gpt-5-nano",
  },
  projectName: "my-project",
});

Multi-Provider Model Support

The function supports models from multiple providers:

// OpenAI
model: "gpt-5-nano";
// Anthropic
model: "claude-3-5-sonnet-latest";
// Google Gemini
model: "gemini-2.0-flash";
// Or use provider-specific LanguageModel instances
import { anthropic } from "@ai-sdk/anthropic";
const claude = anthropic("claude-3-5-sonnet-latest");
model: claude;

Linking to Prompt Versions

Track which prompt versions are used in evaluations:

import { Opik, evaluatePrompt } from "opik";
const client = new Opik();
// Create or get a prompt
const prompt = await client.createPrompt({
  name: "customer-support-prompt",
  prompt: "{{system_message}}\n\nUser: {{user_query}}",
});
// Link the prompt to the evaluation
await evaluatePrompt({
  dataset,
  messages: [
    { role: "system", content: "{{system_message}}" },
    { role: "user", content: "{{user_query}}" },
  ],
  model: "gpt-5-nano",
  prompts: [prompt], // Link to prompt
  experimentName: "Customer Support - v2.1",
  projectName: "my-project",
});

Template Types

Mustache Templates (Default)

await evaluatePrompt({
  dataset,
  messages: [
    {
      role: "user",
      content: "Hello {{name}}, your order #{{order_id}} is ready.",
    },
  ],
  templateType: "mustache", // This is the default
  projectName: "my-project",
});

Jinja2 Templates

await evaluatePrompt({
  dataset,
  messages: [
    {
      role: "user",
      content: "Hello {{ name }}, your order #{{ order_id }} is ready.",
    },
  ],
  templateType: "jinja2",
  projectName: "my-project",
});

Scoring Key Mapping

Map dataset fields to metric parameter names:

// Dataset has: { question: "...", reference_answer: "..." }
// Metric expects: { input: "...", expected: "..." }
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{question}}" }],
  scoringMetrics: [new ExactMatch()],
  scoringKeyMapping: {
    input: "question",
    expected: "reference_answer",
  },
  projectName: "my-project",
});

Subset Evaluation

Evaluate only a subset of the dataset for quick iteration:

await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{prompt}}" }],
  nbSamples: 5, // Only evaluate first 5 items
  experimentName: "Quick Test",
  projectName: "my-project",
});

How It Works

When you call evaluatePrompt, the following happens:

Template Formatting: For each dataset item, message templates are formatted with item variables
Model Invocation: The formatted messages are sent to the specified model to generate a response
Experiment Creation: An experiment is created (or updated) with metadata
Metric Scoring: If metrics are provided, each output is scored
Result Aggregation: Results are collected and returned

Experiment Configuration

The function automatically enriches experiment configuration with:

prompt_template: The message templates used
model: The model identifier (name or type)

You can add additional metadata via experimentConfig:

experimentConfig: {
  // Auto-added by evaluatePrompt:
  // prompt_template: [{ role: 'user', content: '...' }]
  // model: 'gpt-5-nano'
  // Your custom metadata:
  temperature: 0.7,
  version: "v2.0",
  author: "team-ai",
  description: "Testing improved prompt structure",
};

Best Practices

1. Start Simple, Then Add Metrics

Begin with basic prompt evaluation, then add metrics as needed:

// Step 1: Basic evaluation to see outputs
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  projectName: "my-project",
});
// Step 2: Add metrics after reviewing outputs
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  scoringMetrics: [new Hallucination()],
  projectName: "my-project",
});

2. Use Descriptive Experiment Names

Make it easy to find and compare experiments:

experimentName: "Translation - GPT-4o - v2.3 - 2025-01-15";

3. Version Your Prompts

Link evaluations to prompt versions for better tracking:

const prompt = await client.createPrompt({
  name: "qa-prompt",
  prompt: "Answer: {{question}}",
  version: "v2.3",
});
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "Answer: {{question}}" }],
  prompts: [prompt],
  projectName: "my-project",
});

4. Start with Small Samples

Use nbSamples for quick iteration before full evaluation:

// Quick test with 10 samples
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  nbSamples: 10,
  projectName: "my-project",
});
// Full evaluation once satisfied
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  projectName: "my-project",
  // Evaluate entire dataset
});

5. Include Context in System Messages

Structure your prompts with clear system messages:

messages: [
  {
    role: "system",
    content:
      "You are an expert {{domain}} assistant. Provide accurate, concise answers.",
  },
  {
    role: "user",
    content: "{{question}}",
  },
];

Error Handling

The function validates inputs and throws errors for common issues:

try {
  await evaluatePrompt({
    dataset,
    messages: [],
  });
} catch (error) {
  console.error(error.message);
  // Error: Messages array is required and cannot be empty
}

Common validation errors:

Missing required dataset parameter
Empty messages array
Invalid experimentConfig (must be plain object)
Invalid templateType (must be ‘mustache’ or ‘jinja2’)

evaluatePrompt Function