> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://www.comet.com/docs/opik/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://www.comet.com/docs/opik/_mcp/server.

# Evaluation Metrics

In Opik 2.0, experiments are project-scoped. When using metrics in evaluations, specify a `projectName` in the `evaluate()` call so results are associated with the correct project.

Metrics are a fundamental component of the Opik evaluation function. They provide quantitative assessments of your AI models' outputs, enabling objective comparisons and performance tracking over time.

### What Are Metrics?

In Opik, a metric is a function that calculates a score based on specific inputs, such as model outputs and reference answers. All metrics in Opik extend the `BaseMetric` abstract class, which provides the core functionality for validation and tracking.

```typescript
abstract class BaseMetric<GenericZodObjectType> {
  public readonly name: string;
  public readonly trackMetric: boolean;
  public abstract readonly validationSchema: GenericZodObjectType;

  abstract score(
    input: Infer<GenericZodObjectType>
  ):
    | EvaluationScoreResult
    | EvaluationScoreResult[]
    | Promise<EvaluationScoreResult>
    | Promise<EvaluationScoreResult[]>;
}
```

### How Metrics Calculate Scores

Each metric must implement the `score` method, which:

1. Accepts an `input` object containing combined data from the task output, dataset item, and scoringKeyMapping
2. Processes the inputs to produce a score
3. Returns an `EvaluationScoreResult` or array of results, which includes:
   * `name`: The metric name
   * `value`: The numerical score (typically 0.0-1.0)
   * `reason`: A human-readable explanation for the score

### Types of Metrics

Opik supports different types of metrics:

1. **Heuristic metrics**: Simple rule-based evaluations (e.g., exact match, contains, regex match)
2. **LLM Judge metrics**: AI-powered evaluations that use language models to assess output quality

## Built-in Metrics

Opik provides several built-in metrics for common evaluation scenarios:

### ExactMatch

Checks if the model output exactly matches the expected output:

```typescript
const exactMatch = new ExactMatch();
// Usage requires both 'output' and 'expected' parameters
```

### Contains

Checks if the model output contains specific text:

```typescript
const contains = new Contains();
// Usage requires both 'output' and 'expected' parameters
```

### RegexMatch

Checks if the model output matches a regular expression pattern:

```typescript
const regexMatch = new RegexMatch();
// Usage requires 'output' and 'pattern' parameters
```

### IsJson

Checks if the output is valid JSON:

```typescript
const isJson = new IsJson();
// Usage requires 'output' parameter
```

## Metric Configuration

### Custom Naming and Tracking

Each metric can be configured with a custom name and tracking option:

```typescript
// Create metric with custom name
const exactMatch = new ExactMatch("my_exact_match");

// Create metric with tracking disabled
const regexMatch = new RegexMatch("custom_regex", false);
```

### Combining Multiple Metrics

You can use multiple metrics in a single evaluation:

```typescript
const metrics = [new ExactMatch(), new Contains(), new RegexMatch()];

// In your evaluation configuration
await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: metrics,
});
```

## Input Requirements

### Validation Schema

Each metric defines a Zod validation schema that specifies required inputs:

```typescript
// ExactMatch validation schema example
const validationSchema = z.object({
  output: z.string(), // The model output
  expected: z.string(), // The expected output
});
```

The validation system ensures all required parameters are present before executing the metric.

### Mapping Inputs

You can map dataset fields and task outputs to metric inputs using `scoringKeyMapping`:

```typescript
await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: [new ExactMatch()],
  scoringKeyMapping: {
    // Map dataset/task fields to metric parameter names
    output: "model.response",
    expected: "dataset.answer",
  },
});
```

## Score Interpretation

### Score Ranges

Most metrics in Opik return scores between 0.0 and 1.0:

* 1.0: Perfect match or ideal performance
* 0.0: No match or complete failure
* Intermediate values: Partial matches or varying degrees of success

## Creating Custom Metrics

### Implementing Your Own Metric

To create a custom metric:

1. Extend the `BaseMetric` class
2. Define a validation schema using Zod
3. Implement the `score` method

Here's an example of a custom metric that checks if output length is within a specified range:

```typescript
import z from "zod";
import { BaseMetric } from "@opik/sdk";
import { EvaluationScoreResult } from "@opik/sdk";

// Define validation schema
const validationSchema = z.object({
  output: z.string(),
  minLength: z.number(),
  maxLength: z.number(),
});

// Infer TypeScript type from schema
type Input = z.infer<typeof validationSchema>;

export class LengthRangeMetric extends BaseMetric {
  public validationSchema = validationSchema;

  constructor(name = "length_range", trackMetric = true) {
    super(name, trackMetric);
  }

  async score(input: Input): Promise<EvaluationScoreResult> {
    const { output, minLength, maxLength } = input;
    const length = output.length;

    // Calculate score (1.0 if within range, 0.0 otherwise)
    const isWithinRange = length >= minLength && length <= maxLength;
    const score = isWithinRange ? 1.0 : 0.0;

    // Return result with explanation
    return {
      name: this.name,
      value: score,
      reason: isWithinRange
        ? `Output length (${length}) is within range ${minLength}-${maxLength}`
        : `Output length (${length}) is outside range ${minLength}-${maxLength}`,
    };
  }
}
```

### Validation Best Practices

When creating custom metrics:

1. **Define clear validation schemas**:

   ```typescript
   const validationSchema = z.object({
     output: z.string().min(1, "Output is required"),
     threshold: z.number().min(0).max(1),
   });
   ```

2. **Return meaningful reasons**:

   ```typescript
   return {
     name: this.name,
     value: score,
     reason: `Score ${score.toFixed(2)} because [detailed explanation]`,
   };
   ```

3. **Normalize scores** to a consistent range (typically 0.0-1.0) for easier comparison with other metrics

## LLM Judge Metrics

LLM Judge metrics use language models to evaluate the quality of LLM outputs. These metrics provide more nuanced evaluation than simple heuristic checks.

### AnswerRelevance

Evaluates how relevant the output is to the input question:

```typescript
import { AnswerRelevance } from "opik";

// Using default model (gpt-5-nano)
const metric = new AnswerRelevance();

// With custom model ID
const metricWithModel = new AnswerRelevance({
  model: "claude-3-5-sonnet-latest",
});

// With LanguageModel instance
import { openai } from "@ai-sdk/openai";
const customModel = openai("gpt-5-nano");
const metricWithCustomModel = new AnswerRelevance({ model: customModel });

// Usage
const score = await metric.score({
  input: "What is the capital of France?",
  output: "The capital of France is Paris.",
  context: ["France is a country in Western Europe."], // Optional
});

console.log(score.value); // 0.0 to 1.0
console.log(score.reason); // Explanation of the score
```

#### Parameters

* `input` (required): The question or prompt
* `output` (required): The model's response to evaluate
* `context` (optional): Additional context for evaluation

#### Score Range

* **1.0**: Perfect relevance - output directly addresses the input
* **0.5**: Partial relevance - output is somewhat related but incomplete
* **0.0**: No relevance - output doesn't address the input

### Hallucination

Detects whether the output contains hallucinated or unfaithful information:

```typescript
import { Hallucination } from "opik";

const metric = new Hallucination();

// Without context - checks against general knowledge
const score1 = await metric.score({
  input: "What is the capital of France?",
  output:
    "The capital of France is Paris. It is famous for its iconic Eiffel Tower.",
});

// With context - checks faithfulness to provided context
const score2 = await metric.score({
  input: "What is the capital of France?",
  output:
    "The capital of France is Paris. It is famous for its iconic Eiffel Tower.",
  context: [
    "France is a country in Western Europe. Its capital is Paris, which is known for landmarks like the Eiffel Tower.",
  ],
});

console.log(score2.value); // 1.0 = hallucination detected, 0.0 = no hallucination
console.log(score2.reason); // Array of reasons for the score
```

#### Parameters

* `input` (required): The original question or prompt
* `output` (required): The model's response to evaluate
* `context` (optional): Reference information to check against

#### Score Values

* **0.0**: No hallucination - output is faithful to context/facts
* **1.0**: Hallucination detected - output contains false or unsupported information

### Moderation

Checks if the output contains harmful, inappropriate, or unsafe content:

```typescript
import { Moderation } from "opik";

const metric = new Moderation();

const score = await metric.score({
  input: "Tell me about safety guidelines",
  output: "Here are some safety guidelines...",
});

console.log(score.value); // 1.0 = harmful content detected, 0.0 = safe
console.log(score.reason); // Explanation of moderation decision
```

#### Parameters

* `input` (required): The original prompt
* `output` (required): The model's response to evaluate

#### Score Values

* **0.0**: Safe - no harmful content detected
* **1.0**: Harmful - inappropriate or unsafe content detected

### Usefulness

Evaluates how useful the output is in addressing the input:

```typescript
import { Usefulness } from "opik";

const metric = new Usefulness();

const score = await metric.score({
  input: "How do I reset my password?",
  output:
    "To reset your password, click 'Forgot Password' on the login page, enter your email, and follow the instructions sent to your inbox.",
});

console.log(score.value); // 0.0 to 1.0
console.log(score.reason); // Explanation of usefulness score
```

#### Parameters

* `input` (required): The question or request
* `output` (required): The model's response to evaluate

#### Score Range

* **1.0**: Very useful - comprehensive and actionable
* **0.5**: Somewhat useful - partially helpful
* **0.0**: Not useful - doesn't help address the input

### GEval

GEval is a task-agnostic LLM-as-a-judge metric that allows you to define custom evaluation criteria. The metric first generates a chain-of-thought (CoT) evaluation plan, then scores the output on a 0-10 scale (normalized to 0.0-1.0).

```typescript
import { GEval } from "opik";

const metric = new GEval({
  taskIntroduction: "You evaluate the politeness of customer service responses.",
  evaluationCriteria: "Score from 0 (rude) to 10 (very polite). Consider tone, word choice, and empathy.",
  model: "gpt-4o",
});

const score = await metric.score({
  output: "Thanks so much for your patience! I'm happy to help resolve this for you.",
});

console.log(score.value); // 0.0 to 1.0
console.log(score.reason); // Explanation of the score
```

#### Parameters

* `taskIntroduction` (required): Description of what should be evaluated
* `evaluationCriteria` (required): Detailed criteria defining what "good" looks like
* `model` (optional): Model to use for evaluation (defaults to "gpt-4o")
* `name` (optional): Custom metric name (defaults to "g\_eval\_metric")
* `temperature` (optional): Sampling temperature for generation
* `seed` (optional): Seed for reproducible outputs
* `maxTokens` (optional): Maximum response length
* `modelSettings` (optional): Advanced model configuration

#### Score Range

* **1.0**: Perfect score (10/10 from the judge)
* **0.5**: Average score (5/10 from the judge)
* **0.0**: Lowest score (0/10 from the judge)

#### How It Works

GEval uses a two-stage process:

1. **Chain of Thought Generation**: Creates step-by-step evaluation instructions based on your task and criteria (cached for reuse)
2. **Scoring**: Evaluates the output using the CoT, returning a score with reasoning

When using OpenAI models, GEval leverages logprobs to compute a weighted average of score probabilities for more robust scoring.

### Built-in GEval Judges

Opik provides pre-configured GEval judges for common evaluation scenarios. Each extends `GEval` with domain-specific prompts:

#### QARelevanceJudge

Evaluates whether an answer directly addresses the question:

```typescript
import { QARelevanceJudge } from "opik";

const judge = new QARelevanceJudge({ model: "gpt-4o" });

const score = await judge.score({
  output: `QUESTION: What causes rainbows?
ANSWER: Rainbows are caused by refraction and reflection of light in water droplets.`,
});

console.log(score.value); // High score for relevant answer
```

#### SummarizationConsistencyJudge

Checks if a summary is faithful to the source material:

```typescript
import { SummarizationConsistencyJudge } from "opik";

const judge = new SummarizationConsistencyJudge();

const score = await judge.score({
  output: `SOURCE: The company announced Q4 revenue of $2.5M.
SUMMARY: The company had strong Q4 performance with $2.5M revenue.`,
});
```

#### SummarizationCoherenceJudge

Evaluates the structure and clarity of summaries:

```typescript
import { SummarizationCoherenceJudge } from "opik";

const judge = new SummarizationCoherenceJudge();

const score = await judge.score({
  output: "SUMMARY: First, the project started. Then it ended. Finally, it began.",
});

console.log(score.value); // Low score for incoherent summary
```

#### DialogueHelpfulnessJudge

Assesses how helpful an assistant reply is in dialogue context:

```typescript
import { DialogueHelpfulnessJudge } from "opik";

const judge = new DialogueHelpfulnessJudge();

const transcript = `USER: How do I reset my password?
ASSISTANT: Visit settings and click reset.
USER: I cannot see that option.
ASSISTANT: Please contact support.`;

const score = await judge.score({ output: transcript });
```

#### Bias Detection Judges

Detect various forms of bias in responses:

```typescript
import {
  DemographicBiasJudge,
  GenderBiasJudge,
  PoliticalBiasJudge,
  ReligiousBiasJudge,
  RegionalBiasJudge,
} from "opik";

// Demographic bias
const demographicJudge = new DemographicBiasJudge();
const score1 = await demographicJudge.score({
  output: "People from X group are always late.",
});

// Gender bias
const genderJudge = new GenderBiasJudge();
const score2 = await genderJudge.score({
  output: "Women are naturally worse at math.",
});

// Political bias
const politicalJudge = new PoliticalBiasJudge();
const score3 = await politicalJudge.score({
  output: "Vote for candidate X because Y is corrupt.",
});
```

#### Agent Evaluation Judges

Evaluate agent task completion and tool usage:

```typescript
import { AgentTaskCompletionJudge, AgentToolCorrectnessJudge } from "opik";

// Task completion
const taskJudge = new AgentTaskCompletionJudge();
const score1 = await taskJudge.score({
  output: "Agent gathered quotes, compared options, and booked travel.",
});

// Tool correctness
const toolJudge = new AgentToolCorrectnessJudge();
const score2 = await toolJudge.score({
  output: "Tool weather_api called with city='Paris' but response ignored.",
});
```

#### PromptUncertaintyJudge

Estimates how ambiguous a prompt is:

```typescript
import { PromptUncertaintyJudge } from "opik";

const judge = new PromptUncertaintyJudge();

const score = await judge.score({
  output: "Summarise the attached 400 page contract in one sentence and guarantee there are no mistakes.",
});

console.log(score.value); // High score indicates high uncertainty
```

#### ComplianceRiskJudge

Flags non-compliant or risky claims in regulated sectors:

```typescript
import { ComplianceRiskJudge } from "opik";

const judge = new ComplianceRiskJudge({ model: "gpt-4o" });

const score = await judge.score({
  output: "This pill cures diabetes in a week.",
});

console.log(score.value); // High score indicates high risk
```

#### Available Judges

All built-in GEval judges:

* `QARelevanceJudge` - Answer relevance to questions
* `SummarizationConsistencyJudge` - Summary faithfulness
* `SummarizationCoherenceJudge` - Summary structure and clarity
* `DialogueHelpfulnessJudge` - Assistant helpfulness in dialogue
* `DemographicBiasJudge` - Demographic stereotyping
* `GenderBiasJudge` - Gender stereotyping
* `PoliticalBiasJudge` - Political bias
* `ReligiousBiasJudge` - Religious bias
* `RegionalBiasJudge` - Geographic/cultural bias
* `AgentTaskCompletionJudge` - Agent task fulfillment
* `AgentToolCorrectnessJudge` - Agent tool usage correctness
* `PromptUncertaintyJudge` - Prompt ambiguity
* `ComplianceRiskJudge` - Regulatory compliance risk

## Configuring LLM Judge Metrics

### Model Configuration

All LLM Judge metrics accept a `model` parameter in their constructor:

```typescript
import { openai } from "@ai-sdk/openai";
import { anthropic } from "@ai-sdk/anthropic";
import { google } from "@ai-sdk/google";
import { Hallucination } from "opik";

// Using model ID string
const metric1 = new Hallucination({ model: "gpt-5-nano" });
const metric2 = new Hallucination({ model: "claude-3-5-sonnet-latest" });
const metric3 = new Hallucination({ model: "gemini-2.0-flash" });

// Using LanguageModel instance
const customModel = openai("gpt-5-nano");
const metric4 = new Hallucination({ model: customModel });
```

### Async Scoring

All LLM Judge metrics support asynchronous scoring:

```typescript
import { AnswerRelevance } from "opik";

const metric = new AnswerRelevance();

// Async/await
const score = await metric.score({
  input: "What is TypeScript?",
  output: "TypeScript is a typed superset of JavaScript.",
});

// Promise chain
metric
  .score({
    input: "What is TypeScript?",
    output: "TypeScript is a typed superset of JavaScript.",
  })
  .then((score) => console.log(score.value));
```

### Combining Multiple LLM Judge Metrics

Use multiple metrics together for comprehensive evaluation:

```typescript
import { AnswerRelevance, Hallucination, Moderation, Usefulness } from "opik";
import { evaluate } from "opik";

await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: [
    new AnswerRelevance(),
    new Hallucination(),
    new Moderation(),
    new Usefulness(),
  ],
});
```

### Custom Model for Each Metric

Different metrics can use different models:

```typescript
import { openai } from "@ai-sdk/openai";
import { anthropic } from "@ai-sdk/anthropic";
import { AnswerRelevance, Hallucination } from "opik";

// Use GPT-4o for answer relevance
const relevanceMetric = new AnswerRelevance({
  model: openai("gpt-5-nano"),
});

// Use Claude for hallucination detection
const hallucinationMetric = new Hallucination({
  model: anthropic("claude-3-5-sonnet-latest"),
});

await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: [relevanceMetric, hallucinationMetric],
});
```

## LLM Judge Metric Best Practices

### 1. Use Model ID Strings for Simplicity

For most use cases, use model ID strings directly:

```typescript
import { Hallucination } from "opik";

const metric = new Hallucination({ model: "gpt-5-nano" });
```

The Opik SDK handles model configuration internally for optimal evaluation performance.

### 2. Provide Context When Available

Context improves evaluation accuracy:

```typescript
// Better: With context
await metric.score({
  input: "What is the capital?",
  output: "The capital is Paris.",
  context: ["France is a country in Europe. Its capital is Paris."],
});

// OK: Without context (relies on general knowledge)
await metric.score({
  input: "What is the capital of France?",
  output: "The capital is Paris.",
});
```

### 3. Choose Appropriate Models

Match model capabilities to metric requirements:

```typescript
// Complex reasoning: Use GPT-5 or Claude Sonnet
const complexMetric = new AnswerRelevance({ model: "gpt-5" });

// Simple checks: Use faster, cheaper models
const simpleMetric = new Moderation({ model: "gpt-5-nano" });
```

### 4. Handle Errors Gracefully

LLM calls can fail - handle errors appropriately:

```typescript
try {
  const score = await metric.score({
    input: "What is TypeScript?",
    output: "TypeScript is a typed superset of JavaScript.",
  });
  console.log(score);
} catch (error) {
  console.error("Metric evaluation failed:", error);
  // Implement fallback or retry logic
}
```

### 5. Batch Evaluations

Use the evaluate function for efficient batch processing:

```typescript
// More efficient for multiple items
await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: [new Hallucination()],
  scoringWorkers: 5, // Parallel scoring
});

// Less efficient for batch processing
for (const item of datasetItems) {
  await metric.score(item); // Sequential scoring
}
```

## Score Interpretation

### Understanding LLM Judge Scores

LLM Judge metrics return structured scores with:

```typescript
interface EvaluationScoreResult {
  name: string; // Metric name
  value: number; // Numerical score (0.0-1.0 typically)
  reason: string | string[]; // Explanation for the score
}
```

### Example Score Results

```typescript
// AnswerRelevance
{
  name: "answer_relevance",
  value: 0.95,
  reason: "The answer directly addresses the question with accurate information"
}

// Hallucination
{
  name: "hallucination",
  value: 0.0,
  reason: ["All information is supported by the context", "No contradictions found"]
}

// Moderation
{
  name: "moderation",
  value: 0.0,
  reason: "Content is safe and appropriate"
}
```

## Generation Parameters

### Configuring Temperature, Seed, and MaxTokens

All LLM Judge metrics support generation parameters in their constructor:

```typescript
import { Hallucination, AnswerRelevance } from "opik";

// Configure generation parameters
const metric = new Hallucination({
  model: "gpt-5-nano",
  temperature: 0.3, // Lower = more deterministic
  seed: 42, // For reproducible outputs
  maxTokens: 1000, // Maximum response length
});

// Different settings for different metrics
const relevanceMetric = new AnswerRelevance({
  model: "claude-3-5-sonnet-latest",
  temperature: 0.7, // Higher = more creative
  seed: 12345,
});

// Use the metrics
const score = await metric.score({
  input: "What is the capital of France?",
  output: "The capital of France is Paris.",
  context: ["France is a country in Western Europe."],
});
```

### Advanced Model Settings

For provider-specific advanced parameters, use `modelSettings`:

```typescript
import { Hallucination } from "opik";

const metric = new Hallucination({
  model: "gpt-5-nano",
  temperature: 0.5,
  modelSettings: {
    topP: 0.9, // Nucleus sampling
    topK: 50, // Top-K sampling
    presencePenalty: 0.1, // Reduce repetition
    frequencyPenalty: 0.2, // Reduce phrase repetition
    stopSequences: ["END"], // Custom stop sequences
  },
});
```

For provider-specific options not exposed through `modelSettings`, use `LanguageModel` instances:

```typescript
import { openai } from "@ai-sdk/openai";
import { anthropic } from "@ai-sdk/anthropic";
import { google } from "@ai-sdk/google";

// OpenAI with structured outputs
const openaiModel = openai("gpt-5-nano", {
  structuredOutputs: true,
});

// Anthropic with cache control
const anthropicModel = anthropic("claude-3-5-sonnet-latest", {
  cacheControl: true,
});

// Google Gemini with specific configuration
const googleModel = google("gemini-2.0-flash");

const metric1 = new Hallucination({ model: openaiModel });
const metric2 = new Hallucination({ model: anthropicModel });
const metric3 = new Hallucination({ model: googleModel });
```

See [Vercel AI SDK Provider Documentation](https://sdk.vercel.ai/providers) for provider-specific options:

* [OpenAI Provider Options](https://sdk.vercel.ai/providers/ai-sdk-providers/openai)
* [Anthropic Provider Options](https://sdk.vercel.ai/providers/ai-sdk-providers/anthropic)
* [Google Provider Options](https://ai-sdk.dev/providers/ai-sdk-providers/google)

## See Also

* [Models](/reference/typescript-sdk/evaluation/models) - Configuring language models for metrics
* [evaluate Function](/reference/typescript-sdk/evaluation/evaluate_function) - Using metrics in evaluations
* [evaluatePrompt Function](/reference/typescript-sdk/evaluation/evaluate_prompt_function) - Using metrics with prompt evaluation