Overview

Evaluation in Opik helps you assess and measure the quality of your LLM outputs across different dimensions. It provides a framework to systematically test your prompts and models against datasets, using various metrics to measure performance.

Opik also provides a set of pre-built metrics for common evaluation tasks. These metrics are designed to help you quickly and effectively gauge the performance of your LLM outputs and include metrics such as Hallucination, Answer Relevance, Context Precision/Recall and more. You can learn more about the available metrics in the Metrics Overview section.

Running an Evaluation

Each evaluation is defined by a dataset, an evaluation task and a set of evaluation metrics:

1. Dataset: A dataset is a collection of samples that represent the inputs and, optionally, expected outputs for your LLM application.
2. Evaluation task: This maps the inputs stored in the dataset to the output you would like to score. The evaluation task is typically a prompt template or the LLM application you are building.
3. Metrics: The metrics you would like to use when scoring the outputs of your LLM

To simplify the evaluation process, Opik provides two main evaluation methods: evaluate_prompt for evaluation prompt templates and a more general evaluate method for more complex evaluation scenarios.

1
import opik
2
from opik.evaluation import evaluate_prompt
3
from opik.evaluation.metrics import Hallucination
4
5
# Create a dataset that contains the samples you want to evaluate
6
opik_client = opik.Opik()
7
dataset = opik_client.get_or_create_dataset("Evaluation test dataset", project_name="my-project")
8
dataset.insert([
9
    {"input": "Hello, world!", "expected_output": "Hello, world!"},
10
    {"input": "What is the capital of France?", "expected_output": "Paris"},
11
])
12
13
# Run the evaluation
14
result = evaluate_prompt(
15
    dataset=dataset,
16
    messages=[{"role": "user", "content": "Translate the following text to French: {{input}}"}],
17
    model="gpt-3.5-turbo",  # or your preferred model
18
    scoring_metrics=[Hallucination()],
19
    project_name="my-project"
20
)

Analyzing Evaluation Results

Once the evaluation is complete, Opik allows you to manually review the results and compare them with previous iterations.

In the experiment pages, you will be able to:

1. Review the output provided by the LLM for each sample in the dataset
2. Deep dive into each sample by clicking on the item ID
3. Review the experiment configuration to know how the experiment was Run
4. Compare multiple experiments side by side

Analyzing Evaluation Results in Python

To analyze the evaluation results in Python, you can use the EvaluationResult.aggregate_evaluation_scores() method to retrieve the aggregated score statistics:

1
import opik
2
from opik.evaluation import evaluate_prompt
3
from opik.evaluation.metrics import Hallucination
4
5
# Create a dataset that contains the samples you want to evaluate
6
opik_client = opik.Opik()
7
dataset = opik_client.get_or_create_dataset("Evaluation test dataset", project_name="my-project")
8
dataset.insert([
9
    {"input": "Hello, world!", "expected_output": "Hello, world!"},
10
    {"input": "What is the capital of France?", "expected_output": "Paris"},
11
])
12
# Run the evaluation
13
result = evaluate_prompt(
14
    dataset=dataset,
15
    messages=[{"role": "user", "content": "Translate the following text to French: {{input}}"}],
16
    model="gpt-5",  # or model that you want to evaluate
17
    scoring_metrics=[Hallucination()],
18
    verbose=2,  # print detailed statistics report for every metric
19
    project_name="my-project",
20
)
21
22
# Retrieve and print the aggregated scores statistics (mean, min, max, std) per metric
23
scores = result.aggregate_evaluation_scores()
24
for metric_name, statistics in scores.aggregated_scores.items():
25
    print(f"{metric_name}: {statistics}")

You can use aggregated scores to compare the performance of different models or different versions of the same model.

Experiment-Level Metrics

In addition to per-item metrics, you can compute experiment-level aggregate metrics that are calculated across all test results. These experiment scores are displayed in the Opik UI alongside feedback scores and can be used for sorting and filtering experiments.

Learn more

You can learn more about Opik’s evaluation features in:

1. Evaluation concepts
2. Evaluate prompts
3. Evaluate threads
4. Evaluate complex LLM applications
5. Evaluation metrics
6. Manage datasets