EvaluationAdvanced

Manually logging experiments

Step by step guide to logging evaluation results using Python SDK and REST API
View as Markdown

In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a project_name when creating datasets and running experiments so they are associated with the correct project.

Evaluating your LLM application allows you to have confidence in the performance of your LLM application. In this guide, we will walk through manually creating experiments using data you have already computed.

This guide focuses on logging pre-computed evaluation results. If you’re looking to run evaluations with Opik computing the metrics, refer to the Evaluate your agent guide.

The process involves these key steps:

  1. Create a dataset with your test cases
  2. Prepare your evaluation results
  3. Log experiment items in bulk

1. Create a Dataset

First, you’ll need to create a dataset containing your test cases. This dataset will be linked to your experiments.

import { Opik } from "opik";
const client = new Opik({
  apiKey: "your-api-key",
  apiUrl: "https://www.comet.com/opik/api",
  projectName: "your-project-name",
  workspaceName: "your-workspace-name",
});
const dataset = await client.getOrCreateDataset("My dataset");
await dataset.insert([
  {
    user_question: "What is the capital of France?",
    expected_output: "Paris"
  },
  {
    user_question: "What is the capital of Japan?",
    expected_output: "Tokyo"
  },
  {
    user_question: "What is the capital of Brazil?",
    expected_output: "Brasília"
  }
]);

Dataset item IDs will be automatically generated if not provided. If you do provide your own IDs, ensure they are in UUID7 format.

2. Prepare Evaluation Results

Structure your evaluation results with the necessary fields. Each experiment item should include:

  • dataset_item_id: The ID of the dataset item being evaluated
  • evaluate_task_result: The output from your LLM application
  • feedback_scores: Array of evaluation metrics (optional)
const datasetItems = await dataset.getItems();
const mockResponses = {
  "What is the capital of France?": "The capital of France is Paris.",
  "What is the capital of Japan?": "Japan's capital is Tokyo.",
  "What is the capital of Brazil?": "The capital of Brazil is Rio de Janeiro."
}
// This would be replaced by your specific logic, the goal is simply to have an array of
// evaluation items with a dataset_item_id, evaluate_task_result and feedback_scores
const evaluationItems = datasetItems.map(item => {  
  const response = mockResponses[item.user_question] || "I don't know";
    return {
      dataset_item_id: item.id,
      evaluate_task_result: { prediction: response },
      feedback_scores: [{ name: "accuracy", value: response.includes(item.expected_output) ? 1.0 : 0.0, source: "sdk" }]
    }
});

3. Log Experiment Items in Bulk

Use the bulk endpoint to efficiently log multiple evaluation results at once.

project_name sets the project for the experiment, dataset, and auto-created traces. When provided, it is used — and the request fails (409) if it conflicts with the existing experiment’s or dataset’s project. When omitted, the project is derived from the existing experiment or dataset; if neither applies, it falls back to the deprecated Default Project (the response includes an X-Opik-Deprecation header).

import { Opik } from "opik";
const projectName = "your-project-name";
const client = new Opik({
  apiKey: "your-api-key",
  apiUrl: "https://www.comet.com/opik/api",
  projectName,
  workspaceName: "your-workspace-name",
});
const experimentName = "Bulk experiment upload";
const datasetName = "geography-questions";
const items = [
  {
    dataset_item_id: "dataset-item-id-1",
    evaluate_task_result: { prediction: "The capital of France is Paris." },
    feedback_scores: [{ name: "accuracy", value: 1.0, source: "sdk" }]
  }
];
await client.api.experiments.experimentItemsBulk({ experimentName, datasetName, projectName, items });

Request Size Limit: The maximum allowed payload size is 4MB. For larger submissions, divide the data into smaller batches.

If you wish to divide the data into smaller batches, just add the experiment_id to the payload so experiment items can be added to an existing experiment.

Below is an example of splitting the evaluation_items into two batches which will both be added to the same experiment:

import { generateId } from "opik";
const experimentId = generateId();
const experimentName = "Bulk experiment upload";
// Split evaluation_items into two batches
const mid = Math.floor(evaluationItems.length / 2);
const halves = [
  evaluationItems.slice(0, mid),
  evaluationItems.slice(mid)
];
for (const half of halves) {
  await client.restClient.experiments.experimentItemsBulk({
    experimentId: experimentId,
    experimentName: experimentName,
    datasetName: "geography-questions",
    projectName: "your-project-name",
    items: half.map(item => ({
      datasetItemId: item.datasetItemId,
      evaluateTaskResult: item.evaluateTaskResult,
      feedbackScores: item.feedbackScores.map(score => ({
        ...score,
        source: "sdk"
      }))
    }))
  });
}

4. Analyzing the results

Once you have logged your experiment items, you can analyze the results in the Opik UI and even compare different experiments to one another.

Complete Example

Here’s a complete example that puts all the steps together:

import { Opik } from "opik";
const projectName = "your-project-name";
const datasetName = "geography-questions";
// Configure Opik
const client = new Opik({
apiKey: "your-api-key",
apiUrl: "https://www.comet.com/opik/api",
projectName,
workspaceName: "your-workspace-name",
});
// Step 1: Create dataset
const dataset = await client.getOrCreateDataset(datasetName);
const localDatasetItems = [
{
  user_question: "What is the capital of France?",
  expected_output: "Paris"
},
{
  user_question: "What is the capital of Japan?",
  expected_output: "Tokyo"
}
];
await dataset.insert(localDatasetItems);
// Step 2: Get dataset items and prepare evaluation results
const datasetItems = await dataset.getItems();
// Helper function to get dataset item ID
const getDatasetItem = (country: string) => {
return datasetItems.find(item =>
  item.user_question.toLowerCase().includes(country.toLowerCase())
);
};
// Prepare evaluation results
const evaluationItems = [
{
  dataset_item_id: getDatasetItem("France")?.id,
  evaluate_task_result: { prediction: "The capital of France is Paris." },
  feedback_scores: [{ name: "accuracy", value: 1.0 }]
},
{
  dataset_item_id: getDatasetItem("Japan")?.id,
  evaluate_task_result: { prediction: "Japan's capital is Tokyo." },
  feedback_scores: [{ name: "accuracy", value: 1.0 }]
}
];
// Step 3: Log experiment results
const experimentName = `geography-bot-${Math.random().toString(36).substr(2, 4)}`;
await client.api.experiments.experimentItemsBulk({
experimentName,
datasetName,
projectName,
items: evaluationItems.map(item => ({
  datasetItemId: item.dataset_item_id,
  evaluateTaskResult: item.evaluate_task_result,
  feedbackScores: item.feedback_scores.map(score => ({
    ...score,
    source: "sdk"
  }))
}))
});
console.log(`Experiment '${experimentName}' created successfully!`);

Advanced Usage

Including Traces and Spans

You can include full execution traces with your experiment items for complete observability, to do achieve this, add a trace and spans field to your experiment items:

[
  {
    "dataset_item_id": "your-dataset-item-id",
    "trace": {
      "name": "geography_query",
      "input": { "question": "What is the capital of France?" },
      "output": { "answer": "Paris" },
      "metadata": { "model": "gpt-3.5-turbo" },
      "start_time": "2024-01-01T00:00:00Z",
      "end_time": "2024-01-01T00:00:01Z"
    },
    "spans": [
      {
        "name": "llm_call",
        "type": "llm",
        "start_time": "2024-01-01T00:00:00Z",
        "end_time": "2024-01-01T00:00:01Z",
        "input": { "prompt": "What is the capital of France?" },
        "output": { "response": "Paris" }
      }
    ],
    "feedback_scores": [{ "name": "accuracy", "value": 1.0, "source": "sdk" }]
  }
]
Important: You may supply either evaluate_task_result or trace — not both.

Java Example

For Java developers, here’s how to integrate with Opik using Jackson and HttpClient:

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.fasterxml.jackson.databind.node.ArrayNode;
public class OpikExperimentLogger {
    public static void main(String[] args) {
        ObjectMapper mapper = new ObjectMapper();
        String baseURI = System.getenv("OPIK_URL_OVERRIDE");
        String workspaceName = System.getenv("OPIK_WORKSPACE");
        String apiKey = System.getenv("OPIK_API_KEY");
        String datasetName = "geography-questions";
        String experimentName = "geography-bot-v1";
        try (var client = HttpClient.newHttpClient()) {
            // Stream dataset items
            var streamRequest = HttpRequest.newBuilder()
                    .uri(URI.create(baseURI).resolve("/v1/private/datasets/items/stream"))
                    .header("Content-Type", "application/json")
                    .header("Accept", "application/octet-stream")
                    .header("Authorization", apiKey)
                    .header("Comet-Workspace", workspaceName)
                    .POST(HttpRequest.BodyPublishers.ofString(
                        mapper.writeValueAsString(Map.of("dataset_name", datasetName))
                    ))
                    .build();
            HttpResponse<InputStream> streamResponse = client.send(
                streamRequest,
                HttpResponse.BodyHandlers.ofInputStream()
            );
            List<JsonNode> experimentItems = new ArrayList<>();
            try (var reader = new BufferedReader(new InputStreamReader(streamResponse.body()))) {
                String line;
                while ((line = reader.readLine()) != null) {
                    JsonNode datasetItem = mapper.readTree(line);
                    String question = datasetItem.get("data").get("user_question").asText();
                    UUID datasetItemId = UUID.fromString(datasetItem.get("id").asText());
                    // Call your LLM application
                    JsonNode llmOutput = callYourLLM(question);
                    // Calculate metrics
                    List<JsonNode> scores = calculateMetrics(llmOutput);
                    // Build experiment item
                    ArrayNode scoresArray = JsonNodeFactory.instance.arrayNode().addAll(scores);
                    JsonNode experimentItem = JsonNodeFactory.instance.objectNode()
                            .put("dataset_item_id", datasetItemId.toString())
                            .setAll(Map.of(
                                "evaluate_task_result", llmOutput,
                                "feedback_scores", scoresArray
                            ));
                    experimentItems.add(experimentItem);
                }
            }
            // Send experiment results in bulk
            var bulkBody = JsonNodeFactory.instance.objectNode()
                    .put("dataset_name", datasetName)
                    .put("experiment_name", experimentName)
                    .setAll(Map.of("items",
                        JsonNodeFactory.instance.arrayNode().addAll(experimentItems)
                    ));
            var bulkRequest = HttpRequest.newBuilder()
                    .uri(URI.create(baseURI).resolve("/v1/private/experiments/items/bulk"))
                    .header("Content-Type", "application/json")
                    .header("Authorization", apiKey)
                    .header("Comet-Workspace", workspaceName)
                    .PUT(HttpRequest.BodyPublishers.ofString(bulkBody.toString()))
                    .build();
            HttpResponse<String> bulkResponse = client.send(
                bulkRequest,
                HttpResponse.BodyHandlers.ofString()
            );
            if (bulkResponse.statusCode() == 204) {
                System.out.println("Experiment items successfully created.");
            } else {
                System.err.printf("Failed to create experiment items: %s %s",
                    bulkResponse.statusCode(), bulkResponse.body());
            }
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}

Using the REST API with local deployments

If you are using the REST API with a local deployment, you can all the endpoints using:

# No authentication headers required for local deployments
curl -X PUT 'http://localhost:5173/api/v1/private/experiments/items/bulk' \
  -H 'Content-Type: application/json' \
  -d '{ ... }'

Reference