Manage datasets

Datasets can be used to track test cases you would like to evaluate your LLM on. Each dataset is made up of a dictionary with any key value pairs. When getting started, we recommend having an input and optional expected_output fields for example. These datasets can be created from:

• Python SDK: You can use the Python SDK to create a dataset and add items to it.
• TypeScript SDK: You can use the TypeScript SDK to create a dataset and add items to it.
• Traces table: You can add existing logged traces (from a production application for example) to a dataset.
• The Opik UI: You can manually create a dataset and add items to it.

Once a dataset has been created, you can run Experiments on it. Each Experiment will evaluate an LLM application based on the test cases in the dataset using an evaluation metric and report the results back to the dataset.

Create a dataset via the UI

The simplest and fastest way to create a dataset is directly in the Opik UI. This is ideal for quickly bootstrapping datasets from CSV files without needing to write any code.

Steps:

1. Navigate to Evaluation > Datasets in the Opik UI.
2. Click Create new dataset.
3. In the pop-up modal:
   • Provide a name and an optional description
   • Optionally, upload a CSV file with your data
4. Click Create dataset.

If you need to create a dataset with more than 1,000 rows, you can use the SDK.

Understanding dataset versioning

Dataset versioning in Opik creates immutable snapshots of your data. Every time you modify a dataset—whether adding, editing, or deleting items—a new version is automatically created. This ensures complete reproducibility, provides an audit trail of all changes, and allows easy rollback to any previous state.

Each dataset version contains:

• Version name: Auto-generated sequential name (v1, v2, v3, etc.)
• Change description: Optional note describing what changed
• Tags: Labels for categorizing versions (e.g., production, baseline)
• Item statistics: Count of items added, modified, and deleted
• Timestamp and author: When the version was created and by whom

Once a version is created, its data cannot be changed—any modification creates a new version instead. Restoring a previous version also creates a new version with the same data, preserving your complete version timeline.

Working with draft mode (UI)

When making changes to a dataset in the Opik UI, all modifications go into a draft state first. This gives you a staging area to review changes before committing them as a new version. The draft is visible only to you, and AI-generated samples from “Expand with AI” also go to draft for review.

When a dataset has unsaved draft changes, an orange “Draft” tag appears next to the dataset name, and Save changes / Discard changes buttons appear in the toolbar. Items show colored borders: green for newly added items, amber for modified items.

Saving or discarding changes

To commit your draft as a new version:

1. Click Save changes in the toolbar
2. Enter a version note describing what changed
3. Optionally add tags to categorize this version
4. Click Save

To abandon your draft, click Discard changes and confirm. If you try to navigate away with unsaved changes, Opik displays a warning to prevent accidental loss of work.

Version history

To view the complete timeline of dataset changes, navigate to your dataset and click the Version history tab. The table shows each version’s name, change summary (items added/modified/deleted), version note, tags, item count, and creation timestamp.

From this view you can:

• View items: Click a version row and select View items to see the exact data at that point in time
• Restore: Click the ⋮ menu and select Restore this version to create a new version with that data
• Edit metadata: Click the ⋮ menu and select Edit to update the version note or tags (the data itself remains immutable)

Adding traces to a dataset

One of the most powerful ways to build evaluation datasets is by converting production traces into dataset items. This allows you to leverage real-world interactions from your LLM application to create test cases for evaluation.

Adding traces via the UI

To add traces to a dataset from the Opik UI:

1. Navigate to the traces page
2. Select one or more traces you want to add to a dataset
3. Click the Add to dataset button in the toolbar
4. In the dialog that appears:
   • Select an existing dataset or create a new one
   • Choose which trace metadata to include:
     • Nested spans: Include all child spans within the trace
     • Tags: Include trace tags
     • Feedback scores: Include any feedback scores attached to the trace
     • Comments: Include comments added to the trace
     • Usage metrics: Include token usage and cost information
     • Metadata: Include custom metadata fields
5. Click on the dataset name to add the selected traces

What gets added to the dataset

When you add a trace to a dataset, the following structure is created:

• input: The trace’s input data
• expected_output: The trace’s output data (stored as expected_output for evaluation purposes)
• spans (optional): Array of nested spans with their inputs, outputs, and metadata
• tags (optional): Array of tags associated with the trace
• feedback_scores (optional): Array of feedback scores with name, value, and source
• comments (optional): Array of comments with text and ID
• usage (optional): Token usage and cost information
• metadata (optional): Custom metadata fields

This rich structure allows you to:

• Evaluate complex multi-step workflows by including nested spans
• Filter and analyze based on tags and metadata
• Use existing feedback scores as ground truth for evaluation
• Preserve context through comments and annotations

Creating a dataset using the SDK

You can create a dataset and log items to it using the get_or_create_dataset method:

1
import { Opik } from "opik";
2
3
// Create a dataset
4
const client = new Opik();
5
const dataset = await client.getOrCreateDataset("My dataset");

If a dataset with the given name already exists, the existing dataset will be returned.

Insert items

Inserting dictionary items

You can insert items to a dataset using the insert method:

1
import { Opik } from "opik";
2
const client = new Opik();
3
const dataset = await client.getOrCreateDataset("My dataset");
4
5
dataset.insert([
6
  { user_question: "Hello, world!", expected_output: { assistant_answer: "Hello, world!" } },
7
  { user_question: "What is the capital of France?", expected_output: { assistant_answer: "Paris" } },
8
]);

Once the items have been inserted, you can view them in the Opik UI:

Inserting items from a JSONL file

You can also insert items from a JSONL file:

1
import opik
2
3
client = opik.Opik()
4
dataset = client.get_or_create_dataset(name="My dataset")
5
6
dataset.read_jsonl_from_file("path/to/file.jsonl")

Inserting items from a Pandas DataFrame

You can also insert items from a Pandas DataFrame:

1
import opik
2
3
client = opik.Opik()
4
dataset = client.get_or_create_dataset(name="My dataset")
5
6
dataset.insert_from_pandas(dataframe=df)
7
8
# You can also specify an optional keys_mapping parameter
9
dataset.insert_from_pandas(dataframe=df, keys_mapping={"Expected output": "expected_output"})

Deleting items

You can delete items in a dataset by using the delete method:

1
import { Opik } from "opik";
2
3
// Get or create a dataset
4
client = new Opik();
5
dataset = await client.getDataset("My dataset")
6
7
await dataset.delete(["123", "456"])
8
9
// Or to delete all items
10
await dataset.clear()

Downloading a dataset from Opik

You can download a dataset from Opik using the get_dataset method:

1
import { Opik } from "opik";
2
3
const client = new Opik();
4
const dataset = await client.getDataset("My dataset");
5
6
const items = await dataset.getItems();
7
console.log(items);

Filtering datasets programmatically

You can filter dataset items using the filter_string parameter on the get_items() method or when running evaluations with evaluate_prompt(). This allows you to work with specific subsets of your data.

Basic filtering

1
from opik import Opik
2
3
client = Opik()
4
dataset = client.get_dataset(name="my_dataset")
5
6
# Get filtered items
7
failed_items = dataset.get_items(filter_string='tags contains "failed"')

Filter syntax

The filter string uses Opik Query Language (OQL) syntax. Supported columns include:

Filter examples

1
from opik import Opik
2
3
client = Opik()
4
dataset = client.get_dataset(name="my_dataset")
5
6
# Filter by tag
7
failed_items = dataset.get_items(filter_string='tags contains "failed"')
8
9
# Filter by data field
10
finance_items = dataset.get_items(filter_string='data.category = "finance"')
11
12
# Filter by date
13
recent_items = dataset.get_items(
14
    filter_string='created_at >= "2024-06-01T00:00:00Z"'
15
)
16
17
# Multiple conditions
18
filtered_items = dataset.get_items(
19
    filter_string='tags contains "production" AND data.difficulty = "hard"'
20
)

Running experiments with dataset versions

When you run an experiment, Opik automatically links it to the specific dataset version that was used. This ensures complete reproducibility—you can always know exactly which data was used for any experiment.

Automatic version association

Every experiment records which dataset version it used:

• When running from the UI or SDK without specifying a version, the latest version is used
• The experiment results page shows the associated dataset version
• You can click the version to see the exact data that was evaluated

This association is permanent. Even if you later modify the dataset, your experiment results remain linked to the original version used.

Selecting a specific version in Playground

When running experiments from the Playground:

1. Open the Playground and configure your prompt
2. In the dataset selector, choose your dataset
3. A nested dropdown appears showing available versions
4. Select the specific version you want to use, or choose latest for the most recent

Selecting a specific version in the SDK

When running experiments programmatically, you can specify which dataset version to use by passing a DatasetVersion object to evaluate():

1
from opik import Opik
2
from opik.evaluation import evaluate
3
4
client = Opik()
5
dataset = client.get_dataset(name="My dataset")
6
7
# Run experiment on the latest version (default behavior)
8
result = evaluate(
9
    experiment_name="baseline-experiment",
10
    dataset=dataset,
11
    task=my_task_function,
12
    scoring_metrics=[my_metric],
13
)
14
15
# Run experiment on a specific version
16
v1_view = dataset.get_version_view("v1")
17
result = evaluate(
18
    experiment_name="v1-experiment",
19
    dataset=v1_view,  # Pass the DatasetVersion object
20
    task=my_task_function,
21
    scoring_metrics=[my_metric],
22
)

Working with dataset versions programmatically

The Python SDK provides methods for inspecting and working with dataset versions:

1
from opik import Opik
2
3
client = Opik()
4
dataset = client.get_dataset(name="My dataset")
5
6
# Get the current (latest) version name
7
current_version = dataset.get_current_version_name()
8
print(f"Current version: {current_version}")  # e.g., "v3"
9
10
# Get detailed version info (returns DatasetVersionPublic)
11
version_info = dataset.get_version_info()
12
print(f"Version ID: {version_info.id}")
13
print(f"Version name: {version_info.version_name}")
14
print(f"Items total: {version_info.items_total}")
15
print(f"Created at: {version_info.created_at}")
16
17
# Get a read-only view of a specific version
18
v1_view = dataset.get_version_view("v1")
19
20
# Access version metadata
21
print(f"Version: {v1_view.version_name}")
22
print(f"Items in v1: {v1_view.items_total}")
23
print(f"Items added: {v1_view.items_added}")
24
print(f"Items modified: {v1_view.items_modified}")
25
print(f"Items deleted: {v1_view.items_deleted}")
26
27
# Get items from a specific version
28
v1_items = v1_view.get_items()
29
30
# Export version data
31
v1_df = v1_view.to_pandas()
32
v1_json = v1_view.to_json()

Expanding a dataset with AI

Dataset expansion allows you to use AI to generate additional synthetic samples based on your existing dataset. This is particularly useful when you have a small dataset and want to create more diverse test cases to improve your evaluation coverage.

The AI analyzes the patterns in your existing data and generates new samples that follow similar structures while introducing variations. This helps you:

• Increase dataset size for more comprehensive evaluation
• Create edge cases and variations you might not have considered
• Improve model robustness by testing against diverse inputs
• Scale your evaluation without manual data creation

How to expand a dataset

To expand a dataset with AI:

1. Navigate to your dataset in the Opik UI (Evaluation > Datasets > [Your Dataset])
2. Click the “Expand with AI” button in the dataset view
3. Configure the expansion settings:
   • Model: Choose the LLM model to use for generation (supports GPT-4, GPT-5, Claude, and other models)
   • Sample Count: Specify how many new samples to generate (1-100)
   • Preserve Fields: Select which fields from your original data to keep unchanged
   • Variation Instructions: Provide specific guidance on how to vary the data (e.g., “Create variations that test edge cases” or “Generate examples with different complexity levels”)
   • Custom Prompt: Optionally provide a custom prompt template instead of the auto-generated one
4. Start the expansion - The AI will analyze your data and generate new samples
5. Review the results - Generated samples are added to your draft. You can review, edit, or remove them before saving to create a new version

Configuration options

Sample Count: Start with a smaller number (10-20) to review the quality before generating larger batches.

Preserve Fields: Use this to maintain consistency in certain fields while allowing variation in others. For example, preserve the category field while varying the input and expected_output.

Variation Instructions: Provide specific guidance such as:

• “Create variations with different difficulty levels”
• “Generate edge cases and error scenarios”
• “Add examples with different input formats”
• “Include multilingual variations”

Best practices

• Start small: Generate 10-20 samples first to evaluate quality before scaling up
• Review generated content: Always review AI-generated samples for accuracy and relevance
• Use variation instructions: Provide clear guidance on the type of variations you want
• Preserve key fields: Use field preservation to maintain important categorizations or metadata
• Iterate and refine: Use the custom prompt option to fine-tune generation for your specific needs

Managing dataset item tags

Tags are a powerful way to organize, categorize, and filter your dataset items. You can use tags to:

• Categorize test cases by type, difficulty, or domain (e.g., edge-case, production, multilingual)
• Track data sources where items originated from (e.g., user-feedback, synthetic, real-world)
• Mark review status during dataset curation (e.g., needs-review, validated, archived)
• Filter for evaluation to run experiments on specific subsets of your data
• Organize workflows by marking items for different stages or teams

Each dataset item can have multiple tags.

Adding tags to dataset items

Adding tags to individual items

To add tags to a single dataset item:

1. Navigate to your dataset in the Opik UI (Evaluation > Datasets > [Your Dataset])
2. Click on any dataset item to open the details panel
3. In the Tags section, click the ”+” button
4. Type the tag name and press Enter
5. The tag will be immediately added and saved

You can remove tags by clicking the ”×” icon next to any tag in the details panel.

Adding tags to multiple items (batch operation)

To add the same tag to multiple dataset items at once:

1. Navigate to your dataset in the Opik UI
2. Select multiple items by clicking the checkboxes next to each item
3. Click the “Add tags” button in the toolbar (visible when items are selected)
4. Enter the tag name in the dialog that appears
5. Click “Add tag” to apply the tag to all selected items

This is particularly useful when you want to categorize a group of related test cases or mark items from the same data source.

Filtering dataset items by tags

Once you’ve tagged your dataset items, you can filter them to work with specific subsets:

1. Navigate to your dataset in the Opik UI
2. Click the “Filters” button next to the search bar
3. Select “Tags” from the Column dropdown
4. Choose “contains” as the operator
5. Enter the tag name you want to filter by
6. Close the dialog to apply the filter

The dataset items table will update to show only items matching your filter criteria. You can:

• View filtered items to focus on specific categories
• Run experiments on filtered subsets by using the filtered view
• Export filtered data for specific test case groups
• Combine with other filters to create complex queries

The filter is saved in the URL, so you can bookmark or share specific filtered views of your dataset.

Bulk operations

Opik supports bulk operations for efficiently managing large datasets. These operations help you work with many items at once without tedious individual selections.

Select all functionality

When working with datasets that span multiple pages:

1. Select items on the current page using the checkbox in the table header
2. A banner appears offering to “Select all items” across all pages
3. Click to select all items matching your current filter criteria

This works with filtered views too—if you have a filter applied, “Select all” only selects items matching that filter.

Available bulk operations

Once you have items selected, the toolbar shows available operations:

• Add tags: Apply one or more tags to all selected items
• Delete: Remove selected items (creates a new version with items removed)
• Export: Download selected items as CSV or JSON

Processing indicators

For large bulk operations:

• A loading indicator shows “Your dataset is still processing…”
• The operation runs in the background—you can continue browsing
• A success message appears when processing completes