Experiment¶

class opik.api_objects.experiment.experiment.Experiment(id: str, name: str | None, dataset_name: str, rest_client: OpikApi, streamer: Streamer, experiments_client: ExperimentsClient, prompts: List[BasePrompt] | None = None, tags: List[str] | None = None, project_name: str | None = None)¶

Bases: object

__init__(id: str, name: str | None, dataset_name: str, rest_client: OpikApi, streamer: Streamer, experiments_client: ExperimentsClient, prompts: List[BasePrompt] | None = None, tags: List[str] | None = None, project_name: str | None = None) → None¶

property project_name: str | None¶

property id: str¶

property dataset_name: str¶

property name: str¶

property tags: List[str] | None¶

property prompts: List[BasePrompt] | None¶

property dataset_id: str¶

property experiments_rest_client: ExperimentsClient¶

get_experiment_data() → ExperimentPublic¶

insert(experiment_items_references: List[ExperimentItemReferences]) → None¶

Creates a new experiment item by linking the existing trace and dataset item.

Parameters:: experiment_items_references – The list of ExperimentItemReferences objects, containing trace id and dataset item id to link together into experiment item.
Returns:: None

batch_upload_items(items: Iterable[ExperimentItemBulkRecord], project_name: str | None = None, num_threads: int = constants.EXPERIMENT_ITEMS_BULK_NUM_THREADS, validate_before_upload: bool = True) → None¶

Upload experiment items together with their traces, spans and feedback scores.

Unlike insert(), which only links already-existing traces to dataset items, this method creates the traces and spans as part of the same request.

Items are split into batches that respect the backend’s 1000-item and 4MB-per-request limits, and sent with automatic retry on rate limiting (HTTP 429).

Any iterable is accepted, including a generator, and a single-pass source is consumed lazily: no item is retained once its batch has been sent, and the batches in flight are bounded, so peak memory follows the batch size rather than the upload. Passing a list instead holds the whole upload resident before the first request goes out, which for large records is the dominant cost – 286 MiB of records cost 8 MiB from a generator and 306 MiB from a list.

By default every item is validated before the first batch is sent. That needs a second pass over the items, so it applies only to a re-iterable source: from a list nothing is sent until every item has passed, and a bad one raises with nothing delivered. A generator cannot be walked twice, so each item is validated as it is reached instead – which is what validate_before_upload=False asks for explicitly – and a later invalid item is found with earlier batches already delivered.

A batch is built from the byte count of the records it will send, but a deployment may cap a request below the SDK’s own cap, so one can still be rejected as too large. A rejected batch is halved and retried, down to a single item, which raises rather than being split further. The halves are sent in order, so a batch that fails this way can leave some of its own items delivered; a half re-sends the bytes already serialised, so it carries the same ids and duplicates nothing.

If a batch fails the exception propagates and the experiment is left partially populated, but what “remaining” means depends on the worker count. With num_threads=1 nothing after the failed batch is sent. With the parallel default, batches not yet started are dropped and those already started are told to stop: one waiting out a rate limit or a retry gives up without sending again, but a request already on the wire cannot be interrupted, so a few batches after the failed one may still land. The call waits up to a few seconds for started batches to stop, then returns and logs a warning for any still running in the background. Rate-limit retries re-send the identical payload, so they never duplicate anything. Calling this method again, however, mints new ids for any trace or span left without one, which would duplicate whatever the first call did manage to write — set id on the traces and spans you pass in if you intend to retry a failed upload.

Parameters:

items – The experiment items to upload. Each item must provide exactly one of evaluate_task_result or trace.
project_name – Project for traces auto-created from items that provide evaluate_task_result. Defaults to the experiment’s project; blank is treated as unset. When set, every item-level trace.project_name must match it.
num_threads – Number of batches to upload concurrently. Defaults to 8; pass 1 to upload sequentially, which is the only way to guarantee batches arrive in order. Capped at constants.EXPERIMENT_ITEMS_BULK_MAX_THREADS, and at the number of batches where that is known before the upload starts – from a single-pass source it is not, so the pool is left at num_threads.
validate_before_upload – Whether every item is checked before the upload starts, rather than as it goes. Every item is validated either way, so this decides when a bad one is reported, not whether it is. True (the default) walks the list once first, so a bad item – failing validation, or too large to fit a request on its own – raises before any request is sent and all failures are reported together, at the cost of converting each item twice. False uploads in a single pass and checks each item as it is sent, so a bad one raises when it is reached, with the batches before it already delivered and no rollback.

Returns:

None

Raises:

opik.exceptions.ValidationError – If any item fails validation, if a single item is too large to fit in one request, if num_threads is less than 1, or if validate_before_upload is not a bool.

get_items(max_results: int | None = 10000, truncate: bool = False, *, page_size: int = constants.EXPERIMENT_ITEMS_READ_PAGE_SIZE, num_threads: int = constants.DATASET_ITEMS_READ_NUM_THREADS) → List[ExperimentItemContent]¶

Retrieves and returns a list of experiment items for this experiment.

Parameters:

max_results – Maximum number of experiment items to retrieve. Defaults to 10000 if not specified.
truncate – Whether to truncate the items returned by the backend. Defaults to False.
page_size – Number of dataset items requested per page. Must be a positive integer not exceeding constants.EXPERIMENT_ITEMS_READ_MAX_PAGE_SIZE. The read is round-trip bound, so this mostly trades request count against per-request size; lower it only if the backend struggles with the default response size.
num_threads – Number of pages fetched concurrently after the first one, which is read on its own to learn how many pages there are. Must be a positive integer not exceeding constants.DATASET_ITEMS_READ_MAX_THREADS. Pass 1 to read sequentially.

Returns:

List of ExperimentItemContent objects for this experiment.

Raises:

ValueError – If page_size is not a positive integer or exceeds constants.EXPERIMENT_ITEMS_READ_MAX_PAGE_SIZE, or if num_threads is not a positive integer or exceeds constants.DATASET_ITEMS_READ_MAX_THREADS.

Note

To look at an experiment’s results without writing code, AI coding assistants connected to the Opik MCP server can call list("dataset_item", experiment_ids=[...]), which shows each case’s scores and compares several experiments case by case. See https://www.comet.com/docs/opik/mcp-server

log_experiment_scores(score_results: List[score_result.ScoreResult]) → None¶: Log experiment-level scores to the backend.