TestSuite

class opik.TestSuite(name: str, dataset_: dataset.Dataset, client: 'opik_client_module.Opik' | None = None)

Bases: object

A pre-configured regression test suite for LLM applications.

Test Suites let you: - Define test cases with inputs and context - Configure assertions that will be checked by an LLM - Run tests against any task function

Suite-level assertions and execution policy are stored in the dataset’s metadata and read by the evaluation engine when running the suite.

Example

>>> import opik
>>>
>>> client = opik.Opik()
>>>
>>> suite = client.create_test_suite(
...     name="Refund Policy Tests",
...     description="Regression tests for refund scenarios",
...     global_assertions=[
...         "Response does not contain hallucinated information",
...         "Response is helpful to the user",
...     ],
... )
>>>
>>> suite.insert([
...     {
...         "data": {"user_input": "How do I get a refund?", "user_tier": "premium"},
...         "assertions": ["Response is polite"],
...     },
... ])
>>>
>>> results = opik.run_tests(test_suite=suite, task=my_llm_function)

__init__(name: str, dataset_: dataset.Dataset, client: 'opik_client_module.Opik' | None = None)

Internal constructor — not part of the public API.

Use opik.Opik.create_test_suite() or opik.Opik.get_or_create_test_suite() instead.

property id: str

The ID of the test suite.

property name: str

The name of the test suite.

property description: str | None

The description of the test suite.

property project_name: str | None

The project name associated with the test suite.

property items_count: int | None

The total number of items in the test suite.

get_tags() → List[str]

Get the tags for the suite.

Returns:

List of tag strings.

get_current_version_name() → str | None

Get the current version name of the test suite.

Returns:

The current version name (e.g., ‘v1’, ‘v2’), or None if no version exists.

get_version_info() → DatasetVersionPublic | None

Get version information for the current (latest) version.

Returns:

DatasetVersionPublic containing the current version’s metadata, or None if no version exists yet.

get_version_view(version_name: str) → TestSuiteVersion

Get a read-only view of a specific version.

Parameters:

version_name – The version name (e.g., ‘v1’, ‘v2’).

Returns:

A read-only TestSuiteVersion for accessing the specified version’s items, assertions, and execution policy.

Raises:

opik.exceptions.DatasetVersionNotFound – If the version does not exist.

get_items(nb_samples: int | None = None, filter_string: str | None = None) → List[TestSuiteItem]

Retrieve suite items as a list of dictionaries.

Each item dict has keys: id, data, description, assertions, execution_policy.

Parameters:

• nb_samples – Maximum number of items to retrieve. If None, all items are returned.

• filter_string – Optional OQL filter string to filter items.

Returns:

A list of item dictionaries.

to_pandas() → pd.DataFrame

Convert the test suite items to a pandas DataFrame.

Requires the pandas library to be installed.

Returns:

A pandas DataFrame containing all items with columns such as id, data, assertions, description, and execution_policy.

to_json() → str

Convert the test suite items to a JSON string.

Returns:

A JSON string representation of all items.

insert_from_json(json_array: str, keys_mapping: Dict[str, str] | None = None, ignore_keys: List[str] | None = None) → None

Insert test suite items from a JSON string.

Each JSON object must map to valid test suite item keys after applying keys_mapping:

• data (required) — dict of test case inputs

• assertions — list of assertion strings

• description — item description

• execution_policy — dict with runs_per_item and pass_threshold

• id — item identifier (auto-generated if omitted)

Parameters:

• json_array – JSON string of format [{...}, {...}].

• keys_mapping – Maps JSON keys to the target keys listed above. Example: {"test_data": "data", "checks": "assertions"}

• ignore_keys – Keys in the JSON dicts to skip during import.

insert_from_pandas(dataframe: pd.DataFrame, keys_mapping: Dict[str, str] | None = None, ignore_keys: List[str] | None = None) → None

Insert test suite items from a pandas DataFrame.

Requires the pandas library to be installed.

Each DataFrame row must map to valid test suite item keys after applying keys_mapping:

• data (required) — dict of test case inputs

• assertions — list of assertion strings

• description — item description

• execution_policy — dict with runs_per_item and pass_threshold

• id — item identifier (auto-generated if omitted)

Parameters:

• dataframe – pandas DataFrame.

• keys_mapping – Maps column names to the target keys listed above. Example: {"test_data": "data", "checks": "assertions"}

• ignore_keys – Column names in the DataFrame to skip during import.

insert_from_jsonl_file(file_path: str, keys_mapping: Dict[str, str] | None = None, ignore_keys: List[str] | None = None) → None

Read JSONL from a file and insert items into the test suite.

Each line must be a JSON object that maps to valid test suite item keys after applying keys_mapping:

• data (required) — dict of test case inputs

• assertions — list of assertion strings

• description — item description

• execution_policy — dict with runs_per_item and pass_threshold

• id — item identifier (auto-generated if omitted)

Parameters:

• file_path – Path to the JSONL file.

• keys_mapping – Maps JSON keys to the target keys listed above. Example: {"test_data": "data", "checks": "assertions"}

• ignore_keys – Keys in the JSON objects to skip during import.

update_test_settings(*, global_execution_policy: ExecutionPolicy | None = None, global_assertions: List[str] | None = None) → None

Update the suite-level assertions and/or execution policy.

Supports partial updates: any parameter not provided will retain its current value. If the new values are identical to the current values, no new version is created.

Parameters:

• global_execution_policy – New execution policy for the suite. If not provided, the current policy is kept.

• global_assertions – New suite-level assertions. Each string describes an expected behavior that will be checked by an LLM. If not provided, the current assertions are kept.

Raises:

ValueError – If nothing to update is provided.

delete(items_ids: List[str]) → None

Delete items from the test suite by their IDs.

Parameters:

items_ids – List of item IDs to delete.

clear() → None

Delete all items from the test suite.

get_global_execution_policy() → ExecutionPolicy

Get the suite-level execution policy.

Returns:

ExecutionPolicy dict with runs_per_item and pass_threshold.

get_global_assertions() → List[str]

Get the suite-level assertions.

Returns:

List of assertion strings.

update(items: List[TestSuiteItem]) → None

Update existing items in the test suite.

Each item dict must include an "id" key identifying the item to update. The remaining keys ("data", "assertions", "description", "execution_policy") replace the previous values.

Parameters:

items – List of item dicts to update. Each must contain "id".

Raises:

DatasetItemUpdateOperationRequiresItemId – If any item is missing an "id" key.

insert(items: List[TestSuiteItem]) → None

Insert test cases into the test suite.

Parameters:

items – List of test case items to add.

Example

>>> suite.insert([
...     {"data": {"question": "How do I get a refund?"}},
...     {
...         "data": {"question": "Is my account hacked?"},
...         "assertions": ["Response treats the concern with urgency"],
...         "execution_policy": {"runs_per_item": 5, "pass_threshold": 4},
...     },
... ])