Test Suites

Test suites are pre-configured regression tests for your LLM application. They let you validate that prompt changes, model updates, or code modifications don’t break existing functionality — codifying real production failures as reusable test cases.

A test suite pairs:

• Items — a dataset of inputs (and any context they need), each with optional item-level assertions and execution policy overrides
• Global assertions — natural-language checks (evaluated by an LLM judge) applied to every item
• An execution policy — how many times to run each item, and how many of those runs must pass

For a walkthrough of building suites via Ollie, the UI, or the SDK, see Building Test Suites. This page is the SDK API reference.

Creating and Managing Test Suites

The TypeScript SDK provides several methods to create and manage test suites through the OpikClient class.

1
import { Opik } from "opik";
2
3
const opik = new Opik();
4
5
// Create a new test suite with suite-level assertions
6
const suite = await opik.createTestSuite({
7
  name: "customer-support-qa",
8
  description: "Regression tests for the support agent",
9
  projectName: "my-project",
10
  globalAssertions: [
11
    "The response is grounded in the provided documentation context",
12
    "The response directly addresses the user's question",
13
  ],
14
  globalExecutionPolicy: { runsPerItem: 2, passThreshold: 2 },
15
});
16
17
// Get an existing test suite, or create it if it doesn't exist
18
const suite2 = await opik.getOrCreateTestSuite({
19
  name: "customer-support-qa",
20
  projectName: "my-project",
21
});
22
23
// Get an existing test suite by name
24
const existing = await opik.getTestSuite("customer-support-qa", "my-project");
25
26
// List test suites
27
const suites = await opik.getTestSuites(100, "my-project");
28
29
// Delete a test suite
30
await opik.deleteTestSuite("customer-support-qa", "my-project");

Working with Test Suite Items

1
// Insert items — item-level assertions/execution policy are optional
2
// and add to (not replace) the suite-level ones
3
await suite.insert([
4
  {
5
    data: { question: "How do I reset my password?" },
6
  },
7
  {
8
    data: { question: "Can I use this with Kubernetes?" },
9
    assertions: ["The response does NOT claim Kubernetes is supported"],
10
    executionPolicy: { runsPerItem: 3, passThreshold: 2 },
11
  },
12
]);
13
14
// Update existing items (each item must include an `id`)
15
await suite.update([
16
  { id: "item-1", assertions: ["The response is under 3 sentences"] },
17
]);
18
19
// Retrieve items, with item-level and suite-level policy merged
20
const items = await suite.getItems();
21
22
// Retrieve items with the raw, unmerged stored payload — evaluators as
23
// EvaluatorItemWrite[] rather than decoded assertion strings, and the
24
// item-level executionPolicy only (not merged with the suite default)
25
const rawItems = await suite.getRawItems();
26
27
// Delete specific items
28
await suite.delete(["item-1", "item-2"]);
29
30
// Delete all items in the suite
31
await suite.clear();

Updating a single item’s assertions or execution policy

1
await suite.updateItemAssertions("item-1", [
2
  "The response cites a specific step from the provided context",
3
]);
4
5
await suite.updateItemExecutionPolicy("item-1", {
6
  runsPerItem: 5,
7
  passThreshold: 4,
8
});
9
10
// Or update both at once
11
await suite.updateItem("item-1", {
12
  assertions: ["The response is polite"],
13
  executionPolicy: { runsPerItem: 3, passThreshold: 2 },
14
});

Execution Policies

Execution policies control how many times each item is run and how many of those runs must pass — useful for handling non-deterministic LLM outputs.

• A run passes if all its assertions pass
• An item passes if runsPassed >= passThreshold
• The suite’s pass rate is the ratio of passed items to total items

1
// Read the suite-level defaults
2
const assertions = await suite.getGlobalAssertions();
3
const policy = await suite.getGlobalExecutionPolicy();
4
5
// Update suite-level assertions and/or execution policy
6
await suite.updateTestSettings({
7
  globalAssertions: [
8
    "The response is grounded in the provided context",
9
    "The response is concise",
10
  ],
11
  globalExecutionPolicy: { runsPerItem: 5, passThreshold: 3 },
12
});

Running a Test Suite

runTests is the primary entry point for executing a suite: it runs your task against every item, scores results using the suite’s configured assertions, and returns pass/fail results.

1
import { runTests } from "opik";
2
3
const result = await runTests({
4
  testSuite: suite,
5
  task: async (item) => ({
6
    input: item,
7
    output: await myLLM(item.question),
8
  }),
9
});
10
11
console.log(result.allItemsPassed, result.passRate);

Each run creates a separate experiment in Opik, so you can compare multiple runs (e.g. two prompt versions) against the same suite in the dashboard. evaluateTestSuite is the lower-level function runTests builds on, useful if you need the raw EvaluationResult rather than the suite-shaped TestSuiteResult.

Test Suite Versions

Like datasets, test suites are versioned. See Datasets for the general versioning model — the same DatasetVersion-style methods are available on TestSuite:

1
const currentVersion = await suite.getCurrentVersionName();
2
const versionInfo = await suite.getVersionInfo();
3
const v2 = await suite.getVersionView("v2");

API Reference

OpikClient Test Suite Methods

createTestSuite

Creates a new test suite.

Arguments:

• options: CreateTestSuiteOptions - { name, description?, globalAssertions?, globalExecutionPolicy?, tags?, projectName? }

Returns: Promise<TestSuite> - The created test suite

getTestSuite

Retrieves an existing test suite by name.

Arguments:

• name: string - The name of the test suite to retrieve
• projectName?: string - Optional project name to scope the lookup. If not provided, uses the client’s configured project.

Returns: Promise<TestSuite>

Throws: DatasetNotFoundError if the test suite doesn’t exist

getOrCreateTestSuite

Retrieves an existing test suite by name, or creates it if it doesn’t exist.

Arguments:

• options: CreateTestSuiteOptions - Same shape as createTestSuite

Returns: Promise<TestSuite> - The existing or newly created test suite

deleteTestSuite

Deletes a test suite by name.

Arguments:

• name: string - The name of the test suite to delete
• projectName?: string - Optional project name to scope the lookup. If not provided, uses the client’s configured project.

Returns: Promise<void>

getTestSuites

Returns all test suites up to the specified limit.

Arguments:

• maxResults?: number - Maximum number of test suites to return (default: 1000)
• projectName?: string - Optional project name to filter by. If not provided, uses the client’s configured project.

Returns: Promise<TestSuite[]>

getTestSuiteExperiments

Retrieves all experiments associated with a test suite.

Arguments:

• name: string - The name of the test suite
• maxResults?: number - Maximum number of experiments to return (default: 100)
• projectName?: string - Optional project name to scope the suite lookup. If not provided, uses the client’s configured project.

Returns: Promise<TestSuiteExperiment[]> - Each entry carries the suite-specific assertion aggregates (passRate, passedCount, totalCount, assertionScores) populated by the backend

Throws: DatasetNotFoundError if the test suite doesn’t exist

TestSuite Class Methods

insert

Inserts new items into the test suite.

Arguments:

• items: TestSuiteItem[] - { data, assertions?, description?, executionPolicy? }[]

Returns: Promise<void>

update

Updates existing items. Each item must include an id.

Arguments:

• items: UpdateTestSuiteItem[] - { id, data?, assertions?, description?, executionPolicy? }[]

Returns: Promise<void>

Throws: Error if any item is missing an id

getItems

Retrieves items with item-level assertions decoded and the item-level execution policy merged with the suite-level default.

Arguments:

• nbSamples?: number - Max items to retrieve (default: all)
• lastRetrievedId?: string - Opaque cursor for pagination

Returns: Promise<Array<{ id, data, description?, assertions, executionPolicy }>>

getRawItems

Retrieves items with the stored payload preserved verbatim — evaluators as raw EvaluatorItemWrite[] (not decoded to assertion strings) and executionPolicy as the item-level value only (not merged with the suite default). Use this when you need to inspect or forward the stored config as-is.

Arguments:

• nbSamples?: number - Max items to retrieve (default: all)
• lastRetrievedId?: string - Opaque cursor for pagination

Returns: Promise<RawTestSuiteItem[]>

delete

Deletes items from the test suite.

Arguments:

• itemIds: string[] - List of item IDs to delete

Returns: Promise<void>

clear

Deletes all items from the test suite.

Returns: Promise<void>

getGlobalAssertions

Returns: Promise<string[]> - The suite-level assertions

getGlobalExecutionPolicy

Returns: Promise<Required<ExecutionPolicy>> - The suite-level execution policy ({ runsPerItem, passThreshold })

updateTestSettings

Updates suite-level assertions and/or execution policy. Fields you omit retain their current value.

Arguments:

• options: UpdateTestSuiteOptions - { globalAssertions?, globalExecutionPolicy? } — at least one is required

Returns: Promise<void>

updateItem

Updates a single item’s assertions and/or execution policy.

Arguments:

• itemId: string
• options: { assertions?: string[]; executionPolicy?: ExecutionPolicy } - At least one is required

Returns: Promise<void>

updateItemAssertions

Shorthand for updateItem(itemId, { assertions }).

Arguments:

• itemId: string
• assertions: string[]

Returns: Promise<void>

updateItemExecutionPolicy

Shorthand for updateItem(itemId, { executionPolicy }).

Arguments:

• itemId: string
• executionPolicy: ExecutionPolicy

Returns: Promise<void>

getTags

Returns: Promise<string[]>

getItemsCount

Returns: Promise<number | undefined>

getCurrentVersionName

Returns: Promise<string | undefined> - The latest version name (e.g. "v1"), or undefined if no versions exist

getVersionInfo

Returns: Promise<DatasetVersionPublic | undefined>

getVersionView

Arguments:

• versionName: string - e.g. "v1", "v2"

Returns: Promise<DatasetVersion>

Throws: DatasetVersionNotFoundError if the version doesn’t exist

runTests

The primary entry point for running a test suite.

1
async function runTests(options: RunTestsOptions): Promise<TestSuiteResult>;

Returns: Promise<TestSuiteResult>

evaluateTestSuite

The lower-level function runTests is built on. Runs a test suite using the evaluators and execution policy stored in the suite’s dataset version metadata, and returns the raw EvaluationResult rather than a suite-shaped TestSuiteResult.

1
async function evaluateTestSuite(
2
  options: EvaluateTestSuiteOptions
3
): Promise<EvaluationResult>;

Arguments: same shape as evaluate, plus dataset (the suite’s dataset).

Returns: Promise<EvaluationResult>

Data Structures

TestSuiteResult

Result of running a test suite. Returned by runTests.

1
class TestSuiteResult {
2
  readonly allItemsPassed: boolean;
3
  readonly itemsPassed: number;
4
  readonly itemsTotal: number;
5
  readonly passRate: number | undefined;
6
  readonly itemResults: Map<string, ItemResult>;
7
  readonly experimentId: string;
8
  readonly experimentName?: string;
9
  readonly experimentUrl?: string;
10
  readonly suiteName?: string;
11
  readonly totalTime?: number;
12
13
  // Converts the result to a plain report object (camelCase keys), grouping
14
  // test results by trial and including per-item pass/fail and assertion detail
15
  toReportDict(): Record<string, unknown>;
16
17
  // Alias for toReportDict()
18
  toDict(): Record<string, unknown>;
19
}

ItemResult

Result for a single test suite item, keyed by datasetItemId in TestSuiteResult.itemResults.

1
type ItemResult = {
2
  datasetItemId: string;
3
  passed: boolean;
4
  // Whether this item had at least one assertion evaluated across any of its runs
5
  hasAssertions: boolean;
6
  runsPassed: number;
7
  runsTotal: number;
8
  // Configured runsPerItem from the execution policy
9
  configuredRunsPerItem: number;
10
  passThreshold: number;
11
  testResults: EvaluationTestResult[];
12
};

TestSuiteExperiment

Represents an experiment run against a test suite. Extends Experiment (see Experiments) with the aggregate assertion statistics the backend populates only for test suite experiments (undefined for regular dataset experiments).

1
class TestSuiteExperiment extends Experiment {
2
  readonly passRate?: number;
3
  readonly passedCount?: number;
4
  readonly totalCount?: number;
5
  readonly assertionScores?: AssertionScoreAveragePublic[];
6
}

CreateTestSuiteOptions

1
interface CreateTestSuiteOptions {
2
  name: string;
3
  description?: string;
4
  globalAssertions?: string[];
5
  globalExecutionPolicy?: ExecutionPolicy;
6
  tags?: string[];
7
  projectName?: string;
8
}

ExecutionPolicy

1
interface ExecutionPolicy {
2
  runsPerItem?: number;
3
  passThreshold?: number;
4
}