> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://www.comet.com/docs/opik/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://www.comet.com/docs/opik/_mcp/server.

# Getting started with Evaluation

Opik provides two approaches to evaluation. Choose the one that fits your use case:

* **Test Suites**: Define assertions in natural language and let an LLM judge test them. Best for pass/fail behavioral testing.
* **Datasets & Metrics**: Score outputs against a dataset using quantitative metrics. Best for measuring quality across many traces.

## Quick start

#### Test Suites

Test Suites let you define expected behaviors as natural-language assertions and run them
against your agent. An LLM judge checks each assertion automatically.

**`Python`**

```python title="Python"
import opik
from openai import OpenAI
from opik.integrations.openai import track_openai

openai_client = track_openai(OpenAI())
opik_client = opik.Opik()

# Create a suite with assertions
suite = opik_client.get_or_create_test_suite(
    name="my-agent-tests",
    project_name="my-agent",
    global_assertions=[
        "The response directly addresses the user's question",
        "The response is concise (3 sentences or fewer)",
    ],
    global_execution_policy={"runs_per_item": 2, "pass_threshold": 2},
)

# Add test cases
suite.insert([
    {"data": {"question": "How do I create a new project?", "context": "Go to Dashboard and click 'New Project'."}},
    {"data": {"question": "What are the pricing tiers?", "context": "Free ($0/month), Pro ($29/month), Enterprise (custom)."}},
])

# Define the task
def task(item):
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer based ONLY on the provided context."},
            {"role": "user", "content": f"Question: {item['question']}\n\nContext:\n{item['context']}"},
        ],
    )
    return {"input": item, "output": response.choices[0].message.content}

# Run the evaluation
result = opik.run_tests(test_suite=suite, task=task)
print(f"Pass rate: {result.pass_rate:.0%}")
```

**`Typescript`**

```ts title="Typescript"
import { Opik, TestSuite, runTests } from "opik";
import OpenAI from "openai";

const client = new Opik();
const openai = new OpenAI();

// Create a suite with assertions
const suite = await TestSuite.getOrCreate(client, {
    name: "my-agent-tests",
    projectName: "my-agent",
    globalAssertions: [
        "The response directly addresses the user's question",
        "The response is concise (3 sentences or fewer)",
    ],
    globalExecutionPolicy: { runsPerItem: 2, passThreshold: 2 },
});

// Add test cases
await suite.insert([
    { data: { question: "How do I create a new project?", context: "Go to Dashboard and click 'New Project'." } },
    { data: { question: "What are the pricing tiers?", context: "Free ($0/month), Pro ($29/month), Enterprise (custom)." } },
]);

// Define the task
const task = async (item: Record<string, string>) => {
    const response = await openai.chat.completions.create({
        model: "gpt-4o-mini",
        messages: [
            { role: "system", content: "Answer based ONLY on the provided context." },
            { role: "user", content: `Question: ${item.question}\n\nContext:\n${item.context}` },
        ],
    });
    return { input: item, output: response.choices[0].message.content };
};

// Run the evaluation
const result = await runTests({ testSuite: suite, task });
console.log(`Pass rate: ${((result.passRate ?? 0) * 100).toFixed(0)}%`);
```

Each run creates an experiment in the Opik dashboard for easy comparison.

![Test suite experiment results showing pass/fail per item with assertion details](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/opik.docs.buildwithfern.com/61092f479035e393df1d1868fb5503f2285aa6cd4896da09a1bdda89a71cfed0/img/v2/evaluation/test-suite-run-results.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260912T200029Z&X-Amz-Expires=604800&X-Amz-Signature=937c6e98d111fb089ac9b83fc1e542ccae9caa53f5090342fca79c6d046c2c4e&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject)

See the [Building Test Suites](/evaluation/advanced/building-test-suites) guide for the full walkthrough.

#### Datasets & Metrics

Dataset-based evaluation scores your agent's outputs using quantitative metrics like
hallucination detection, answer relevance, or custom scoring functions.

**`Python`**

```python title="Python"
import opik
from opik.evaluation import evaluate
from opik.evaluation.metrics import Hallucination

opik.configure()
client = opik.Opik()

# Create a dataset
dataset = client.get_or_create_dataset(name="my-eval-dataset")
dataset.insert([
    {"input": "What is the capital of France?", "expected_output": "Paris"},
    {"input": "What is 2+2?", "expected_output": "4"},
])

# Define the task
def task(item):
    # Your LLM call here
    result = call_llm(item["input"])
    return {"output": result}

# Run evaluation with metrics
evaluate(
    dataset=dataset,
    task=task,
    scoring_metrics=[Hallucination()],
    experiment_name="my-experiment-v1",
)
```

**`Typescript`**

```ts title="Typescript"
import { Opik } from "opik";

const client = new Opik();

// Create a dataset
const dataset = await client.getOrCreateDataset({ name: "my-eval-dataset" });
await dataset.insert([
    { input: "What is the capital of France?", expectedOutput: "Paris" },
    { input: "What is 2+2?", expectedOutput: "4" },
]);

// Run evaluation with metrics
await client.evaluate({
    dataset,
    task: async (item) => {
        const result = await callLlm(item.input);
        return { output: result };
    },
    experimentName: "my-experiment-v1",
});
```

See the [Datasets & Experiments](/evaluation/advanced/evaluate_your_llm) guide for the full walkthrough
and the [Metrics](/evaluation/metrics/overview) section for all available metrics.

**Recommended if you build with an AI coding assistant.** Either approach can be run by your
assistant rather than by you. One command — `opik configure` — installs both the
[MCP server](/mcp-server) and the Opik skills, and evaluation becomes part of its development
loop: it changes the code, runs the suite or the evaluation, reads the scores, and iterates.

An example prompt:

*"Set up an Opik evaluation for this agent, then improve the agent and re-run the evaluation after
each change, showing me the scores each time."*