> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://www.comet.com/docs/opik/llms.txt.

# Configuring LLM Providers

> Learn how to configure different LLM providers like OpenAI, Anthropic, Gemini, Azure, and Ollama for use with the Opik Agent Optimizer.

> **Note**
>
> In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.

The Opik Agent Optimizer uses [LiteLLM](https://docs.litellm.ai/) under the hood, giving you access to 100+ LLM providers with a unified interface. This guide shows you how to configure different providers for both your **ChatPrompt** (the model that runs your prompt) and the **Optimizer** (the model that improves your prompt).

## Understanding the Two Model Types

When using the Opik Optimizer, there are two distinct models to configure:

| Model Type           | Purpose                                                         | Recommendation                                        |
| -------------------- | --------------------------------------------------------------- | ----------------------------------------------------- |
| **ChatPrompt model** | The model that executes your prompt during evaluation           | Use the same model as your production application     |
| **Optimizer model**  | The model that analyzes failures and generates improved prompts | Use the most capable model available for best results |

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

# ChatPrompt model - this is the model your prompt runs on
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"  # Your production model
)

# Optimizer model - this is the model that improves your prompt
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o"  # Use a powerful model for optimization
)
```

## LiteLLM Model Format

All models use the LiteLLM format: `provider/model-name`

```python
# Examples of the LiteLLM model format
model="openai/gpt-4o"                          # OpenAI
model="anthropic/claude-sonnet-4-6"            # Anthropic
model="gemini/gemini-2.0-flash"                # Google Gemini
model="azure/my-deployment-name"               # Azure OpenAI
model="ollama/llama3"                          # Ollama (local)
model="openrouter/google/gemini-2.0-flash"     # OpenRouter
```

## Provider Configuration

#### OpenAI

### OpenAI

**Environment Variable:**

```bash
export OPENAI_API_KEY="sk-..."
```

**Available Models:**

* `openai/gpt-4o` - Most capable model
* `openai/gpt-4o-mini` - Fast and cost-effective
* `openai/gpt-4-turbo` - Previous generation
* `openai/o1` - Reasoning model
* `openai/o3-mini` - Efficient reasoning model

**Example:**

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="openai/gpt-4o-mini"
)

optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
```

#### Anthropic

### Anthropic (Claude)

**Environment Variable:**

```bash
export ANTHROPIC_API_KEY="sk-ant-..."
```

**Available Models:**

* `anthropic/claude-sonnet-4-6` - Best balance of speed and capability
* `anthropic/claude-opus-4-7` - Most capable
* `anthropic/claude-haiku-4-5` - Fastest

**Example:**

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="anthropic/claude-sonnet-4-6"
)

optimizer = MetaPromptOptimizer(model="anthropic/claude-sonnet-4-6")
```

#### Google Gemini

### Google Gemini

**Environment Variable:**

```bash
export GOOGLE_API_KEY="..."
# or
export GEMINI_API_KEY="..."
```

> **Tip**
>
> Get your API key from [Google AI Studio](https://aistudio.google.com/apikey).

**Available Models:**

* `gemini/gemini-2.0-flash` - Latest fast model
* `gemini/gemini-1.5-pro` - Most capable
* `gemini/gemini-1.5-flash` - Fast and efficient
* `gemini/gemini-1.5-flash-8b` - Lightweight model

**Example:**

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"
)

optimizer = MetaPromptOptimizer(model="gemini/gemini-1.5-pro")
```

**Complete Gemini Example:**

```python
import os
from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio
import opik

# Set your API key
os.environ["GOOGLE_API_KEY"] = "your-api-key-here"

# Initialize Opik client and create dataset
client = opik.Opik()
dataset = client.get_or_create_dataset(name="gemini-optimization-demo", project_name="my-project")
dataset.insert([
    {"question": "What is machine learning?", "answer": "Machine learning is a subset of AI that enables systems to learn from data."},
    {"question": "What is Python?", "answer": "Python is a high-level programming language known for its readability."},
])

# Define metric
def answer_quality(dataset_item, llm_output):
    metric = LevenshteinRatio()
    return metric.score(reference=dataset_item["answer"], output=llm_output)

# Configure prompt with Gemini
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "Answer the question concisely."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"
)

# Configure optimizer (can use same or different provider)
optimizer = MetaPromptOptimizer(
    model="gemini/gemini-1.5-pro",
    n_threads=4
)

# Run optimization
result = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=answer_quality,
    max_trials=5,
    n_samples=2
)

result.display()
```

#### Azure OpenAI

### Azure OpenAI

**Environment Variables:**

```bash
export AZURE_API_KEY="..."
export AZURE_API_BASE="https://your-resource.openai.azure.com"
export AZURE_API_VERSION="2024-02-15-preview"
```

**Model Format:**

```python
# Use your deployment name
model="azure/your-deployment-name"
```

**Example:**

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="azure/gpt-4o-deployment"  # Your Azure deployment name
)

optimizer = MetaPromptOptimizer(model="azure/gpt-4o-deployment")
```

#### Ollama (Local)

### Ollama (Local Models)

**Setup:**

1. Install Ollama from [ollama.ai](https://ollama.ai)
2. Pull a model: `ollama pull llama3`
3. Ollama runs on `http://localhost:11434` by default

**Environment Variable (optional):**

```bash
export OLLAMA_API_BASE="http://localhost:11434"
```

**Available Models:**

* `ollama/llama3` - Meta's Llama 3
* `ollama/mistral` - Mistral 7B
* `ollama/codellama` - Code-focused Llama
* `ollama/phi3` - Microsoft Phi-3

**Example:**

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="ollama/llama3"
)

# Note: Local models may be slower for optimization
optimizer = MetaPromptOptimizer(model="ollama/llama3")
```

#### OpenRouter

### OpenRouter

OpenRouter provides access to multiple providers through a single API.

**Environment Variable:**

```bash
export OPENROUTER_API_KEY="sk-or-..."
```

**Model Format:**

```python
# Format: openrouter/provider/model
model="openrouter/google/gemini-2.0-flash"
model="openrouter/anthropic/claude-sonnet-4.6"
model="openrouter/meta-llama/llama-3-70b-instruct"
```

**Example:**

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="openrouter/google/gemini-2.0-flash"
)

optimizer = MetaPromptOptimizer(model="openrouter/anthropic/claude-sonnet-4.6")
```

## Environment Variables Reference

| Provider      | Environment Variable                                   | How to Get                                                           |
| ------------- | ------------------------------------------------------ | -------------------------------------------------------------------- |
| OpenAI        | `OPENAI_API_KEY`                                       | [platform.openai.com/api-keys](https://platform.openai.com/api-keys) |
| Anthropic     | `ANTHROPIC_API_KEY`                                    | [console.anthropic.com](https://console.anthropic.com/settings/keys) |
| Google Gemini | `GOOGLE_API_KEY` or `GEMINI_API_KEY`                   | [aistudio.google.com/apikey](https://aistudio.google.com/apikey)     |
| Azure OpenAI  | `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION` | Azure Portal                                                         |
| OpenRouter    | `OPENROUTER_API_KEY`                                   | [openrouter.ai/keys](https://openrouter.ai/keys)                     |
| Ollama        | None required (local)                                  | [ollama.ai](https://ollama.ai)                                       |

## Model Parameters

You can pass additional parameters to control model behavior:

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

# Configure model parameters for the ChatPrompt
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="openai/gpt-4o-mini",
    model_parameters={
        "temperature": 0.7,
        "max_tokens": 1000,
        "top_p": 0.9
    }
)

# Configure model parameters for the Optimizer
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o",
    model_parameters={
        "temperature": 0.1,  # Lower temperature for more consistent optimization
        "max_tokens": 4096
    }
)
```

## Mixing Providers

You can use different providers for the ChatPrompt and Optimizer:

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer

# Use a cost-effective model for prompt evaluation
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"  # Fast and affordable
)

# Use a powerful model for optimization reasoning
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o"  # Best reasoning capabilities
)
```

> **Tip**
>
> **Recommendation:** Use a capable model like `gpt-4o` or `claude-sonnet-4-6` for the optimizer, even if your production application uses a smaller model. The optimizer only runs during development, so the cost is minimal compared to the quality improvements you'll achieve.

## Troubleshooting

#### AuthenticationError: Invalid API key

Ensure your API key is correctly set in the environment:

```bash
# Check if the key is set
echo $OPENAI_API_KEY

# Set it if missing
export OPENAI_API_KEY="sk-..."
```

#### Model not found

Verify the model name follows the LiteLLM format `provider/model-name`:

```python
# ✅ Correct
model="openai/gpt-4o"
model="gemini/gemini-2.0-flash"

# ❌ Incorrect
model="gpt-4o"  # Missing provider prefix
model="google/gemini-2.0-flash"  # Wrong provider name (use 'gemini')
```

#### Rate limiting errors

If you encounter rate limits, try:

* Reducing `n_threads` in the optimizer
* Using a model with higher rate limits
* Adding delays between API calls

## Next Steps

* Learn about [Optimization Concepts](/development/optimization-runs/optimization/concepts)
* Explore different [Optimization Algorithms](/development/optimization-runs/algorithms/overview)
* Check out the [API Reference](/development/optimization-runs/advanced/api_reference)

> **Info**
>
> For a complete list of supported providers and models, see the [LiteLLM documentation](https://docs.litellm.ai/docs/providers).