Configuring LLM Providers | Opik Documentation

The Opik Agent Optimizer uses LiteLLM under the hood, giving you access to 100+ LLM providers with a unified interface. This guide shows you how to configure different providers for both your ChatPrompt (the model that runs your prompt) and the Optimizer (the model that improves your prompt).

Understanding the Two Model Types

When using the Opik Optimizer, there are two distinct models to configure:

Model Type	Purpose	Recommendation
ChatPrompt model	The model that executes your prompt during evaluation	Use the same model as your production application
Optimizer model	The model that analyzes failures and generates improved prompts	Use the most capable model available for best results

1 from opik_optimizer import ChatPrompt, MetaPromptOptimizer
2 
3 # ChatPrompt model - this is the model your prompt runs on
4 prompt = ChatPrompt(
5     messages=[
6         {"role": "system", "content": "You are a helpful assistant."},
7         {"role": "user", "content": "{question}"}
8     ],
9     model="gemini/gemini-2.0-flash"  # Your production model
10 )
11 
12 # Optimizer model - this is the model that improves your prompt
13 optimizer = MetaPromptOptimizer(
14     model="openai/gpt-4o"  # Use a powerful model for optimization
15 )

LiteLLM Model Format

All models use the LiteLLM format: provider/model-name

1 # Examples of the LiteLLM model format
2 model="openai/gpt-4o"                          # OpenAI
3 model="anthropic/claude-3-5-sonnet-20241022"   # Anthropic
4 model="gemini/gemini-2.0-flash"                # Google Gemini
5 model="azure/my-deployment-name"               # Azure OpenAI
6 model="ollama/llama3"                          # Ollama (local)
7 model="openrouter/google/gemini-2.0-flash"     # OpenRouter

Provider Configuration

OpenAI

Anthropic

Google Gemini

Azure OpenAI

Ollama (Local)

OpenRouter

OpenAI

Environment Variable:

$ export OPENAI_API_KEY="sk-..."

Available Models:

openai/gpt-4o - Most capable model
openai/gpt-4o-mini - Fast and cost-effective
openai/gpt-4-turbo - Previous generation
openai/o1 - Reasoning model
openai/o3-mini - Efficient reasoning model

Example:

1 from opik_optimizer import ChatPrompt, MetaPromptOptimizer
2 
3 prompt = ChatPrompt(
4     messages=[
5         {"role": "system", "content": "You are a helpful assistant."},
6         {"role": "user", "content": "{question}"}
7     ],
8     model="openai/gpt-4o-mini"
9 )
10 
11 optimizer = MetaPromptOptimizer(model="openai/gpt-4o")

Environment Variables Reference

Provider	Environment Variable	How to Get
OpenAI	`OPENAI_API_KEY`	platform.openai.com/api-keys
Anthropic	`ANTHROPIC_API_KEY`	console.anthropic.com
Google Gemini	`GOOGLE_API_KEY` or `GEMINI_API_KEY`	aistudio.google.com/apikey
Azure OpenAI	`AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`	Azure Portal
OpenRouter	`OPENROUTER_API_KEY`	openrouter.ai/keys
Ollama	None required (local)	ollama.ai

Model Parameters

You can pass additional parameters to control model behavior:

1 from opik_optimizer import ChatPrompt, MetaPromptOptimizer
2 
3 # Configure model parameters for the ChatPrompt
4 prompt = ChatPrompt(
5     messages=[
6         {"role": "system", "content": "You are a helpful assistant."},
7         {"role": "user", "content": "{question}"}
8     ],
9     model="openai/gpt-4o-mini",
10     model_parameters={
11         "temperature": 0.7,
12         "max_tokens": 1000,
13         "top_p": 0.9
14     }
15 )
16 
17 # Configure model parameters for the Optimizer
18 optimizer = MetaPromptOptimizer(
19     model="openai/gpt-4o",
20     model_parameters={
21         "temperature": 0.1,  # Lower temperature for more consistent optimization
22         "max_tokens": 4096
23     }
24 )

Mixing Providers

You can use different providers for the ChatPrompt and Optimizer:

1 from opik_optimizer import ChatPrompt, MetaPromptOptimizer
2 
3 # Use a cost-effective model for prompt evaluation
4 prompt = ChatPrompt(
5     messages=[
6         {"role": "system", "content": "You are a helpful assistant."},
7         {"role": "user", "content": "{question}"}
8     ],
9     model="gemini/gemini-2.0-flash"  # Fast and affordable
10 )
11 
12 # Use a powerful model for optimization reasoning
13 optimizer = MetaPromptOptimizer(
14     model="openai/gpt-4o"  # Best reasoning capabilities
15 )

Recommendation: Use a capable model like gpt-4o or claude-3-5-sonnet for the optimizer, even if your production application uses a smaller model. The optimizer only runs during development, so the cost is minimal compared to the quality improvements you’ll achieve.

Troubleshooting

AuthenticationError: Invalid API key

Ensure your API key is correctly set in the environment:

$ # Check if the key is set
> echo $OPENAI_API_KEY
> 
> # Set it if missing
> export OPENAI_API_KEY="sk-..."

Model not found

Verify the model name follows the LiteLLM format provider/model-name:

1 # ✅ Correct
2 model="openai/gpt-4o"
3 model="gemini/gemini-2.0-flash"
4 
5 # ❌ Incorrect
6 model="gpt-4o"  # Missing provider prefix
7 model="google/gemini-2.0-flash"  # Wrong provider name (use 'gemini')

Rate limiting errors

If you encounter rate limits, try:

Reducing n_threads in the optimizer
Using a model with higher rate limits
Adding delays between API calls

Next Steps

Learn about Optimization Concepts
Explore different Optimization Algorithms
Check out the API Reference

For a complete list of supported providers and models, see the LiteLLM documentation.