Skip to navigation
DevelopmentOptimization runsOptimization

Configuring LLM Providers

Use any LLM provider with the Opik Optimizer
View as Markdown

In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a project_name when creating datasets and running experiments so they are associated with the correct project.

The Opik Agent Optimizer uses LiteLLM under the hood, giving you access to 100+ LLM providers with a unified interface. This guide shows you how to configure different providers for both your ChatPrompt (the model that runs your prompt) and the Optimizer (the model that improves your prompt).

Understanding the Two Model Types

When using the Opik Optimizer, there are two distinct models to configure:

Model TypePurposeRecommendation
ChatPrompt modelThe model that executes your prompt during evaluationUse the same model as your production application
Optimizer modelThe model that analyzes failures and generates improved promptsUse the most capable model available for best results
from opik_optimizer import ChatPrompt, MetaPromptOptimizer
# ChatPrompt model - this is the model your prompt runs on
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"  # Your production model
)
# Optimizer model - this is the model that improves your prompt
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o"  # Use a powerful model for optimization
)

LiteLLM Model Format

All models use the LiteLLM format: provider/model-name

# Examples of the LiteLLM model format
model="openai/gpt-4o"                          # OpenAI
model="anthropic/claude-sonnet-4-6"            # Anthropic
model="gemini/gemini-2.0-flash"                # Google Gemini
model="azure/my-deployment-name"               # Azure OpenAI
model="ollama/llama3"                          # Ollama (local)
model="openrouter/google/gemini-2.0-flash"     # OpenRouter

Provider Configuration

OpenAI

Environment Variable:

export OPENAI_API_KEY="sk-..."

Available Models:

  • openai/gpt-4o - Most capable model
  • openai/gpt-4o-mini - Fast and cost-effective
  • openai/gpt-4-turbo - Previous generation
  • openai/o1 - Reasoning model
  • openai/o3-mini - Efficient reasoning model

Example:

from opik_optimizer import ChatPrompt, MetaPromptOptimizer
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="openai/gpt-4o-mini"
)
optimizer = MetaPromptOptimizer(model="openai/gpt-4o")

Environment Variables Reference

ProviderEnvironment VariableHow to Get
OpenAIOPENAI_API_KEYplatform.openai.com/api-keys
AnthropicANTHROPIC_API_KEYconsole.anthropic.com
Google GeminiGOOGLE_API_KEY or GEMINI_API_KEYaistudio.google.com/apikey
Azure OpenAIAZURE_API_KEY, AZURE_API_BASE, AZURE_API_VERSIONAzure Portal
OpenRouterOPENROUTER_API_KEYopenrouter.ai/keys
OllamaNone required (local)ollama.ai

Model Parameters

You can pass additional parameters to control model behavior:

from opik_optimizer import ChatPrompt, MetaPromptOptimizer
# Configure model parameters for the ChatPrompt
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="openai/gpt-4o-mini",
    model_parameters={
        "temperature": 0.7,
        "max_tokens": 1000,
        "top_p": 0.9
    }
)
# Configure model parameters for the Optimizer
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o",
    model_parameters={
        "temperature": 0.1,  # Lower temperature for more consistent optimization
        "max_tokens": 4096
    }
)

Mixing Providers

You can use different providers for the ChatPrompt and Optimizer:

from opik_optimizer import ChatPrompt, MetaPromptOptimizer
# Use a cost-effective model for prompt evaluation
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"  # Fast and affordable
)
# Use a powerful model for optimization reasoning
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o"  # Best reasoning capabilities
)

Recommendation: Use a capable model like gpt-4o or claude-sonnet-4-6 for the optimizer, even if your production application uses a smaller model. The optimizer only runs during development, so the cost is minimal compared to the quality improvements you’ll achieve.

Troubleshooting

Ensure your API key is correctly set in the environment:

# Check if the key is set
echo $OPENAI_API_KEY
# Set it if missing
export OPENAI_API_KEY="sk-..."

Verify the model name follows the LiteLLM format provider/model-name:

# ✅ Correct
model="openai/gpt-4o"
model="gemini/gemini-2.0-flash"
# ❌ Incorrect
model="gpt-4o"  # Missing provider prefix
model="google/gemini-2.0-flash"  # Wrong provider name (use 'gemini')

If you encounter rate limits, try:

  • Reducing n_threads in the optimizer
  • Using a model with higher rate limits
  • Adding delays between API calls

Next Steps

For a complete list of supported providers and models, see the LiteLLM documentation.