Optimizer Frequently Asked Questions

The Agent Optimizer provides a unified interface for optimizing your existing prompts and agents with cutting edge optimization algorithms. In addition to giving you access to cutting edge academic optimizers like GEPA, we also provide a set of algorithms that have been developed in-house based on production applications.

The optimizer will allow you to improve the performance of your agents without the need for manual prompt engineering. You can also use the optimizer to reduce the size of the prompts in your agents, reducing cost and latency while maintaining performance.

To get started, you will need:

1. The prompt you want to optimize
2. A dataset of examples to optimizer on, you can start with as little as 10
3. A metric to evaluate the performance of the prompt

Once you have these, check out the Quickstart Guide to run your first optimization.

Yes, we would be more than happy to help you setup the Opik Optimizer for your use case ! You can join our Slack community and ask for help there.

Opik Agent Optimizer supports a wide range of optimization algorithms including:

• Hierarchical Reflective Optimization
• GEPA Optimization
• Evolutionary Optimization
• Few-shot Bayesian Optimization
• MetaPrompt Optimization

If you would like us to add a new optimization algorithm, simply create an issue on our GitHub repository and we will be happy to add it !

Knowing which optimizer to use depends on your specific needs. As a rule of thumb, we recommend starting with the Hierarchical Reflective Optimization as this has been shown to be a strong baseline for most tasks.

You can also try to use:

1. GEPA: This is one of the top performing academic optimizers and is a good option if you have a complex task.
2. FewShotBayesianOptimizer: If you have a task that is quite repetitive in the formatting of the prompt and responses then this is a good option.

While some optimizations run for shorts period of time, it is common for optimizations to take a couple of hours to complete. As you are starting out, we recommend setting the max_trials parameter to a reasonable number and increasing / decreasing it as you go.

The number of samples you need depends on your task. As a rule of thumb, we recommend starting with at least 10 samples. The more samples you have, the more accurate the optimization will be.

It’s important to note that there are two models at play when it comes to optimizing prompts:

1. The chatPrompt model: This is the model that you use the prompt with, should be the same as your application.
2. The optimizer model: This is the model that the optimizer uses to improve your prompt, you will get the best performance by using the most powerful model for the optimization.

The Opik Agent Optimizer uses LiteLLM to support a wide range of models including:

• OpenAI (e.g., GPT-5, GPT-4.5, GPT-4)
• Anthropic (e.g., Claude 3.7 Sonnet, Claude 3.7 Haiku)
• Google (e.g., Gemini 2.0 Flash, Gemini 2.0 Pro)
• Cohere (e.g., Command, Command-Pro)
• Mistral AI (e.g., Mistral-7B-Instruct, Mistral-7B-Instruct-Pro)
• Locally hosted models (e.g., via Ollama, Hugging Face Inference Endpoints)
• And many others supported by LiteLLM.

For comprehensive details on how to specify different model providers, pass API keys, and configure endpoints, please refer to our dedicated LiteLLM Support for Optimizers guide.

There are a few things you can try to improve the performance of your optimization:

1. Review your optimization metric, ideally it should provide the model with an insightful reason for the score it gives. By improving the quality of the metric, the optimizer will be able to make better optimizations.
2. Review your dataset, ideally it should be a diverse set of examples that cover the different scenarios you want to optimize for.
3. Use more powerful models for both the chatPrompt model and the optimizer, as models get more powerful, they will be able to generate better optimizations.

This error occurs when you pass an incorrect type to the optimizer’s optimize_prompt() method.

Solution: Ensure you’re using the ChatPrompt class to define your prompt:

1
from opik_optimizer import ChatPrompt
2
3
prompt = ChatPrompt(
4
    messages=[
5
        {"role": "system", "content": "Your system prompt here"},
6
        {"role": "user", "content": "Your user prompt with {variable}"}
7
    ],
8
    model="gpt-4"
9
)

This error occurs when the dataset passed to the optimizer is not a proper Dataset object.

Solution: Use the Dataset class to create your dataset:

1
import opik
2
3
client = opik.Opik()
4
dataset = client.get_or_create_dataset(name='your-dataset-name')
5
dataset.insert([
6
    {"input": "example 1", "output": "expected 1"},
7
    {"input": "example 2", "output": "expected 2"}
8
])

This error occurs when the metric parameter is not callable or doesn’t have the correct signature.

Solution: Ensure your metric is a function that takes dataset_item and llm_output as arguments and returns a ScoreResult:

1
from opik.evaluation.metrics import score_result
2
3
def my_metric(dataset_item, llm_output):
4
    # Your scoring logic here
5
    score = calculate_score(dataset_item, llm_output)
6
    return ScoreResult(
7
      name="my-metric",
8
        value=score,
9
        reason="Explanation for the score"
10
    )

This error occurs when your prompt template contains placeholders (e.g., {variable}) that don’t match your dataset fields.

Solution: Ensure all placeholders in your prompt match the keys in your dataset:

1
# Prompt with {question} placeholder
2
prompt = ChatPrompt(
3
    user="Answer: {question}",
4
    model="gpt-4"
5
)
6
7
# Dataset must have 'question' field
8
dataset = Dataset.from_list([
9
    {"question": "What is AI?", "output": "..."}
10
])

This error occurs when trying to use the GepaOptimizer without the required gepa package installed.

Solution: Install the gepa package:

$
pip install gepa

This error typically occurs when the LLM provider API key is not configured in your environment.

Solution: Set the appropriate environment variable for your LLM provider:

$
# For OpenAI
>
export OPENAI_API_KEY="your-api-key"
>
>
# For Anthropic
>
export ANTHROPIC_API_KEY="your-api-key"
>
>
# For other providers, check the LiteLLM documentation