Opik Agent Optimization Overview

Opik Agent Optimizer is a comprehensive toolkit designed to enhance the performance and efficiency of your Large Language Model (LLM) applications. Rather than manually editing prompts and running evaluation, you can use Opik Agent Optimizer to automatically optimize your prompts.

Why optimize prompts ?

Prompt engineering is a skill that can be difficult to master as highlighted by the Anthropic, OpenAI and Google prompt engineering guides. There are a lot of techniques that can be used to help LLMs generate the desired output.

Prompt optimization solves many of the issues that come with prompt engineering:

1. Prompt engineering is not easily repeatable or scalable alone
2. Possible performance degration across models, you need to tune prompts for each model
3. Optimization may unlock performance, cost and reliability improvements
4. As systems evolve, manually tuning multiple prompts becomes increasingly difficult

So when should you use prompt optimization?

Optimizing a prompt

You can optimize any prompt in just a few lines of code

1
import opik_optimizer
2
from opik.evaluation.metrics import LevenshteinRatio
3
4
# Load a demo dataset
5
dataset = opik_optimizer.datasets.tiny_test()
6
7
# Define a metric
8
def levenshtein_ratio(dataset_item, llm_output):
9
    metric = LevenshteinRatio()
10
    return metric.score(reference=dataset_item['label'], output=llm_output)
11
12
# Define the prompt to optimize
13
prompt = opik_optimizer.ChatPrompt(
14
  messages=[
15
    {"role": "system", "content": "You are a helpful assistant."},
16
    {"role": "user", "content": "{text}"},
17
  ]
18
)
19
20
# Run the optimization
21
optimizer = opik_optimizer.MetaPromptOptimizer(model="openai/gpt-4")
22
opt_result = optimizer.optimize_prompt(
23
  prompt=prompt,
24
  dataset=dataset,
25
  metric=levenshtein_ratio
26
)
27
28
print(opt_result.prompt)

You can learn more about running your first optimization in the Quickstart guide.

Optimization Algorithms

Supported algorithms

The Opik Agent Optimizer is an experimental Python library that aims at implementing Prompt and Agent Optimization algorithms in a consistent format.

The following algorithms have been implemented:

Benchmark results

Each optimization algorithm is evaluated against different use-cases and datasets:

1. Arc: The ai2_arc dataset contains a set of multiple choice science questions
2. GSM8K: The gsm8k dataset contains a set of math problems
3. medhallu: The medhallu dataset contains a set of medical questions
4. RagBench: The ragbench dataset contains a set of retrieval (RAG) examples.

The results above are benchmarks tested against gpt-4o-mini, we are using various metrics depending on the dataset including Levenshtein Ratio, Answer Relevance and Hallucination. The results might change if you use a different model, configuration, dataset and starting prompt(s).