Overview

Prompt engineering is the practice of designing and refining prompts to help LLMs generate the desired output. Typically prompt engineering is a manual process that involves editing the prompt, evaluating it, reviewing the results and trying again.

Prompt optimization is the process of automating the prompt engineering process.

Why optimize prompts ?

Prompt engineering is a skill that can be difficult to master as highlighted by the Anthropic , OpenAI and Google prompt engineering guides. There are a lot of techniques that can be used to help LLMs generate the desired output.

Prompt optimization solves many of the issues that come with prompt engineering:

Prompt engineering is not easily repeatable or scalable alone
Variations across models could lead to performance degration, you need to tune prompts for each model
Optimization may unlock performance, cost and reliability improvements
As systems evolve to be more interdependent, manually tuning multiple prompts becomes increasingly difficult.

So when should you use prompt optimization?

Aspect	Prompt Engineering	Prompt Optimization
Scope	Broad — includes designing, experimenting, refining	Narrow — improving already existing prompts
Goal	Create a working prompt for a specific task	Maximize performance (accuracy, efficiency, etc.)
Involves	Initial drafting, understanding model behavior	Tweaking wording, structure, or context
When used	Early in task setup or experimentation	After a baseline prompt is in place

Optimization algorithms

Supported algorithms

The Opik Optimizer is an experimental Python library that aims at implementing Prompt and Agent Optimization algorithms in a consistent format.

The following algorithms have been implemented:

Algorithm	Description	Guide
MetaPrompt Optimization	A prompt engineering algorithm that uses a meta-prompt to generate a set of candidate prompts and then uses a Bayesian optimization algorithm to identify the best prompt.	MetaPrompt Optimization
Few-shot Bayesian Optimization	A prompt engineering algorithm that uses a few-shot learning approach to generate a set of candidate prompts and then uses a Bayesian optimization algorithm to identify the best prompt.	Few-shot Bayesian Optimization
MIPRO Optimization	A prompt engineering algorithm that uses a MIPRO (Multi-Instance Prompt Refinement) approach to generate a set of candidate prompts and then uses a Bayesian optimization algorithm to identify the best prompt.	Mipro Optimization

If you would like us to implement another optimization algorithm, reach out to us on Github.

Benchmark results

We are currently working on the benchmarking results, these are early preliminary results and are subject to change.

Each optimization algorithm is evaluated against different use-cases and datasets:

Arc: The ai2_arc dataset contains a set of multiple choice science questions
GSM8K: The gsm8k dataset contains a set of math problems
Medhallucination: The medhallucination dataset contains a set of medical questions
RagBench: The ragbench dataset contains a set of questions and answers

Rank	Algorithm	Average Score	Arc dataset	GSM8K dataset	Medhallucination	RagBench
1	Few-shot Bayesian Optimization	0.6733	0.2809	0.5926	0.9180	0.9015
2	Mipro Optimization	0.5591	0.0197	0.3970	0.9270	0.8928
3	MetaPrompt Optimization	0.5201	0.2500	0.2693	0.9179	0.6431
4	No optimization	0.1199	0.0169	0.2406	0.1238	0.0981

The results above are for gpt-40-mini, the results might change if you use a different model.

Note: This results are preliminary and subject to change, you can learn more about our benchmarks here._