MetaPrompt Optimizer

Refine and improve LLM prompts with systematic analysis.

The MetaPrompter is a specialized optimizer designed for meta-prompt optimization. It focuses on improving the structure and effectiveness of prompts through systematic analysis and refinement of prompt templates, instructions, and examples.

The MetaPromptOptimizer is a strong choice when you have an initial instruction prompt and want to iteratively refine its wording, structure, and clarity using LLM-driven suggestions. It excels at general-purpose prompt improvement where the core idea of your prompt is sound but could be phrased better for the LLM, or when you want to explore variations suggested by a reasoning model.

How it works

The MetaPromptOptimizer automates the process of prompt refinement by using a “reasoning” LLM to critique and improve your initial prompt. Here’s a conceptual breakdown:

The optimizer is open-source, you can check out the code in the Opik repository.

Quickstart

You can use the MetaPromptOptimizer to optimize a prompt by following these steps:

1 from opik_optimizer import MetaPromptOptimizer
2 from opik.evaluation.metrics import LevenshteinRatio
3 from opik_optimizer import datasets, ChatPrompt
4 
5 # Initialize optimizer
6 optimizer = MetaPromptOptimizer(
7     model="openai/gpt-4",
8     model_parameters={
9         "temperature": 0.1,
10         "max_tokens": 5000
11     },
12     n_threads=8,
13     seed=42
14 )
15 
16 # Prepare dataset
17 dataset = datasets.hotpot(count=300)
18 
19 # Define metric and task configuration (see docs for more options)
20 def levenshtein_ratio(dataset_item, llm_output):
21     return LevenshteinRatio().score(reference=dataset_item['answer'], output=llm_output)
22 
23 prompt = ChatPrompt(
24     messages=[
25         {"role": "system", "content": "Provide an answer to the question."},
26         {"role": "user", "content": "{question}"}
27     ]
28 )
29 
30 # Run optimization
31 results = optimizer.optimize_prompt(
32     prompt=prompt,
33     dataset=dataset,
34     metric=levenshtein_ratio,
35     n_samples=100
36 )
37 
38 # Access results
39 results.display()

Configuration Options

Optimizer parameters

The optimizer has the following parameters:

model

strDefaults to openai/gpt-5-nano

LiteLLM model name for optimizer’s internal reasoning/generation calls

model_parameters

dict[str, typing.Any] | None

Optional dict of LiteLLM parameters for optimizer’s internal LLM calls. Common params: temperature, max_tokens, max_completion_tokens, top_p.

prompts_per_round

intDefaults to 4

Number of candidate prompts to generate per optimization round

enable_context

boolDefaults to True

Whether to include task-specific context when reasoning about improvements

n_threads

intDefaults to 12

Number of parallel threads for prompt evaluation

verbose

intDefaults to 1

Controls internal logging/progress bars (0=off, 1=on)

seed

intDefaults to 42

Random seed for reproducibility

`optimize_prompt` parameters

The optimize_prompt method has the following parameters:

prompt

ChatPrompt

The ChatPrompt to optimize. Can include system/user/assistant messages, tools, and model configuration.

dataset

Dataset

Opik Dataset containing evaluation examples. Each item is passed to the prompt during evaluation.

metric

Callable

Evaluation function that takes (dataset_item, llm_output) and returns a score (float). Higher scores indicate better performance.

experiment_config

dict | None

Optional metadata dictionary to log with Opik experiments. Useful for tracking experiment parameters and context.

n_samples

int | float | str | None

Number of dataset items to use per evaluation. Use counts (e.g., 50), fractions (e.g., 0.1), percentages (e.g., “10%”), or “all”/“full”/None for the full dataset.

n_samples_minibatch

int | None

Optional number of samples for inner-loop minibatches (defaults to n_samples).

n_samples_strategy

str | None

Sampling strategy for subsampling (default: “random_sorted”).

auto_continue

boolDefaults to False

If True, optimizer may continue beyond max_trials if improvements are still being found.

agent_class

type[opik_optimizer.optimizable_agent.OptimizableAgent] | None

Custom agent class for prompt execution. If None, uses default LiteLLM-based agent. Must inherit from OptimizableAgent.

project_name

strDefaults to Optimization

Opik project name for logging traces and experiments. Default: “Optimization”

max_trials

intDefaults to 10

Maximum total number of prompts to evaluate across all rounds. Optimizer stops when this limit is reached.

mcp_config

opik_optimizer.mcp_utils.mcp_workflow.MCPExecutionConfig | None

Optional MCP (Model Context Protocol) execution configuration for prompts that use external tools. Enables tool-calling workflows. Default: None

candidate_generator

collections.abc.Callable[..., list[opik_optimizer.api_objects.chat_prompt.ChatPrompt]] | None

Optional custom function to generate candidate prompts. Overrides default meta-reasoning generator. Should return list[ChatPrompt].

candidate_generator_kwargs

dict[str, typing.Any] | None

Optional kwargs to pass to candidate_generator.

args

Any

kwargs

Any

Model Support

There are two models to consider when using the MetaPromptOptimizer:

MetaPromptOptimizer.model: The model used for the reasoning and candidate generation.
ChatPrompt.model: The model used to evaluate the prompt.

The model parameter accepts any LiteLLM-supported model string (e.g., "gpt-4o", "azure/gpt-4", "anthropic/claude-3-opus", "gemini/gemini-1.5-pro"). You can also pass in extra model parameters using the model_parameters parameter:

1 optimizer = MetaPromptOptimizer(
2     model="anthropic/claude-3-opus-20240229",
3     model_parameters={
4         "temperature": 0.7,
5         "max_tokens": 4096
6     }
7 )

MCP Tool Calling Support

The MetaPrompt Optimizer is the only optimizer that currently supports MCP (Model Context Protocol) tool calling optimization. This means you can optimize prompts that include MCP tools and function calls.

MCP tool calling optimization is a specialized feature that allows the optimizer to understand and optimize prompts that use external tools and functions through the Model Context Protocol. This is particularly useful for complex agent workflows that require tool usage.

For comprehensive information about tool optimization, see the Tool Optimization Guide.

Research and References

Meta-Prompting for Language Models

1	from opik_optimizer import MetaPromptOptimizer
2	from opik.evaluation.metrics import LevenshteinRatio
3	from opik_optimizer import datasets, ChatPrompt
4
5	# Initialize optimizer
6	optimizer = MetaPromptOptimizer(
7	model="openai/gpt-4",
8	model_parameters={
9	"temperature": 0.1,
10	"max_tokens": 5000
11	},
12	n_threads=8,
13	seed=42
14	)
15
16	# Prepare dataset
17	dataset = datasets.hotpot(count=300)
18
19	# Define metric and task configuration (see docs for more options)
20	def levenshtein_ratio(dataset_item, llm_output):
21	return LevenshteinRatio().score(reference=dataset_item['answer'], output=llm_output)
22
23	prompt = ChatPrompt(
24	messages=[
25	{"role": "system", "content": "Provide an answer to the question."},
26	{"role": "user", "content": "{question}"}
27	]
28	)
29
30	# Run optimization
31	results = optimizer.optimize_prompt(
32	prompt=prompt,
33	dataset=dataset,
34	metric=levenshtein_ratio,
35	n_samples=100
36	)
37
38	# Access results
39	results.display()

1	optimizer = MetaPromptOptimizer(
2	model="anthropic/claude-3-opus-20240229",
3	model_parameters={
4	"temperature": 0.7,
5	"max_tokens": 4096
6	}
7	)

How it works

Quickstart

Configuration Options

Optimizer parameters

optimize_prompt parameters

Model Support

MCP Tool Calling Support

Research and References

How it works

Quickstart

Configuration Options

Optimizer parameters

optimize_prompt parameters

Model Support

MCP Tool Calling Support

Research and References

`optimize_prompt` parameters

`optimize_prompt` parameters