Opik Agent Optimizer Guide

Opik Agent Optimizer Core Concepts

Common terminology and how we optimize

Overview

Understanding the core concepts of the Opik Agent Optimizer is essential for unlocking its full potential in LLM evaluation and optimization. This section explains the foundational terms, processes, and strategies that underpin effective agent and prompt optimization within Opik.

What is Agent Optimization (and Prompt Optimization)?

In Opik, Agent Optimization refers to the systematic process of refining and evaluating the prompts, configurations, and overall design of language model-based applications to maximize their performance. This is an iterative approach leveraging continuous testing, data-driven refinement, and advanced evaluation techniques.

Prompt Optimization is a crucial subset of Agent Optimization. It focuses specifically on improving the instructions (prompts) given to Large Language Models (LLMs) to achieve desired outputs more accurately, consistently, and efficiently. Since prompts are the primary way to interact with and guide LLMs, optimizing them is fundamental to enhancing any LLM-powered agent or application.

Opik Agent Optimizer provides tools for both: directly optimizing individual prompt strings and also for optimizing more complex agentic structures that might involve multiple prompts, few-shot examples, or tool interactions.

Key Terms

A single execution of a prompt optimization process using a specific configuration. For example, calling optimizer.optimize_prompt(...) once constitutes a Run. Each Run is typically logged to the Opik platform for tracking.

Often used interchangeably with “Run” in the context of Opik Agent Optimizer documentation referring to a single optimization execution. More broadly in experimental design, a trial might involve multiple runs to ensure statistical significance, but for Opik Agent Optimizer, focus on it as one complete optimize_prompt cycle. Within some optimizers (e.g., FewShotBayesianOptimizer using Optuna, or MetaPromptOptimizer’s candidate evaluations), “trial” can also refer to an individual evaluation point within the larger optimization process (e.g., testing one specific set of few-shot examples).

A specialized algorithm within the Opik Agent Optimizer SDK designed to enhance prompt effectiveness. Each optimizer (e.g., MetaPromptOptimizer, FewShotBayesianOptimizer, MiproOptimizer, EvolutionaryOptimizer) employs unique strategies and configurable parameters to address specific optimization goals.

Specific to the MetaPromptOptimizer, this is an LLM used by the optimizer to analyze the current prompt and task, and then generate new candidate prompt suggestions and the reasoning behind them. It can be the same as or different from the model used for evaluating the prompts.

In the context of DSPy (which is leveraged by MiproOptimizer), a teleprompter is a component that optimizes a DSPy program. It does this by, for example, generating and refining instructions or few-shot examples for the modules within the program. MIPRO is one such advanced teleprompter.

Relevant for multi-objective optimization, as seen in the EvolutionaryOptimizer when enable_moo=True. A Pareto front represents a set of solutions where no solution is strictly better than another across all objectives. For example, one solution might have a higher accuracy but also a longer prompt, while another has slightly lower accuracy but a much shorter prompt. Neither is definitively superior without knowing the user’s specific priorities.

A collection of data items, typically with inputs and expected outputs (ground truth), used to guide and evaluate the prompt optimization process. See Datasets and Testing for more.

The object to optimize which contains your chat messages with placeholders for variables that change on each example. See the API Reference.

An object defining how to measure the performance of a prompt. The metric functions should accept two parameters:

  • dataset_item: A dictionary with the dataset item keys
  • llm_output: This will be populated with the LLM response

It should return either a ScoreResult object or a float.

The expected or correct output for a given input in your dataset. This is used as a benchmark by evaluation metrics to score the LLM’s actual responses.

Quantitative measures (e.g., accuracy, Levenshtein distance, F1 score, ROUGE, custom scores) used to assess the quality of an LLM’s output compared to the ground truth, and thus the effectiveness of a prompt. Opik provides several built-in metrics and supports custom ones.

The Role of Evaluation in Optimization

At the core of every optimizer is a robust evaluation process. The optimizers don’t just guess what makes a good prompt; they systematically test candidate prompts using the same underlying evaluation principles and tools available elsewhere in the Opik platform.

Here’s how it generally works:

  1. Dataset as Ground Truth: The dataset you provide, with its input samples and expected outputs (ground truth), serves as the benchmark.
  2. Metric as Judge: The metric defines how a candidate prompt’s performance is measured. It specifies the metric (e.g., LevenshteinRatio, Accuracy, or a custom metric) and how to map the LLM’s output and dataset fields to that metric’s inputs. It should be a function that takes two arguments: dataset_item and llm_output.
  3. Iterative Evaluation: As an optimizer generates or selects new candidate prompts (or prompt components), it internally runs evaluations. For each candidate, it typically:
    • Uses the candidate prompt to get responses from the specified LLM for a selection of samples from your dataset.
    • Compares these LLM responses against the ground truth in the dataset using the logic defined in your Metric.
    • Calculates a score based on this comparison.
  4. Guided Search: The scores obtained from these internal evaluations guide the optimizer’s search strategy. It learns which types of prompts perform better and focuses its efforts accordingly, whether it’s refining wording, selecting different few-shot examples, or evolving prompt structures.

Understanding Opik’s broader evaluation capabilities can help you better configure and interpret optimizer results.

  • For a general overview of evaluation, see: Evaluation Overview
  • For details on evaluating individual prompts (similar to what optimizers do internally): Evaluate Prompts
  • For evaluating more complex agentic systems (relevant if your optimization target is an agent): Evaluate Agents
  • To learn about available metrics: Metrics Overview

Optimization Workflow

  1. Initialization
    • Select the Optimizer appropriate for your task.
    • Configure its parameters (e.g., model, project_name, algorithm-specific settings).
    • Prepare your Dataset, ensuring it has relevant inputs and ground truth outputs.
    • Define your prompt and metric.
  2. Trial Execution (optimize_prompt)
    • Run the optimization process by calling the optimizer’s optimize_prompt method, passing in your dataset, prompt, and metric.
    • The optimizer will iteratively generate/select, and evaluate candidate prompts or prompt configurations.
    • Results and progress are typically logged to the Opik platform.
  3. Analysis & Results
    • The optimize_prompt method returns an OptimizationResult object.
    • Use result.display() for a console summary or inspect its attributes (result.prompt, result.score, result.history) programmatically.
    • Analyze the detailed run information in the Opik UI to understand performance improvements and identify the best prompt(s).
  4. Refinement (Optional / Iterative)
    • Based on the results, you might choose to adjust optimizer parameters, refine your initial ChatPrompt, improve your dataset, or even try a different optimizer and run further optimization trials.
  5. Validation (Recommended)
    • Test the best prompt(s) found during optimization on a separate, unseen validation or test dataset to ensure the improvements generalize well.

Core Optimization Strategies (employed by various optimizers)

Most optimizers work by repeatedly trying variations of a prompt (or prompt components), evaluating them, and using the results to guide the next set of variations. This is a core loop of generate/evaluate/select.

Providing examples of the desired input/output behavior directly within the prompt can significantly improve LLM performance. The FewShotBayesianOptimizer automates finding the best examples for chat models Other optimizers like MiproOptimizer (via DSPy) also leverage demonstrations.

Using an LLM itself to suggest improvements to a prompt, or to generate new candidate prompts based on some analysis. The MetaPromptOptimizer is a prime example. The EvolutionaryOptimizer can also use LLMs for its genetic operators.

A strategy for efficiently searching large parameter spaces by building a probabilistic model of how different parameter choices affect the outcome. Used by FewShotBayesianOptimizer (via Optuna) to select few-shot examples.

Techniques inspired by biological evolution, involving a population of solutions (prompts) that undergo selection, crossover (recombination), and mutation over generations to find optimal solutions. Used by the EvolutionaryOptimizer.

In DSPy, “compiling” a program means optimizing its internal prompts and few-shot demonstrations for a given task, dataset, and metric. MiproOptimizer uses this concept.

Best Practices

1

1. Ensure Dataset Quality is Key

Ensure datasets contain sufficient, diverse, and high-quality examples representative of your actual task. Maintain accurate ground truth and consistent formatting.

2

2. Start Simple, then Iterate

Begin with a reasonable baseline prompt and default optimizer settings. Iteratively adjust parameters, dataset, or even optimizer choice based on observed results.

3

3. Conduct Meaningful Evaluation

Use evaluation metrics that genuinely reflect success for your specific task. Always validate final prompts on an unseen test set to check for generalization.

4

4. Understand Your Chosen Optimizer

Read its specific documentation to understand its mechanism, key parameters, and when it’s most effective.

5

5. Log and Track Experiments

Leverage the Opik platform (via project_name) to keep your optimization runs organized and compare results effectively.

Next Steps

📓 Want to see these concepts in action? Check out our Example Projects & Cookbooks for step-by-step, runnable Colab notebooks.