Few-Shot Bayesian Optimizer

The FewShotBayesianOptimizer is a sophisticated prompt optimization tool adds relevant examples from your sample questions to the system prompt using Bayesian optimization techniques.

How It Works

The FewShotBayesianOptimizer uses Bayesian optimization to find the optimal set and number of few-shot examples to include with your base instruction prompt for chat models. It Uses Optuna, a hyperparameter optimization framework, to guide the search for the optimal set and number of few-shot examples.

Quickstart

You can use the FewShotBayesianOptimizer to optimize a prompt by following these steps:

1
from opik_optimizer import FewShotBayesianOptimizer
2
from opik.evaluation.metrics import LevenshteinRatio
3
from opik_optimizer import datasets, ChatPrompt
4
5
# Initialize optimizer
6
optimizer = FewShotBayesianOptimizer(
7
    model="openai/gpt-4",
8
    model_parameters={
9
        "temperature": 0.1,
10
        "max_tokens": 5000
11
    },
12
)
13
14
# Prepare dataset
15
dataset = datasets.hotpot(count=300)
16
17
# Define metric and prompt (see docs for more options)
18
def levenshtein_ratio(dataset_item, llm_output):
19
    return LevenshteinRatio().score(reference=dataset_item["answer"], output=llm_output)
20
21
prompt = ChatPrompt(
22
    messages=[
23
        {"role": "system", "content": "Provide an answer to the question."},
24
        {"role": "user", "content": "{question}"}
25
    ]
26
)
27
28
# Run optimization
29
results = optimizer.optimize_prompt(
30
    prompt=prompt,
31
    dataset=dataset,
32
    metric=levenshtein_ratio,
33
    n_samples=100
34
)
35
36
# Access results
37
results.display()

Configuration Options

Optimizer parameters

The optimizer has the following parameters:

optimize_prompt parameters

The optimize_prompt method has the following parameters:

Model Support

There are two models to consider when using the FewShotBayesianOptimizer:

• FewShotBayesianOptimizer.model: The model used to generate the few-shot template and placeholder.
• ChatPrompt.model: The model used to evaluate the prompt.

The model parameter accepts any LiteLLM-supported model string (e.g., "gpt-4o", "azure/gpt-4", "anthropic/claude-3-opus", "gemini/gemini-1.5-pro"). You can also pass in extra model parameters using the model_parameters parameter:

1
optimizer = FewShotBayesianOptimizer(
2
    model="anthropic/claude-3-opus-20240229",
3
    model_parameters={
4
        "temperature": 0.7,
5
        "max_tokens": 4096
6
    }
7
)

Next Steps

1. Explore specific Optimizers for algorithm details.
2. Refer to the FAQ for common questions and troubleshooting.
3. Refer to the API Reference for detailed configuration options.