Sampling controls

When optimizing prompts, there are two independent sampling layers you can control:

• Dataset subsampling: choose which dataset rows are evaluated (n_samples, n_samples_minibatch, n_samples_strategy).
• Model sampling: request multiple completions per row (n in model_parameters).

Use both to balance cost, stability, and exploration.

Dataset subsampling (n_samples)

n_samples limits how many dataset rows are evaluated per trial. It applies to the evaluation dataset (the validation_dataset if provided, otherwise dataset).

1
result = optimizer.optimize_prompt(
2
    prompt=prompt,
3
    dataset=dataset,
4
    metric=metric,
5
    n_samples=50,
6
)

Notes:

• n_samples accepts an integer, a fractional float, a percent string (e.g. "10%"), or the special values "all", "full", or None.
• If n_samples is larger than the evaluation dataset size, the optimizer falls back to the full dataset and logs a warning.

Deterministic subsampling (n_samples_strategy)

n_samples_strategy controls how dataset rows are selected when n_samples is set. The default strategy is "random_sorted", which:

1. Sorts dataset item IDs.
2. Shuffles them deterministically using the optimizer seed and evaluation phase.
3. Takes the first n_samples IDs.

If your dataset items do not include IDs, the optimizer falls back to the dataset order.

1
result = optimizer.optimize_prompt(
2
    prompt=prompt,
3
    dataset=dataset,
4
    metric=metric,
5
    n_samples=50,
6
    n_samples_strategy="random_sorted",
7
)

Minibatch sampling (n_samples_minibatch)

Some optimizers run inner-loop evaluations (for example, HRPO and GEPA). Use n_samples_minibatch to cap those inner evaluations without reducing the outer evaluation size.

1
result = optimizer.optimize_prompt(
2
    prompt=prompt,
3
    dataset=dataset,
4
    metric=metric,
5
    n_samples=200,
6
    n_samples_minibatch=25,
7
)

If n_samples_minibatch is not set, it defaults to n_samples.

Explicit item selection (dataset_item_ids)

For fully deterministic evaluations, you can pass an explicit list of dataset item IDs to evaluate_prompt. This bypasses the sampling strategy and is mutually exclusive with n_samples.

1
score = optimizer.evaluate_prompt(
2
    prompt=prompt,
3
    dataset=dataset,
4
    metric=metric,
5
    dataset_item_ids=["item-1", "item-2", "item-3"],
6
)

Multiple completions per example (n parameter)

Single-sample evaluation can be noisy. The n parameter lets you generate multiple candidate outputs per example and select the best one, introducing variety and reducing evaluation variance.

How It Works

When you set n > 1 in your prompt’s model_parameters, the optimizer:

1. Requests N completions from the LLM in a single API call (pass@N)
2. Scores each candidate output using your metric
3. Selects the best candidate (best_by_metric policy)
4. Logs all scores and selection info to the Opik trace

In optimizers that already generate multiple prompt variants per round, n is applied per evaluation, so total candidate evaluations scale by prompts_per_round * n.

For tasks that execute generated code (like ARC-AGI or tool-driven agents), this means each prompt produces multiple candidate programs that are executed and scored, and the best candidate is used for optimization feedback.

Configuration

Set the n parameter in your ChatPrompt.model_parameters:

1
from opik_optimizer import ChatPrompt
2
3
# Generate 3 candidates per evaluation, select best
4
prompt = ChatPrompt(
5
    model="gpt-4o-mini",
6
    messages=[
7
        {"role": "system", "content": "You are a helpful assistant."},
8
        {"role": "user", "content": "Answer: {question}"},
9
    ],
10
    model_parameters={
11
        "n": 3,  # Generate 3 completions per call
12
        "temperature": 0.7,  # Higher temp = more variety between candidates
13
    },
14
)

Use Cases

1
# Before: Single sample - noisy evaluation
2
prompt = ChatPrompt(model="gpt-4o-mini", messages=[...])
3
# Score might be 0.6 or 0.9 depending on luck
4
5
# After: Best-of-3 - more stable evaluation
6
prompt = ChatPrompt(
7
    model="gpt-4o-mini",
8
    messages=[...],
9
    model_parameters={"n": 3, "temperature": 0.8},
10
)
11
# Score reflects best achievable output

1
# Optimize for "can this prompt ever get it right?"
2
prompt = ChatPrompt(
3
    model="gpt-4o-mini",
4
    messages=[...],
5
    model_parameters={"n": 5},  # pass@5 style
6
)

1
# Creative task: multiple valid outputs
2
prompt = ChatPrompt(
3
    model="gpt-4o-mini",
4
    messages=[
5
        {"role": "user", "content": "Write a haiku about {topic}"},
6
    ],
7
    model_parameters={"n": 3, "temperature": 1.0},
8
)

Selection Policy

Currently, the optimizer supports these selection policies:

• best_by_metric (default): score each candidate with the metric and pick the best.
• first: pick the first candidate (fast, deterministic, but ignores scoring).
• concat: join all candidates into one output string.
• random: pick a random candidate (seeded if provided).
• max_logprob: pick the candidate with the highest average token logprob (provider support required; logprobs must be enabled in model kwargs).

Use the selection_policy key in model_parameters to override. The optimizer routes these policies through a shared candidate-selection utility so behavior is consistent across optimizers:

1
prompt = ChatPrompt(
2
    model="gpt-4o-mini",
3
    messages=[...],
4
    model_parameters={
5
        "n": 3,
6
        "selection_policy": "first",
7
    },
8
)

For max_logprob, enable logprobs in your model kwargs (provider support varies):

1
prompt = ChatPrompt(
2
    model="gpt-4o-mini",
3
    messages=[...],
4
    model_parameters={
5
        "n": 3,
6
        "selection_policy": "max_logprob",
7
        "logprobs": True,
8
        "top_logprobs": 1,
9
    },
10
)

When selection_policy=best_by_metric, the optimizer:

1. Each candidate is scored independently using your metric function
2. The candidate with the highest score is selected as the final output
3. All scores and the chosen index are logged to the trace metadata

1
# What happens internally:
2
candidates = ["output_1", "output_2", "output_3"]
3
scores = [metric(item, c) for c in candidates]  # [0.7, 0.9, 0.6]
4
best_idx = argmax(scores)  # 1
5
final_output = candidates[best_idx]  # "output_2"

The trace metadata includes:

• n_requested: Number of completions requested
• candidates_scored: Number of candidates evaluated
• candidate_scores: List of all scores (best_by_metric only)
• candidate_logprobs: List of logprob scores (max_logprob only)
• chosen_index: Index of the selected candidate

Cost Considerations