evaluate_prompt

opik.evaluation.evaluate_prompt(dataset: Dataset | DatasetVersion, messages: List[Dict[str, Any]], model: str | OpikBaseModel | None = None, scoring_metrics: List[BaseMetric] | None = None, scoring_functions: List[ScorerFunctionProtocol] | None = None, experiment_name_prefix: str | None = None, experiment_name: str | None = None, project_name: str | None = None, experiment_config: Dict[str, Any] | None = None, verbose: int = 1, nb_samples: int | None = None, task_threads: int = 16, prompt: BasePrompt | None = None, dataset_item_ids: List[str] | None = None, dataset_sampler: BaseDatasetSampler | None = None, trial_count: int = 1, experiment_scoring_functions: List[Callable[[List[TestResult]], ScoreResult | List[ScoreResult]]] | None = None, experiment_tags: List[str] | None = None, dataset_filter_string: str | None = None) → EvaluationResult

Performs prompt evaluation on a given dataset.

Parameters:

• dataset – An Opik Dataset or DatasetVersion instance

• messages – A list of prompt messages to evaluate.

• model – The name of the model to use for evaluation. Defaults to “gpt-3.5-turbo”.

• scoring_metrics – List of metrics to calculate during evaluation. The LLM input and output will be passed as arguments to each metric score(…) method.

• scoring_functions –

  List of scorer functions to be executed during evaluation. Each scorer function includes a scoring method that accepts predefined arguments supplied by the evaluation engine:

  • dataset_item — a dictionary containing the dataset item content,

  • task_outputs — a dictionary containing the LLM task output.

  • task_span - the data collected during the LLM task execution [optional].

• experiment_name_prefix – The prefix to be added to automatically generated experiment names to make them unique but grouped under the same prefix. For example, if you set experiment_name_prefix=”my-experiment”, the first experiment created will be named my-experiment-<unique-random-part>.

• experiment_name – name of the experiment.

• project_name – Deprecated. If the dataset has a project_name set, it is always used and this override is ignored (with a warning). If the dataset has no project_name, traces and spans are logged to this project (or to Default Project when omitted).

• experiment_config – configuration of the experiment.

• verbose – an integer value that controls evaluation output logs such as summary and tqdm progress bar.

• nb_samples – number of samples to evaluate.

• task_threads – amount of thread workers to run scoring metrics.

• prompt – Prompt object to link with experiment.

• dataset_item_ids – list of dataset item ids to evaluate. If not provided, all samples in the dataset will be evaluated.

• dataset_sampler – An instance of a dataset sampler that will be used to sample dataset items for evaluation. If not provided, all samples in the dataset will be evaluated.

• trial_count – number of times to execute the prompt and evaluate the LLM output for every dataset item.

• experiment_scoring_functions – List of callable functions that compute experiment-level scores. Each function takes a list of TestResult objects and returns a list of ScoreResult objects. These scores are computed after all test results are collected and represent aggregate metrics across the entire experiment.

• experiment_tags – List of tags to be associated with the experiment.

• dataset_filter_string –

  Optional OQL filter string to filter dataset items. Supports filtering by tags, data fields, metadata, etc.

  Supported columns include: - id, source, trace_id, span_id: String fields - data: Dictionary field (use dot notation, e.g., “data.category”) - tags: List field (use “contains” operator) - created_at, last_updated_at: DateTime fields (ISO 8601 format) - created_by, last_updated_by: String fields

  Examples: - tags contains “failed” - Items with ‘failed’ tag - data.category = “test” - Items with specific data field value - created_at >= “2024-01-01T00:00:00Z” - Items created after date