evaluate

opik.evaluation.evaluate(dataset: Dataset | DatasetVersion, task: Callable[[Dict[str, Any]], Dict[str, Any]], scoring_metrics: List[BaseMetric] | None = None, scoring_functions: List[ScorerFunctionProtocol] | None = None, experiment_name_prefix: str | None = None, experiment_name: str | None = None, project_name: str | None = None, experiment_config: Dict[str, Any] | None = None, verbose: int = 1, nb_samples: int | None = None, task_threads: int = 16, prompt: BasePrompt | None = None, prompts: List[BasePrompt] | None = None, scoring_key_mapping: Dict[str, str | Callable[[Dict[str, Any]], Any]] | None = None, dataset_item_ids: List[str] | None = None, dataset_sampler: BaseDatasetSampler | None = None, trial_count: int = 1, experiment_scoring_functions: List[Callable[[List[TestResult]], ScoreResult | List[ScoreResult]]] | None = None, experiment_tags: List[str] | None = None, dataset_filter_string: str | None = None, blueprint_id: str | None = None) → EvaluationResult

Performs task evaluation on a given dataset. You can use either scoring_metrics or scorer_functions to calculate evaluation metrics. The scorer functions doesn’t require scoring_key_mapping and use reserved parameters to receive inputs and outputs from the task.

Parameters:

  • dataset – An Opik Dataset or DatasetVersion instance

  • task – A callable object that takes dict with dataset item content as input and returns dict which will later be used for scoring.

  • experiment_name_prefix – The prefix to be added to automatically generated experiment names to make them unique but grouped under the same prefix. For example, if you set experiment_name_prefix=”my-experiment”, the first experiment created will be named my-experiment-<unique-random-part>.

  • experiment_name – The name of the experiment associated with evaluation run. If None, a generated name will be used.

  • project_name – Deprecated. If the dataset has a project_name set, it is always used and this override is ignored (with a warning). If the dataset has no project_name, traces and spans are logged to this project (or to Default Project when omitted).

  • experiment_config – The dictionary with parameters that describe experiment

  • scoring_metrics – List of metrics to calculate during evaluation. Each metric has score(…) method, arguments for this method are taken from the task output, check the signature of the score method in metrics that you need to find out which keys are mandatory in task-returned dictionary. If no value provided, the experiment won’t have any scoring metrics.

  • scoring_functions –

    List of scorer functions to be executed during evaluation. Each scorer function includes a scoring method that accepts predefined arguments supplied by the evaluation engine:

    • dataset_item — a dictionary containing the dataset item content,

    • task_outputs — a dictionary containing the LLM task output.

    • task_span - the data collected during the LLM task execution [optional].

  • verbose – an integer value that controls evaluation output logs such as summary and tqdm progress bar. 0 - no outputs, 1 - outputs are enabled (default), 2 - outputs are enabled and detailed statistics are displayed.

  • nb_samples – number of samples to evaluate. If no value is provided, all samples in the dataset will be evaluated.

  • task_threads – number of thread workers to run tasks. If set to 1, no additional threads are created, all tasks executed in the current thread sequentially. are executed sequentially in the current thread. Use more than 1 worker if your task object is compatible with sharing across threads.

  • prompt – Prompt object to link with experiment. Deprecated, use prompts argument instead.

  • prompts – A list of Prompt objects to link with experiment.

  • scoring_key_mapping – A dictionary that allows you to rename keys present in either the dataset item or the task output so that they match the keys expected by the scoring metrics. For example if you have a dataset item with the following content: {“user_question”: “What is Opik ?”} and a scoring metric that expects a key “input”, you can use scoring_key_mapping {“input”: “user_question”} to map the “user_question” key to “input”.

  • dataset_item_ids – list of dataset item ids to evaluate. If not provided, all samples in the dataset will be evaluated.

  • dataset_sampler – An instance of a dataset sampler that will be used to sample dataset items for evaluation. If not provided, all samples in the dataset will be evaluated.

  • trial_count – number of times to run the task and evaluate the task output for every dataset item.

  • experiment_scoring_functions – List of callable functions that compute experiment-level scores. Each function takes a list of TestResult objects and returns a list of ScoreResult objects. These scores are computed after all test results are collected and represent aggregate metrics across the entire experiment.

  • experiment_tags – Optional list of tags to associate with the experiment.

  • dataset_filter_string –

    Optional OQL filter string to filter dataset items. Supports filtering by tags, data fields, metadata, etc.

    Supported columns include: - id, source, trace_id, span_id: String fields - data: Dictionary field (use dot notation, e.g., “data.category”) - tags: List field (use “contains” operator) - created_at, last_updated_at: DateTime fields (ISO 8601 format) - created_by, last_updated_by: String fields

    Examples: - tags contains “failed” - Items with ‘failed’ tag - data.category = “test” - Items with specific data field value - created_at >= “2024-01-01T00:00:00Z” - Items created after date