evaluate_experiment

opik.evaluation.evaluate_experiment(experiment_name: str, scoring_metrics: List[BaseMetric], scoring_functions: List[ScorerFunctionProtocol] | None = None, scoring_threads: int = 16, verbose: int = 1, scoring_key_mapping: Dict[str, str | Callable[[Dict[str, Any]], Any]] | None = None, experiment_id: str | None = None, experiment_scoring_functions: List[Callable[[List[TestResult]], ScoreResult | List[ScoreResult]]] | None = None, project_name: str | None = None) → EvaluationResult

Update the existing experiment with new evaluation metrics. You can use either scoring_metrics or scorer_functions to calculate evaluation metrics. The scorer functions doesn’t require scoring_key_mapping and use reserved parameters to receive inputs and outputs from the task. The experiment requires at least one test case.

Parameters:

• experiment_name – The name of the experiment to update.

• scoring_metrics – List of metrics to calculate during evaluation. Each metric has score(…) method, arguments for this method are taken from the task output, check the signature of the score method in metrics that you need to find out which keys are mandatory in task-returned dictionary.

• scoring_functions –

  List of scorer functions to be executed during evaluation. Each scorer function includes a scoring method that accepts predefined arguments supplied by the evaluation engine:

  • dataset_item — a dictionary containing the dataset item content,

  • task_outputs — a dictionary containing the LLM task output.

  • task_span - the data collected during the LLM task execution [optional].

• scoring_threads – amount of thread workers to run scoring metrics.

• verbose – an integer value that controls evaluation output logs such as summary and tqdm progress bar.

• scoring_key_mapping – A dictionary that allows you to rename keys present in either the dataset item or the task output so that they match the keys expected by the scoring metrics. For example, if you have a dataset item with the following content: {“user_question”: “What is Opik ?”} and a scoring metric that expects a key “input”, you can use scoring_key_mapping {“input”: “user_question”} to map the “user_question” key to “input”.

• experiment_id – The ID of the experiment to evaluate. If not provided, the experiment will be evaluated based on the experiment name.

• experiment_scoring_functions – List of callable functions that compute experiment-level scores. Each function takes a list of TestResult objects and returns a list of ScoreResult objects. These scores are computed after all test results are collected and represent aggregate metrics across the entire experiment.

• project_name – The name of the project to which the experiment belongs. If not provided, the default project will be used.