GEval

class opik.evaluation.metrics.GEval(task_introduction: str, evaluation_criteria: str, model: str | OpikBaseModel | None = None, name: str = 'g_eval_metric', track: bool = True, project_name: str | None = None, temperature: float = 0.0, seed: int | None = None, reasoning_effort: str | None = None)

Bases: BaseMetric

Generalised evaluation metric that prompts an LLM to grade another LLM output.

GEval builds a reusable chain-of-thought using the provided task_introduction and evaluation_criteria prompts, then requests a final score and rationale for each evaluated output.

Parameters:

  • task_introduction – Instruction describing the evaluator’s persona/purpose.

  • evaluation_criteria – Detailed rubric presented to the evaluator.

  • model – Optional model identifier or OpikBaseModel for the judge.

  • name – Display name for the metric result. Defaults to "g_eval_metric".

  • track – Whether to automatically track metric results. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the judge model.

  • seed – Optional seed for reproducible generation (if supported by the model).

  • reasoning_effort – Optional reasoning effort level for the model. Applies to providers/models that expose a reasoning_effort parameter (e.g. OpenAI gpt-5 family). Supported values typically include “minimal”, “low”, “medium”, “high”. Defaults to None (provider default applies — typically “medium” for OpenAI reasoning models). Pass explicitly if you want to cut token spend on reasoning.

Example

>>> from opik.evaluation.metrics.llm_judges.g_eval.metric import GEval
>>> metric = GEval(
...     task_introduction="You evaluate politeness of responses.",
...     evaluation_criteria="Score from 1 (rude) to 5 (very polite).",
...     model="gpt-4",
... )
>>> result = metric.score(output="Thanks so much for your help!")
>>> result.value
0.9
llm_chain_of_thought() → str
async allm_chain_of_thought() → str
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

async ascore(output: str, **ignored_kwargs: Any) → ScoreResult

Async variant of score(), evaluating the provided LLM output using the configured judge model and returning a ScoreResult.