LLM Judge Presets

These presets wrap common GEval prompt templates so you can instantiate judges with one line of code. Provide an appropriate model (for example, "gpt-4o") and ensure the backing provider is configured via opik.configure.

Agent & Task Presets

class opik.evaluation.metrics.AgentTaskCompletionJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Evaluate whether an agent successfully completed the original task.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track judge outputs. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature for the underlying model.

Example

>>> from opik.evaluation.metrics import AgentTaskCompletionJudge
>>> judge = AgentTaskCompletionJudge(model="gpt-4")
>>> result = judge.score(output="Agent delivered the requested summary.")
>>> result.value
0.9
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.AgentToolCorrectnessJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Judge whether an agent invoked and interpreted tools correctly.

Parameters:

  • model – Optional model identifier or pre-configured OpikBaseModel.

  • track – Whether to automatically track judge outputs. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature supplied to the underlying model.

Example

>>> from opik.evaluation.metrics import AgentToolCorrectnessJudge
>>> judge = AgentToolCorrectnessJudge(model="gpt-4")
>>> transcript = "Agent called search_tool and used the answer correctly."
>>> result = judge.score(output=transcript)
>>> result.value
0.8
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

Conversation Quality Presets

class opik.evaluation.metrics.DialogueHelpfulnessJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Judge how helpful an assistant reply is within a dialogue.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track judge outputs. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the underlying model.

Example

>>> from opik.evaluation.metrics import DialogueHelpfulnessJudge
>>> judge = DialogueHelpfulnessJudge(model="gpt-4")
>>> result = judge.score(output="Assistant politely refuses without help.")
>>> result.value
0.3
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.QARelevanceJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Check whether an answer directly addresses the user question.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track judge outputs. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the underlying model.

Example

>>> from opik.evaluation.metrics import QARelevanceJudge
>>> judge = QARelevanceJudge(model="gpt-4")
>>> result = judge.score(output="Answer rambles without addressing the ask.")
>>> result.value
0.2
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.SummarizationCoherenceJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Evaluate the coherence and structure of generated summaries.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track judge outputs. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the underlying model.

Example

>>> from opik.evaluation.metrics import SummarizationCoherenceJudge
>>> judge = SummarizationCoherenceJudge(model="gpt-4")
>>> result = judge.score(output="Summary jumps between unrelated topics.")
>>> result.value
0.5
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.SummarizationConsistencyJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Score how faithful a summary is to its source content.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track judge outputs. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the underlying model.

Example

>>> from opik.evaluation.metrics import SummarizationConsistencyJudge
>>> judge = SummarizationConsistencyJudge(model="gpt-4")
>>> result = judge.score(output="Summary omits key fact.")
>>> result.value
0.4
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.PromptUncertaintyJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Rate how ambiguous or underspecified a prompt feels to the model.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track judge outputs. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the underlying model.

Example

>>> from opik.evaluation.metrics import PromptUncertaintyJudge
>>> judge = PromptUncertaintyJudge(model="gpt-4")
>>> result = judge.score(output="Do the right thing in the best way possible.")
>>> result.value
0.8
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

Risk & Bias Presets

class opik.evaluation.metrics.ComplianceRiskJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Evaluate responses for non-compliant or misleading claims in regulated sectors.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track judge outputs. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the underlying model.

Example

>>> from opik.evaluation.metrics import ComplianceRiskJudge
>>> judge = ComplianceRiskJudge(model="gpt-4")
>>> result = judge.score(output="This pill cures diabetes in a week.")
>>> result.value
0.97
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.DemographicBiasJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Score demographic stereotyping or bias in a response.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track results. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the model.

Example

>>> from opik.evaluation.metrics import DemographicBiasJudge
>>> judge = DemographicBiasJudge(model="gpt-4")
>>> result = judge.score(output="People from X group are always late.")
>>> result.value
0.95
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.GenderBiasJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Detect gender stereotyping or exclusion in generated text.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track results. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the model.

Example

>>> from opik.evaluation.metrics import GenderBiasJudge
>>> judge = GenderBiasJudge(model="gpt-4")
>>> result = judge.score(output="Women are naturally worse at math.")
>>> result.value
0.93
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.PoliticalBiasJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Detect partisan or ideological bias in a response.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track results. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the model.

Example

>>> from opik.evaluation.metrics import PoliticalBiasJudge
>>> judge = PoliticalBiasJudge(model="gpt-4")
>>> result = judge.score(output="Vote for candidate X because Y is corrupt")
>>> result.value
0.87
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.RegionalBiasJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Assess geographic or cultural bias in responses.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track results. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the model.

Example

>>> from opik.evaluation.metrics import RegionalBiasJudge
>>> judge = RegionalBiasJudge(model="gpt-4")
>>> result = judge.score(output="People from region Z are lazy.")
>>> result.value
0.88
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult

class opik.evaluation.metrics.ReligiousBiasJudge(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalPreset

Evaluate responses for religious bias or disrespectful language.

Parameters:

  • model – Optional model identifier or OpikBaseModel instance.

  • track – Whether to automatically track results. Defaults to True.

  • project_name – Optional tracking project name.

  • temperature – Sampling temperature forwarded to the model.

Example

>>> from opik.evaluation.metrics import ReligiousBiasJudge
>>> judge = ReligiousBiasJudge(model="gpt-4")
>>> result = judge.score(output="Believers of X are all foolish.")
>>> result.value
0.9
__init__(model: str | OpikBaseModel | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(output: str, **ignored_kwargs: Any) → ScoreResult

Calculate the G-Eval score for the given LLM’s output.

Parameters:

  • output – The LLM’s output to evaluate.

  • **ignored_kwargs – Additional keyword arguments that are ignored.

Returns:

A ScoreResult object containing the G-Eval score (between 0.0 and 1.0) and a reason for the score.

Return type:

score_result.ScoreResult