Conversation LLM Judges

These evaluators wrap GEval-style LLM judges so you can score full conversations without manually extracting turns. They expect transcripts in the same format used by ConversationThreadMetric and typically rely on an OpenAI- or Azure-compatible backend. Refer to the relevant Fern guides for API keys, rate limits, and pricing considerations.

Core Conversation Judges

class opik.evaluation.metrics.GEvalConversationMetric(judge: BaseMetric, name: str | None = None)

Bases: ConversationThreadMetric

Wrap a GEval-style judge so it can evaluate an entire conversation transcript.

The wrapper extracts the latest assistant turn from the conversation and sends it to the provided judge. Results are normalised into a ScoreResult so they can plug into the wider Opik evaluation pipeline. Any errors raised by the underlying judge are captured and reported as a failed score computation.

The conversation input should match ConversationThreadMetric semantics—a list of dicts containing role and content keys ordered by time.

Parameters:

  • judge – A GEval-compatible metric instance that accepts output as its primary argument.

  • name – Optional override for the metric name used in results. When None the name is derived from the wrapped judge.

Returns:

Mirrors the wrapped judge’s score/value/metadata fields. When the judge fails, scoring_failed is set and value is 0.0.

Return type:

ScoreResult

Example

>>> from opik.evaluation.metrics.conversation.llm_judges.g_eval_wrappers import (
...     GEvalConversationMetric,
... )
>>> from opik.evaluation.metrics.llm_judges.g_eval_presets.qa_suite import DialogueHelpfulnessJudge
>>> conversation = [
...     {"role": "user", "content": "Summarise these notes."},
...     {"role": "assistant", "content": "Here is a concise summary..."},
... ]
>>> metric = GEvalConversationMetric(judge=DialogueHelpfulnessJudge(model="gpt-4"))
>>> result = metric.score(conversation)
>>> result.value
0.83
__init__(judge: BaseMetric, name: str | None = None) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **_: Any) → ScoreResult

Evaluate the final assistant turn in a conversation.

Parameters:

conversation – Sequence of dict-like turns containing role and content keys. Only assistant turns with non-empty content are considered.

Returns:

Normalised output from the wrapped judge. If no assistant message is present, the result is marked as failed with value=0.0.

Return type:

ScoreResult

class opik.evaluation.metrics.ConversationalCoherenceMetric(model: str | OpikBaseModel | None = None, name: str = 'conversational_coherence_score', include_reason: bool = True, track: bool = True, project_name: str | None = None, window_size: int = 10, temperature: float = 1e-8, reasoning_effort: str | None = None)

Bases: ConversationThreadMetric

Calculates the conversational coherence metric for a given conversation thread. This metric assesses the coherence and relevance across a series of conversation turns by evaluating the consistency in responses, logical flow, and overall context maintenance. It evaluates whether the conversation session felt like a natural, adaptive, helpful interaction.

The ConversationalCoherenceMetric builds a sliding window of dialogue turns for each turn in the conversation. It then uses a language model to evaluate whether the final assistant message within each window is relevant and coherent in relation to the preceding conversational context.

It supports both synchronous and asynchronous operations to accommodate the model’s operation type. It returns a score between 0.0 and 1.0, where 0.0 indicates a low coherence score and 1.0 indicates a high coherence score.

Parameters:

  • model – The model to use for evaluating the conversation. If a string is provided, it will be used to fetch the model from the LiteLLM API. If a base_model.OpikBaseModel is provided, it will be used directly. Default is None.

  • name – The name of the metric. The default is “conversational_coherence_score”.

  • include_reason – Whether to include the reason for the score in the result. Default is True.

  • track – Whether to track the metric. Default is True.

  • project_name – The name of the project to track the metric in. Default is None.

  • window_size – The window size to use for calculating the score. It defines the maximal number of historical turns to include in each window when assessing the coherence of the current turn in the conversation. Default is 10.

  • temperature – The temperature to use for the model. Defaults to 1e-8.

  • reasoning_effort – Optional reasoning effort level for the model. Applies to providers/models that expose a reasoning_effort parameter (e.g. OpenAI gpt-5 family). Supported values typically include “minimal”, “low”, “medium”, “high”. Defaults to None (provider default applies — typically “medium” for OpenAI reasoning models). Pass explicitly if you want to cut token spend on reasoning.

Example

>>> from opik.evaluation.metrics import ConversationalCoherenceMetric
>>> conversation = [
>>>     {"role": "user", "content": "Hello!"},
>>>     {"role": "assistant", "content": "Hi there!"},
>>>     {"role": "user", "content": "How are you?"},
>>>     {"role": "assistant", "content": "I'm doing well, thank you!"},
>>> ]
>>> metric = ConversationalCoherenceMetric()
>>> result = metric.score(conversation)
>>> if result.scoring_failed:
>>>     print(f"Scoring failed: {result.reason}")
>>> else:
>>>     print(result.value)
__init__(model: str | OpikBaseModel | None = None, name: str = 'conversational_coherence_score', include_reason: bool = True, track: bool = True, project_name: str | None = None, window_size: int = 10, temperature: float = 1e-8, reasoning_effort: str | None = None)
score(conversation: List[Dict[Literal['role', 'content'], str]], **ignored_kwargs: Any) → ScoreResult

Evaluate a conversation and return a score.

Parameters:

  • conversation – A list of conversation messages. Each message is a dictionary with ‘role’ (either ‘user’ or ‘assistant’) and ‘content’ (the message text).

  • **kwargs – Additional keyword arguments that may be used by specific metric implementations.

Returns:

A ScoreResult object or list of ScoreResult objects containing the evaluation score, metric name, and optional reasoning.

class opik.evaluation.metrics.SessionCompletenessQuality(model: str | OpikBaseModel | None = None, name: str = 'session_completeness_quality', include_reason: bool = True, track: bool = True, project_name: str | None = None, temperature: float = 1e-8, reasoning_effort: str | None = None)

Bases: ConversationThreadMetric

Represents the Session Completeness Quality metric for a conversation thread.

This class is a specific implementation of the ConversationThreadMetric that evaluates the completeness of a session within a conversation thread. The metric is used to judge how well a session addresses the intended context or purpose of the conversation.

The process begins by using an LLM to extract a list of high-level user intentions from the conversation turns. The same LLM is then used to assess whether each intention was addressed and/or fulfilled over the course of the conversation. It returns a score between 0.0 and 1.0, where higher values indicate better session completeness.

Parameters:

  • model – The language model to use for evaluation.

  • name – The name of the metric. The default is “session_completeness_quality”.

  • include_reason – Whether to include a reason for the score.

  • track – Whether to track the metric. Default is True.

  • project_name – The project name to track the metric in.

  • temperature – The temperature to use for the model. Defaults to 1e-8.

  • reasoning_effort – Optional reasoning effort level for the model. Applies to providers/models that expose a reasoning_effort parameter (e.g. OpenAI gpt-5 family). Supported values typically include “minimal”, “low”, “medium”, “high”. Defaults to None (provider default applies — typically “medium” for OpenAI reasoning models). Pass explicitly if you want to cut token spend on reasoning.

Example

>>> from opik.evaluation.metrics import SessionCompletenessQuality
>>> conversation = [
>>>     {"role": "user", "content": "Hello!"},
>>>     {"role": "assistant", "content": "Hi there!"},
>>>     {"role": "user", "content": "How are you?"},
>>>     {"role": "assistant", "content": "I'm doing well, thank you!"},
>>> ]
>>> metric = SessionCompletenessQuality()
>>> result = metric.score(conversation)
>>> if result.scoring_failed:
>>>     print(f"Scoring failed: {result.reason}")
>>> else:
>>>     print(result.value)
0.95
__init__(model: str | OpikBaseModel | None = None, name: str = 'session_completeness_quality', include_reason: bool = True, track: bool = True, project_name: str | None = None, temperature: float = 1e-8, reasoning_effort: str | None = None)
score(conversation: List[Dict[Literal['role', 'content'], str]], **ignored_kwargs: Any) → ScoreResult

Evaluate a conversation and return a score.

Parameters:

  • conversation – A list of conversation messages. Each message is a dictionary with ‘role’ (either ‘user’ or ‘assistant’) and ‘content’ (the message text).

  • **kwargs – Additional keyword arguments that may be used by specific metric implementations.

Returns:

A ScoreResult object or list of ScoreResult objects containing the evaluation score, metric name, and optional reasoning.

class opik.evaluation.metrics.UserFrustrationMetric(model: str | OpikBaseModel | None = None, name: str = 'user_frustration_score', include_reason: bool = True, track: bool = True, project_name: str | None = None, window_size: int = 10, temperature: float = 1e-8, reasoning_effort: str | None = None)

Bases: ConversationThreadMetric

A heuristic score estimating the likelihood that the user experienced confusion, annoyance, or disengagement during the session — due to repetition, lack of adaptation, ignored intent signals, or failure to smoothly conclude.

The UserFrustrationMetric class integrates with LLM models to analyze conversation data in sliding windows and produce a numerical score along with an optional reason for the calculated score. It provides both synchronous and asynchronous methods for calculation and supports customization through attributes like window size and reason inclusion.

This metric can be used to monitor and track user frustration levels during conversations, enabling insights into user experience. The metric makes use of LLM models to score conversational windows and summarize results. It returns a score between 0.0 and 1.0. The higher the score, the more frustrated the user is likely to be.

Parameters:

  • model – The model to use for evaluating the conversation. If a string is provided, it will be used to fetch the model from the LiteLLM API. If a base_model.OpikBaseModel is provided, it will be used directly. Default is None.

  • name – The name of the metric. The default is “user_frustration_score”.

  • include_reason – Whether to include the reason for the score in the result. Default is True.

  • track – Whether to track the metric. Default is True.

  • project_name – The name of the project to track the metric in. Default is None.

  • window_size – The window size to use for calculating the score. It defines the maximal number of historical turns to include in each window when assessing the frustration of the current turn in the conversation. Default is 10.

  • temperature – The temperature to use for the model. Defaults to 1e-8.

  • reasoning_effort – Optional reasoning effort level for the model. Applies to providers/models that expose a reasoning_effort parameter (e.g. OpenAI gpt-5 family). Supported values typically include “minimal”, “low”, “medium”, “high”. Defaults to None (provider default applies — typically “medium” for OpenAI reasoning models). Pass explicitly if you want to cut token spend on reasoning.

Example

>>> from opik.evaluation.metrics import UserFrustrationMetric
>>> conversation = [
>>>     {"role": "user", "content": "How do I center a div using CSS?"},
>>>     {"role": "assistant", "content": "There are many ways to center elements in CSS."},
>>>     {"role": "user", "content": "Okay... can you show me one?"},
>>>     {"role": "assistant", "content": "Sure. It depends on the context — are you centering horizontally, vertically, or both?"},
>>>     {"role": "user", "content": "Both. Just give me a basic example."},
>>> ]
>>> metric = UserFrustrationMetric()
>>> result = metric.score(conversation)
>>> if result.scoring_failed:
>>>     print(f"Scoring failed: {result.reason}")
>>> else:
>>>     print(result.value)
__init__(model: str | OpikBaseModel | None = None, name: str = 'user_frustration_score', include_reason: bool = True, track: bool = True, project_name: str | None = None, window_size: int = 10, temperature: float = 1e-8, reasoning_effort: str | None = None)
score(conversation: List[Dict[Literal['role', 'content'], str]], **ignored_kwargs: Any) → ScoreResult

Evaluate a conversation and return a score.

Parameters:

  • conversation – A list of conversation messages. Each message is a dictionary with ‘role’ (either ‘user’ or ‘assistant’) and ‘content’ (the message text).

  • **kwargs – Additional keyword arguments that may be used by specific metric implementations.

Returns:

A ScoreResult object or list of ScoreResult objects containing the evaluation score, metric name, and optional reasoning.

Specialized Variants

class opik.evaluation.metrics.ConversationComplianceRiskMetric(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalConversationMetric

Evaluate the latest assistant response for compliance and risk exposure.

This metric forwards the final assistant turn to ComplianceRiskJudge and returns its assessment as a conversation-level ScoreResult.

Parameters:

  • model – Optional model name or identifier understood by the judge.

  • track – Whether to automatically track metric results. Defaults to True.

  • project_name – Optional tracking project name. Defaults to None.

  • temperature – Sampling temperature supplied to the underlying judge model.

Returns:

Compliance score emitted by the wrapped judge; failed evaluations set scoring_failed and value=0.0.

Return type:

ScoreResult

Example

>>> from opik.evaluation.metrics import ConversationComplianceRiskMetric
>>> conversation = [
...     {"role": "user", "content": "Generate an employment contract."},
...     {"role": "assistant", "content": "Here is a standard contract template..."},
... ]
>>> metric = ConversationComplianceRiskMetric(model="gpt-4")
>>> result = metric.score(conversation)
>>> result.value
0.12
__init__(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **_: Any) → ScoreResult

Evaluate the final assistant turn in a conversation.

Parameters:

conversation – Sequence of dict-like turns containing role and content keys. Only assistant turns with non-empty content are considered.

Returns:

Normalised output from the wrapped judge. If no assistant message is present, the result is marked as failed with value=0.0.

Return type:

ScoreResult

class opik.evaluation.metrics.ConversationDialogueHelpfulnessMetric(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalConversationMetric

Score how helpful the closing assistant message is within the dialogue.

The metric expects the same conversation shape as ConversationThreadMetric. It uses DialogueHelpfulnessJudge to evaluate usefulness and responsiveness of the final assistant turn.

Parameters:

  • model – Optional model name passed to the judge.

  • track – Whether to automatically track results. Defaults to True.

  • project_name – Optional tracking project. Defaults to None.

  • temperature – Temperature fed into the judge’s underlying model.

Returns:

Helpfulness score from the wrapped judge.

Return type:

ScoreResult

Example

>>> from opik.evaluation.metrics import ConversationDialogueHelpfulnessMetric
>>> conversation = [
...     {"role": "user", "content": "How do I reset my password?"},
...     {"role": "assistant", "content": "Click the reset link and follow the steps."},
... ]
>>> metric = ConversationDialogueHelpfulnessMetric(model="gpt-4")
>>> result = metric.score(conversation)
>>> result.value
0.88
__init__(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **_: Any) → ScoreResult

Evaluate the final assistant turn in a conversation.

Parameters:

conversation – Sequence of dict-like turns containing role and content keys. Only assistant turns with non-empty content are considered.

Returns:

Normalised output from the wrapped judge. If no assistant message is present, the result is marked as failed with value=0.0.

Return type:

ScoreResult

class opik.evaluation.metrics.ConversationQARelevanceMetric(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalConversationMetric

Quantify how relevant the assistant’s final answer is to the preceding query.

This metric expects a conversation sequence compatible with ConversationThreadMetric and wraps QARelevanceJudge and is useful when the conversation emulates a Q&A exchange.

Parameters:

  • model – Optional model name used by the judge backend.

  • track – Whether to automatically track outcomes. Defaults to True.

  • project_name – Optional project for tracked scores. Defaults to None.

  • temperature – Judge sampling temperature.

Returns:

Relevance score from the judge.

Return type:

ScoreResult

Example

>>> from opik.evaluation.metrics import ConversationQARelevanceMetric
>>> conversation = [
...     {"role": "user", "content": "Who wrote Dune?"},
...     {"role": "assistant", "content": "Frank Herbert wrote Dune."},
... ]
>>> metric = ConversationQARelevanceMetric(model="gpt-4")
>>> result = metric.score(conversation)
>>> result.value
1.0
__init__(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **_: Any) → ScoreResult

Evaluate the final assistant turn in a conversation.

Parameters:

conversation – Sequence of dict-like turns containing role and content keys. Only assistant turns with non-empty content are considered.

Returns:

Normalised output from the wrapped judge. If no assistant message is present, the result is marked as failed with value=0.0.

Return type:

ScoreResult

class opik.evaluation.metrics.ConversationSummarizationCoherenceMetric(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalConversationMetric

Assess the coherence of a summary-style assistant response.

The metric expects the conversation schema defined by ConversationThreadMetric and invokes SummarizationCoherenceJudge to rate whether the summary flows naturally and captures the conversation structure.

Parameters:

  • model – Optional model name or identifier for the judge.

  • track – Whether to track metric results automatically. Defaults to True.

  • project_name – Optional project name for tracked scores. Defaults to None.

  • temperature – Sampling temperature passed to the judge model.

Returns:

Coherence score from the judge.

Return type:

ScoreResult

Example

>>> from opik.evaluation.metrics import ConversationSummarizationCoherenceMetric
>>> conversation = [
...     {"role": "user", "content": "Summarise this chat."},
...     {"role": "assistant", "content": "Summary: we discussed timelines and budgets."},
... ]
>>> metric = ConversationSummarizationCoherenceMetric(model="gpt-4")
>>> result = metric.score(conversation)
>>> result.value
0.91
__init__(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **_: Any) → ScoreResult

Evaluate the final assistant turn in a conversation.

Parameters:

conversation – Sequence of dict-like turns containing role and content keys. Only assistant turns with non-empty content are considered.

Returns:

Normalised output from the wrapped judge. If no assistant message is present, the result is marked as failed with value=0.0.

Return type:

ScoreResult

class opik.evaluation.metrics.ConversationSummarizationConsistencyMetric(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalConversationMetric

Check whether a dialogue summary stays faithful to the source turns.

The metric assumes the standard conversation schema and delegates scoring to SummarizationConsistencyJudge and reports the result at the conversation level.

Parameters:

  • model – Optional model name passed through to the judge.

  • track – Whether to automatically track results. Defaults to True.

  • project_name – Optional tracking project. Defaults to None.

  • temperature – Temperature parameter supplied to the judge model.

Returns:

Consistency score from the judge.

Return type:

ScoreResult

Example

>>> from opik.evaluation.metrics import ConversationSummarizationConsistencyMetric
>>> conversation = [
...     {"role": "user", "content": "Give me a summary."},
...     {"role": "assistant", "content": "Summary: project ships next week."},
... ]
>>> metric = ConversationSummarizationConsistencyMetric(model="gpt-4")
>>> result = metric.score(conversation)
>>> result.value
0.95
__init__(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **_: Any) → ScoreResult

Evaluate the final assistant turn in a conversation.

Parameters:

conversation – Sequence of dict-like turns containing role and content keys. Only assistant turns with non-empty content are considered.

Returns:

Normalised output from the wrapped judge. If no assistant message is present, the result is marked as failed with value=0.0.

Return type:

ScoreResult

class opik.evaluation.metrics.ConversationPromptUncertaintyMetric(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0)

Bases: GEvalConversationMetric

Measure how uncertain the assistant appears about executing the prompt.

The metric expects the standard conversation schema and pipes the latest assistant reply into PromptUncertaintyJudge and returns the judge’s score in a conversation-friendly format.

Parameters:

  • model – Optional model name for the judge to use.

  • track – Whether to automatically track the metric. Defaults to True.

  • project_name – Optional tracking project. Defaults to None.

  • temperature – Sampling temperature for the judge model.

Returns:

Uncertainty score from the judge.

Return type:

ScoreResult

Example

>>> from opik.evaluation.metrics import ConversationPromptUncertaintyMetric
>>> conversation = [
...     {"role": "user", "content": "Follow the brief precisely."},
...     {"role": "assistant", "content": "I'm not fully certain which part to prioritise."},
... ]
>>> metric = ConversationPromptUncertaintyMetric(model="gpt-4")
>>> result = metric.score(conversation)
>>> result.value
0.42
__init__(model: str | None = None, track: bool = True, project_name: str | None = None, temperature: float = 0.0) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **_: Any) → ScoreResult

Evaluate the final assistant turn in a conversation.

Parameters:

conversation – Sequence of dict-like turns containing role and content keys. Only assistant turns with non-empty content are considered.

Returns:

Normalised output from the wrapped judge. If no assistant message is present, the result is marked as failed with value=0.0.

Return type:

ScoreResult