Conversation Heuristic Metrics

Use these fast, rule-based metrics when you want lightweight signals about dialogue quality without invoking an LLM judge. They operate over full conversation transcripts and surface issues such as repetition and missing context.

class opik.evaluation.metrics.ConversationDegenerationMetric(name: str = 'conversation_degeneration_metric', track: bool = True, project_name: str | None = None, ngram_size: int = 3, fallback_phrases: List[str] | None = None)

Bases: ConversationThreadMetric

Score how strongly an assistant conversation shows degeneration or repetition.

The metric inspects each assistant turn, measuring repeated n-grams, overlap with the previous reply, low lexical diversity, and presence of known fallback phrases (for example, “as an AI language model…”). Each turn receives a degeneration score between 0.0 and 1.0; the overall metric reports the peak risk observed so you can quickly flag sections where the assistant got stuck or stopped being helpful. Detailed per-turn diagnostics are returned in the ScoreResult.metadata payload.

Parameters:

  • name – Display name for the metric result. Defaults to "conversation_degeneration_metric".

  • track – Whether the metric should automatically track to an Opik project. Defaults to True.

  • project_name – Optional project to store tracked results in. Defaults to None (inherit global setting).

  • ngram_size – Size of the n-grams used to detect repetition within a single response. Must be at least 2. Defaults to 3.

  • fallback_phrases – Custom list of phrases that should be treated as degeneration signatures. If None, a sensible default list is used.

Example

>>> from opik.evaluation.metrics import ConversationDegenerationMetric
>>> conversation = [
...     {"role": "user", "content": "Can you draft a short bio for Ada?"},
...     {"role": "assistant", "content": "Sure, here is a short bio for Ada."},
...     {"role": "user", "content": "Could you add more detail?"},
...     {"role": "assistant", "content": "Sure, here is a short bio for Ada."},
... ]
>>> metric = ConversationDegenerationMetric(ngram_size=3)
>>> result = metric.score(conversation)
>>> float(result.value)
0.75
__init__(name: str = 'conversation_degeneration_metric', track: bool = True, project_name: str | None = None, ngram_size: int = 3, fallback_phrases: List[str] | None = None) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **ignored_kwargs: object) → ScoreResult

Evaluate a conversation and return a score.

Parameters:

  • conversation – A list of conversation messages. Each message is a dictionary with ‘role’ (either ‘user’ or ‘assistant’) and ‘content’ (the message text).

  • **kwargs – Additional keyword arguments that may be used by specific metric implementations.

Returns:

A ScoreResult object or list of ScoreResult objects containing the evaluation score, metric name, and optional reasoning.

class opik.evaluation.metrics.KnowledgeRetentionMetric(name: str = 'knowledge_retention_metric', track: bool = True, project_name: str | None = None, turns_to_consider: int = 5)

Bases: ConversationThreadMetric

Measure how many user-provided facts resurface in the closing assistant reply.

The metric extracts salient, non-stopword terms from earlier user turns, excluding explicit requests (for example, questions or pleas for help). When the final assistant response omits those key terms, the score drops, signalling the assistant may have forgotten essential context. Scores range from 0.0 (nothing retained) to 1.0 (all extracted facts referenced).

Parameters:

  • name – Display name for the metric result. Defaults to "knowledge_retention_metric".

  • track – Whether the metric should automatically track results. Defaults to True.

  • project_name – Optional Opik project name for tracking. Defaults to None.

  • turns_to_consider – How many of the earliest user turns should be mined for facts. Defaults to 5.

Example

>>> from opik.evaluation.metrics import KnowledgeRetentionMetric
>>> conversation = [
...     {"role": "user", "content": "My new router is a Netgear Nighthawk."},
...     {"role": "assistant", "content": "Great choice!"},
...     {"role": "user", "content": "Please remind me about the Netgear router setup."},
...     {
...         "role": "assistant",
...         "content": "Be sure to update the Netgear Nighthawk firmware first.",
...     },
... ]
>>> metric = KnowledgeRetentionMetric(turns_to_consider=3)
>>> result = metric.score(conversation)
>>> float(result.value)
1.0
__init__(name: str = 'knowledge_retention_metric', track: bool = True, project_name: str | None = None, turns_to_consider: int = 5) → None
score(conversation: List[Dict[Literal['role', 'content'], str]], **ignored_kwargs: object) → ScoreResult

Evaluate a conversation and return a score.

Parameters:

  • conversation – A list of conversation messages. Each message is a dictionary with ‘role’ (either ‘user’ or ‘assistant’) and ‘content’ (the message text).

  • **kwargs – Additional keyword arguments that may be used by specific metric implementations.

Returns:

A ScoreResult object or list of ScoreResult objects containing the evaluation score, metric name, and optional reasoning.