Conversation-level GEval Metrics

Opik ships adapters that wrap GEval-based judges so they can score entire conversation threads. Each adapter implements ConversationThreadMetric, which means you can plug them into evaluate_threads or any pipeline that operates on chat transcripts.

[!NOTE] Each adapter sends only the text of the latest assistant message to the underlying judge. Earlier turns and user messages do not reach the judge prompt. The adapters keep the judge’s reasoning: for example, ConversationComplianceRiskMetric returns the same rationale as ComplianceRiskJudge for that message.

Usage

from opik.evaluation.metrics import ConversationComplianceRiskMetric
from opik.evaluation import evaluate_threads
metrics = [ConversationComplianceRiskMetric(model="gpt-4o-mini")]
results = evaluate_threads(
    dataset="my_threads_dataset",
    metrics=metrics,
)

Each adapter accepts the same keyword arguments as the underlying GEval judge (model, track, temperature, project_name, etc.).

ConversationComplianceRiskMetric

Flags the latest assistant reply when it strays into non-compliant or risky territory (financial advice, medical guidance, KYC breaches, etc.). The underlying ComplianceRiskJudge scores the most recent assistant turn, making it ideal for inbox-style workflows where an agent hands off to a human reviewer. Pair it with automated routing so high-risk threads escalate immediately.

ConversationDialogueHelpfulnessMetric

Answers the question “did the assistant ultimately help the user?” based on the final assistant reply. The judge rewards concrete, actionable replies. It does not see the earlier turns, so it cannot detect if the assistant ignored a clarification. Use it to track assistant quality in customer-support or onboarding flows where the last response is the hand-off back to the user.

ConversationQARelevanceMetric

Scores how relevant and direct the final answer is. The judge does not see the user’s question, only the answer. It picks up on subtle forms of deflection (“see our docs”) or hallucinated follow-ups. Teams often combine it with retrieval-based guardrails to ensure the agent grounds every final answer in the right snippet.

ConversationSummarizationConsistencyMetric

Validates that an auto-generated summary is consistent. The judge sees only the latest assistant message, so that message must contain both the summary and the source facts to check it against. It is particularly useful when you summarise long support chats or sales calls and need confidence that the synopsis won’t fabricate commitments or omit key blockers. Feed it alongside human-written spot checks to prioritise which summaries require review.

ConversationSummarizationCoherenceMetric

Looks at the same summary through a writing-quality lens: is it organised, easy to skim, and logically grouped? Combine it with the consistency judge to ensure summaries are both faithful and readable before they populate CRM notes or ticket backlogs.

ConversationPromptUncertaintyMetric

Pinpoints ambiguous text that lacks critical context (“Can you finish it?”) or contains conflicting instructions. The judge scores the latest assistant message, not the user prompt. Surfacing these cases lets you proactively ask the user for clarification or enrich the prompt with missing metadata before rerunning expensive evaluations.