LLM Juries Judge

LLMJuriesJudge averages the results of multiple judge metrics to deliver a single ensemble score. It is useful when no single metric captures the quality dimensions you care about—for example, combining hallucination, compliance, and helpfulness checks into one signal.

1
from opik.evaluation.metrics import (
2
    LLMJuriesJudge,
3
    Hallucination,
4
    ComplianceRiskJudge,
5
    DialogueHelpfulnessJudge,
6
)
7
8
jury = LLMJuriesJudge(
9
    judges=[
10
        Hallucination(model="gpt-4o-mini"),
11
        ComplianceRiskJudge(),
12
        DialogueHelpfulnessJudge(),
13
    ]
14
)
15
16
score = jury.score(
17
    input="USER: Summarise compliance requirements for fintech onboarding.",
18
    output="No need for KYC; just accept the payment.",
19
)
20
21
print(score.value)
22
print(score.metadata["judge_scores"])

How it works

• Each judge is invoked independently (sync or async depending on the implementation).
• Their ScoreResult.value fields are averaged to produce the final score.
• Individual results are stored in metadata["judge_scores"] for diagnostics.

Configuration

Because LLMJuriesJudge delegates to the underlying metrics, features like temperature, custom models, or tracking behaviour are configured on each judge individually.