LLM Juries

Use LLMJuriesJudge when you want to call multiple judges and aggregate the results into a single decision. Provide a list of judge instances plus an aggregation strategy (majority vote, max confidence, etc.).

class opik.evaluation.metrics.LLMJuriesJudge(judges: Iterable[BaseMetric], name: str = 'llm_juries_judge', track: bool = True, project_name: str | None = None)

Bases: BaseMetric

Aggregate multiple judge metrics into a consensus score.

Parameters:

  • judges – Iterable of judge metrics to execute for consensus.

  • name – Display name for the aggregated result. Defaults to "llm_juries_judge".

  • track – Whether to automatically track results. Defaults to True.

  • project_name – Optional tracking project name. Defaults to None.

Example

>>> from opik.evaluation.metrics import LLMJuriesJudge, ComplianceRiskJudge
>>> juries = LLMJuriesJudge(judges=[ComplianceRiskJudge(model="gpt-4")])
>>> result = juries.score(output="Financial guarantees provided.")
>>> result.value
0.6
__init__(judges: Iterable[BaseMetric], name: str = 'llm_juries_judge', track: bool = True, project_name: str | None = None) → None
score(*args: Any, **kwargs: Any) → ScoreResult

Public method that can be called independently.