Summarization Consistency Judge

SummarizationConsistencyJudge compares a generated summary with the original document (or transcript) and scores how faithfully key facts were preserved. It follows the GEval method: expanding your instructions into a chain-of-thought rubric, then grading on a 0.0–1.0 scale (derived from a raw 0–10 judgement) with detailed explanations.

Use it when you automatically summarise support tickets, research reports, or call transcripts and want to catch hallucinations before they reach end users.

Checking summary faithfulness

from opik.evaluation.metrics import SummarizationConsistencyJudge
metric = SummarizationConsistencyJudge(model="gpt-4o")
payload = """CONTEXT: Acme's Q2 revenue grew 12% thanks to the launch of Product Vega.
CONTEXT: Operating margin declined to 14% because of R&D hiring.
SUMMARY: Acme's revenue was flat but margins improved due to new hires.
"""
score = metric.score(output=payload)
print(score.value)   # 0.0–1.0 after normalisation
print(score.reason)

Inputs

Argument	Type	Required	Description
`output`	`str`	Yes	Payload combining the source material and the candidate summary. The judge only sees this string.

input and any other keyword arguments are accepted but ignored: they never reach the judge prompt, so the source document must be included in output.

Configuration

Parameter	Default	Notes
`model`	`gpt-5-nano`	Swap to a larger evaluator for longer or more technical content.
`temperature`	`0.0`	Keep low for deterministic scoring; raise slightly to sample different critiques.
`track`	`True`	Disable to skip sending traces to Opik.
`project_name`	`None`	Override when logging scores.

The evaluator emits an integer between 0 and 10 that Opik normalises to 0–1; the reason field captures the rubric notes explaining the judgement.