Agent Task Completion Judge

AgentTaskCompletionJudge reviews an agent run (often a natural-language summary of what happened) and decides whether the high-level objective was met. It is particularly helpful for multi-step agents where success cannot be inferred from the final response alone.

1
from opik.evaluation.metrics import AgentTaskCompletionJudge
2
3
metric = AgentTaskCompletionJudge()
4
5
payload = """TASK: Extract company name, address, and tax ID from the invoice.
6
OUTCOME: Agent retrieved company name and address but failed to extract the tax ID.
7
"""
8
9
score = metric.score(output=payload)
10
11
print(score.value)  # 0.0–1.0 after normalisation
12
print(score.reason)

Inputs

Configuration

The evaluator returns an integer between 0 and 10; Opik divides it by 10 so score.value falls in the 0.0–1.0 range, while score.reason summarises which sub-tasks were completed or missed.