{"id":20750,"date":"2026-09-29T22:11:53","date_gmt":"2026-09-29T22:11:53","guid":{"rendered":"https:\/\/www.comet.com\/site\/?p=20750"},"modified":"2026-09-29T22:11:53","modified_gmt":"2026-09-29T22:11:53","slug":"jev-model-vs-llm-as-a-judge","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/jev-model-vs-llm-as-a-judge\/","title":{"rendered":"Jev vs. LLM-as-a-Judge for AI Evals"},"content":{"rendered":"\n

A model that answers yes\/no questions instead of writing text came out 3.5\u00d7 cheaper and 3.8\u00d7 faster than our gpt-4o-mini judge on 1,000 real production turns, and agreed with it on 897 of them. Here’s the test that showed it, what the 103 disagreements have in common, and how to run it yourself the next time a cheaper model turns up.<\/em><\/p>\n\n\n\n

\"side-by-side<\/figure>\n\n\n\n

Every team running an LLM in production needs to grade far more traffic than anyone can read, so they set up an LLM-as-a-Judge<\/strong> evaluation (LLM-J from here on). Send each exchange to a frontier model and parse a verdict out of what comes back. Two problems follow. Frontier-model prices force most teams to sample, so the rare failures the judge exists to catch sit in the traffic nobody scored. And almost nobody checks whether the judge itself is right before trusting its dashboard.<\/p>\n\n\n\n

Jev<\/a>, from TypeSafe AI, is a new way to address the first problem. It answers typed questions about a piece of text instead of writing about it. Hand it a user message and a reply, ask whether the reply answered the question, and you get a probability back rather than a paragraph. Nothing to parse, no schema to enforce, no temperature or seed to pin down, and it bills only the tokens it reads, at $0.042 per million.<\/p>\n\n\n\n

Jev is new. Specialized models are not.<\/h2>\n\n\n\n

Jev is early, but the shape of it is not new. Embeddings went to small dedicated encoders years ago, reranking to cross-encoders, moderation to classifiers like Llama Guard. Each began as a general model doing the job well enough at a price that only made sense at low volume. Judging is next, and Jev is the first model of this kind Opik supports directly.<\/p>\n\n\n\n

Jev will not be the last. What carries over from one model to the next is the test: on your own traffic, does the new one do the job better than what you run today, cheaper, faster or more accurately? When it lands close but not quite, sharpen the question you ask it. When even that falls short, the labelled data you built for the test can be used to train your own.<\/p>\n\n\n\n

Using Jev with Opik for LLM Observability & Evals<\/h2>\n\n\n\n

Opik is Comet’s open-source platform for LLM observability<\/a> and evaluation. It records your application’s traces, holds the datasets you evaluate against, and runs judges over live traffic or a fixed set, keeping every result where you can compare it. That matters because swapping a judge is a measurement problem: it only counts if both judges saw the same traces, with cost and latency recorded beside the verdicts. Do it once in Opik and the setup is still there when the next model arrives: same project, same labelled set, one more rule.<\/p>\n\n\n\n

Here are three places Jev plugs into Opik to help improve your agent or LLM application:<\/strong><\/p>\n\n\n\n

1. Trace your LLM application and Jev calls<\/h3>\n\n\n\n

If Jev makes decisions inside your application rather than about it, for routing, triage or moderation, those calls belong in your traces. The typesafe-sdk<\/code> client is not OpenAI-compatible, so Opik ships a native integration:<\/p>\n\n\n\n

from typesafe_sdk import Noul, TypeSafeClient\nfrom opik.integrations.typesafe import track_typesafe\n\nclient = track_typesafe(TypeSafeClient())\n\nresponse = client.system_one(\n    state={\"document\": ticket},\n    questions={\"is_urgent\": Noul(instructions=\"The message conveys urgency\")},\n)<\/code><\/pre>\n\n\n\n
Every system_one<\/code> call becomes an LLM span with state and questions as input, answers as output, plus token usage, provider, and cost alongside. Details in the integration docs<\/a>.<\/p>\n\n\n\n
\"trace<\/figure>\n\n\n\n
2. Score production traffic with Jev<\/h3>\n\n\n\n
The more common case is Jev judging your app rather than running inside it, and that needs no code. Jev is selectable in Opik as the model behind an LLM-as-a-Judge online evaluation rule, under the OpenRouter provider, using your workspace’s existing key. Pick typesafe\/jev-1.13<\/code>; the evaluation span records the served version, in our case typesafe\/jev-1.13-20260917<\/code>.<\/p>\n\n\n\n
Author the rule the way Jev reads it. The prompt is only the data; each score’s description is the question.<\/strong> The prompt holds nothing but INPUT: {{input}}<\/code> and OUTPUT: {{output}}<\/code>, and a Boolean score named answered<\/code> carries “The reply addresses the user’s question” as its description. Put the question in the prompt instead and Jev reads it as part of the text it is judging.<\/p>\n\n\n\n
The form narrows once you pick a Jev model: Boolean scores only, one text-only user message, no {{trace}}<\/code>, no model settings. A probability of 0.5 or more stores a 1<\/code>, and the probability lands in the reason, so you can re-cut your threshold later on data you have already paid for. The judge call is itself traced with billed cost and latency, which is where every number in this post comes from. Full list in the online evaluation docs<\/a>.<\/p>\n\n\n\n
\n
\n
\"UI<\/figure>\n<\/div>\n\n\n\n
\n
\"UI<\/figure>\n<\/div>\n<\/div>\n\n\n\n
3. Use Opik to optimize the question Jev is asked<\/h3>\n\n\n\n
Jev has no temperature and no few-shot examples, so the wording of the question is the entire configuration surface. You can use Opik\u2019s Agent Optimizer to automatically iterate on prompts and surface top candidates for your desired outcome, with six built-in optimization algorithms to choose from. Only the judge’s question is tuned here, and the metric stays fixed: did the verdict match the reference label. What improves is the instrument, not the thing it measures.<\/p>\n\n\n\n
Point the Opik Optimizer<\/a> at a labelled dataset, with the candidate question as the prompt and agreement-with-label as the metric. The user<\/code> template is required: without it the optimizer generates no candidates yet still reports a green summary, so read the trial count.<\/p>\n\n\n\n
optimizer = MetaPromptOptimizer(model=\"openai\/gpt-4o-mini\")\n\nresult = optimizer.optimize_prompt(\n    prompt=ChatPrompt(\n        system=\"The answer addresses the question that was asked\",\n        user=\"INPUT:\\n{question}\\n\\nOUTPUT:\\n{answer}\",\n    ),\n    dataset=labelled_dataset,\n    metric=agrees_with_label,\n    agent=JevAgent(),\n)<\/code><\/pre>\n\n\n\n
The optimizer’s own model only writes candidate questions; Jev does all the judging.<\/p>\n\n\n\n
What We Learned Testing Jev on Ollie<\/h2>\n\n\n\n
Ollie is an AI assistant inside Opik that reads and analyzes traces, diagnoses issues, and suggests fixes, helping users debug the agents they are building. Internally, our team uses Opik in turn to monitor and continually improve Ollie\u2019s performance for our end users. Ollie already had an LLM-J rule on openai\/gpt-4o-mini<\/code> scoring whether a reply answered the user. We sampled 1,000 real turns from 578 threads in its production monitoring project and re-logged them into a dedicated project so both judges scored the same turns without touching live traces. Both ran as online rules at 100% sampling with one Boolean score and the same question: “The answer addresses the question that was asked.”<\/p>\n\n\n\n
<\/th>gpt-4o-mini<\/code><\/th>Jev<\/th><\/tr><\/thead>
Agreement with the other judge<\/td>89.7%<\/td>89.7%<\/td><\/tr>
Cost per 1,000 traces<\/td>$0.0993<\/td>$0.0285<\/strong><\/td><\/tr>
p50 latency<\/td>958 ms<\/td>253 ms<\/strong><\/td><\/tr>
p90 latency<\/td>1,202 ms<\/td>352 ms<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Jev held up for this narrow judging job.<\/strong> 897 of 1,000 turns agree, and Jev is 3.5\u00d7 cheaper and 3.8\u00d7 faster at the median despite sending 1.5\u00d7 more input tokens, because the Decisions API encodes the question structure alongside the text. At $0.042 per million, token count no longer decides your bill, and judging 100% of traffic becomes the default rather than something you argue for.<\/p>\n\n\n\n
The disagreements were predictable.<\/strong> The 103 turns the judges split on sit where Jev itself was unsure: 64.1% of them fall between 0.3 and 0.7 on Jev’s probability, against 16.7% of the agreements. Sort every turn by Jev’s confidence and the pattern is a clean U \u2014 below 0.2 the two judges agree 100% of the time (n=118), above 0.8 they agree 95.6% (n=568), and in the 0.4\u20130.6 band agreement falls to 63.5% (n=115).<\/p>\n\n\n\n
The probability is worth more than the verdict.<\/strong> Jev’s answers were confident but not binary: 568 of 1,000 above 0.8, 115 still in the 0.4\u20130.6 band, spread across 97 distinct values. That middle band is a review queue you can route to a human instead of sampling blindly. A judge that returns only true or false throws it away and leaves you no way to tell a confident call from a coin flip. Use the probability to route \u2014 but not to trust, because none of this says which judge was right.<\/p>\n\n\n\n
We did not establish which judge was right.<\/strong> That needs labelled data, and labelling forced a question we have not fully answered and no model could settle for us: where does Ollie’s scope end? The same gap had already broken the question both judges were given, which scores a polite refusal as a failure. Before you can evaluate a judge, somebody has to write down the policy it is judging against.<\/p>\n\n\n\n
Use Jev where a probability is the whole product.<\/strong> A classifier returns a number and nothing else: no written critique, no tool calls, no non-Boolean scores, no thread-level judgments. So the test against your own judge is not “is this judgment hard?” but “does anything downstream consume more than a number?” If a human reads the reason, keep the generative judge. If the score feeds a dashboard, a filter or a route, the number is the whole product.<\/p>\n\n\n\n
Summary<\/h2>\n\n\n\n
The takeaway here is not “adopt Jev” but to establish a repeatable way to find out whether a new model does your job better on your own traffic<\/strong>. A benchmark cannot tell you; a vendor cannot tell you; your traces can.<\/p>\n\n\n\n