{"id":20965,"date":"2026-11-19T02:06:38","date_gmt":"2026-11-19T02:06:38","guid":{"rendered":"https:\/\/www.comet.com\/site\/?p=20965"},"modified":"2026-11-19T02:06:39","modified_gmt":"2026-11-19T02:06:39","slug":"finetune-judge-vs-jev","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/","title":{"rendered":"Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn&#8217;t)"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><strong><em>TLDR:<\/em><\/strong> <em>We took 1,000 real conversations with Ollie, the AI assistant inside <a href=\"https:\/\/www.comet.com\/site\/products\/opik\/\">Opik<\/a>, labelled each one with three yes\/no questions, and used 600 of them to fine-tune a 1.7B-parameter open model into a judge. On a held-out set of 300 turns, that judge beat both Jev and a gpt-4o-mini LLM-as-a-Judge at the questions it was trained on: whether Ollie declined a request (0.93 accuracy, 0.98 AUC) and whether the request was in scope. On the one question that depends on knowing the right answer, whether Ollie&#8217;s reply was factually sound, Jev stayed ahead. The takeaway is a rule of thumb: train your own judge for decisions about the shape of an interaction; keep an off-the-shelf judge for decisions that need knowledge.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Every Ollie reply is itself a trace in Opik, which makes Ollie a convenient test subject. We already had two online evaluation rules scoring every reply, and all the data needed to compare them. This post is about what happened when we added a third rule to label that data, then trained our own judge on the labels.<\/p>\n\n\n\n<h2 id=\"h-off-the-shelf-judge-or-your-own-decision-model\" class=\"wp-block-heading\">Off-the-shelf judge, or your own decision model?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Our <a href=\"https:\/\/www.comet.com\/site\/blog\/jev-model-vs-llm-as-a-judge\/\">previous post<\/a> ran two off-the-shelf judges over 1,000 real Ollie turns, each asked whether the reply addressed the user&#8217;s question: Jev, a small model that answers typed yes\/no questions with a probability, and gpt-4o-mini as an <a href=\"https:\/\/www.comet.com\/site\/blog\/llm-as-a-judge\/\">LLM-as-a-Judge<\/a>. They agreed 89.7% of the time, and Jev was 3.5\u00d7 cheaper. What we could not say was which one was right, because we had no labels.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Labels change the question. Once you have a few hundred labelled decisions from your own traffic, an off-the-shelf judge is no longer the only option: you can train a small decision model on those labels instead. This post does that and asks whether it was worth it. We build the judge step by step, with every run tracked in Comet Experiment Management, then score it against Jev and gpt-4o-mini on turns none of them had seen.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For what a decision model is, how one is built on a transformer backbone and how to test whether its probabilities can be trusted, see Abby Morgan&#8217;s <a href=\"https:\/\/www.comet.com\/site\/blog\/train-a-decision-model\/\">Train your own decision model<\/a>. This post covers one applied case: a judge for a single product, trained on that product&#8217;s traffic and compared with the judges we were already running.<\/p>\n\n\n\n<h2 id=\"h-how-we-went-about-it\" class=\"wp-block-heading\">How we went about it<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Data.<\/strong> The same 1,000 Ollie turns from the previous post. Each turn is one user message and one Ollie reply, with both existing judges&#8217; scores already attached.<\/li>\n\n\n\n<li><strong>Reference labels.<\/strong> Three yes\/no questions per turn, answered by a strong model (gpt-5.5) through an Opik online evaluation rule: is the request on topic for Ollie, did Ollie decline, and (treated as an attempt to answer) is the reply sound. Across the 1,000 turns, 68% were on topic, 29% were declines, and 67% were judged sound.<\/li>\n\n\n\n<li><strong>Split.<\/strong> 600 turns for training, 100 for validation, 300 for the final test, stratified on the soundness label with a fixed seed. The test split was used once.<\/li>\n\n\n\n<li><strong>Judges compared.<\/strong> Jev 1.13 as shipped, with no task-specific training, asked &#8220;does the reply address the question the user asked?&#8221;. gpt-4o-mini as an LLM-as-a-Judge, asked the same question. And our model, which is a separate decision model and not Jev: Qwen3-1.7B with a LoRA adapter trained on the 600 turns.<\/li>\n\n\n\n<li><strong>Scoring.<\/strong> Each judge&#8217;s yes\/no threshold was chosen on the validation set, so no judge got a lucky cut-off. We report accuracy, agreement with the labels beyond chance (Cohen&#8217;s \u03ba), and AUC for judges that return a probability.<\/li>\n\n\n\n<li><strong>Tracking.<\/strong> Every training run is a Comet Experiment Management experiment, and the training data is a versioned Comet Artifact.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Held-out test set (300 Ollie turns)<\/th><th>Our fine-tuned 1.7B judge<\/th><th>Jev 1.13<\/th><th>LLM-J (gpt-4o-mini)<\/th><\/tr><\/thead><tbody><tr><td>Did Ollie decline? Accuracy<\/td><td><strong>0.933<\/strong><\/td><td>0.857<\/td><td>0.883<\/td><\/tr><tr><td>Did Ollie decline? Agreement with labels (Cohen&#8217;s \u03ba)<\/td><td><strong>0.842<\/strong><\/td><td>0.651<\/td><td>0.720<\/td><\/tr><tr><td>Did Ollie decline? AUC<\/td><td><strong>0.982<\/strong><\/td><td>0.914<\/td><td>n\/a (yes\/no only)<\/td><\/tr><tr><td>Was an attempted answer sound? AUC<\/td><td>0.681<\/td><td><strong>0.724<\/strong><\/td><td>n\/a (yes\/no only)<\/td><\/tr><tr><td>Training needed<\/td><td>600 labelled turns, about 37 minutes on a laptop<\/td><td>None<\/td><td>None<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 id=\"h-building-the-judge-step-by-step\" class=\"wp-block-heading\">Building the judge, step by step<\/h2>\n\n\n\n<h3 id=\"h-1-label-production-decisions-with-an-opik-online-evaluation-rule\" class=\"wp-block-heading\">1. Label production decisions with an Opik online evaluation rule<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">An online evaluation rule in Opik is a scorer attached to a project. Every new trace that lands in the project is passed to a model together with a question, and the model&#8217;s answer is stored on the trace as a feedback score. The Ollie project from the previous post already had two such rules: one running Jev and one running gpt-4o-mini, both asking &#8220;does the assistant&#8217;s reply address the question the user asked?&#8221;.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To get reference labels, we added a third rule. It sends each turn to gpt-5.5 with three strictly worded questions and stores the three answers as feedback scores on the same trace:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>on_topic<\/code>: is the user&#8217;s request within Ollie&#8217;s scope (Opik, Comet, LLM observability, evaluation, tracing, or the product itself)?<\/li>\n\n\n\n<li><code>declined<\/code>: did Ollie decline or redirect instead of attempting a substantive answer?<\/li>\n\n\n\n<li><code>answer_sound<\/code>: treated as an attempt to answer, is the reply responsive and factually sound for what was asked? A decline counts as not sound.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">These three questions are also the scope policy the previous post said was missing. Its single question, whether the reply addressed the user, scored a polite refusal as a failure. Three labels separate a request Ollie should refuse, a refusal and a wrong answer.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once the rule had run over the 1,000 turns, every turn carried five scores: Jev&#8217;s, gpt-4o-mini&#8217;s, and the three reference labels. We exported them with the Opik SDK, split them 600 \/ 100 \/ 300 as described above, and did not look at the test split again until the final table.<\/p>\n\n\n\n<h3 id=\"h-2-start-from-a-decision-model-recipe\" class=\"wp-block-heading\">2. Start from a decision-model recipe<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We started from Nish Tahir&#8217;s recipe for <a href=\"https:\/\/nishtahir.com\/build-your-own-decision-model\/\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">emulating a decision model with an ordinary LLM<\/a>. Take a small open LLM (we used Qwen3-1.7B), show it a multiple-choice question, and read the probability of each option letter from a single forward pass. Nothing is generated, so there is no output to parse.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We framed each judge question as Yes\/No and asked it twice, once with Yes as option A and once with Yes as option B, then averaged the two probabilities. Language models prefer whichever letter comes first, and they lean toward agreeing in general, so this reduces the position bias rather than removing it. Averaging log-odds instead of probabilities would be a small further improvement that we have not made yet.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.comet.com\/site\/blog\/train-a-decision-model\/\">Our post on training your own decision model<\/a> measures the same bias on its own models: with the options listed in reverse, an untrained model changed its top answer on 30% of questions, and trained models still did on 5 to 9%.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def p_yes(model, tokenizer, turn, question):\n    probs = &#91;]\n    for options in &#91;(\"Yes\", \"No\"), (\"No\", \"Yes\")]:\n        prompt = format_prompt(turn, question, options)   # INPUT \/ OUTPUT \/ statement \/ A. B. \/ Answer:\n        logits = model(**tokenizer(prompt, return_tensors=\"pt\")).logits&#91;0, -1]\n        p = torch.softmax(logits&#91;&#91;A_ID, B_ID]], dim=-1)   # constrained to the two option letters\n        probs.append(p&#91;options.index(\"Yes\")])\n    return sum(probs) \/ 2   # reduces, does not remove, position bias<\/code><\/pre>\n\n\n\n<h3 id=\"h-3-track-every-run-in-comet-experiment-management\" class=\"wp-block-heading\">3. Track every run in Comet Experiment Management<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Every training run is a Comet experiment: hyperparameters, a live loss curve, test metrics, the code that produced it and the resulting LoRA adapter, all in one place. That made it easy to compare a dead end against the run that worked. There was a dead end, and it is covered below.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"450\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-experiments.png\" alt=\"Two fine-tuning runs listed in a Comet project: the Ollie judge and an earlier CommonsenseQA run, with batch size, LoRA settings and step counts\" class=\"wp-image-20959\" srcset=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-experiments.png 1600w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-experiments-300x84.png 300w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-experiments-1024x288.png 1024w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-experiments-768x216.png 768w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-experiments-1536x432.png 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">Both fine-tuning runs in the Comet decision model project: the Ollie judge and the earlier CommonsenseQA run, with their batch sizes, LoRA settings and step counts side by side.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"1000\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-hyperparameters.png\" alt=\"Hyperparameters tab of the judge run in Comet, including batch size, LoRA rank and a labels_from field naming the gpt-5.5 online evaluation rule\" class=\"wp-image-20960\" srcset=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-hyperparameters.png 1600w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-hyperparameters-300x188.png 300w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-hyperparameters-1024x640.png 1024w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-hyperparameters-768x480.png 768w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-hyperparameters-1536x960.png 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">Hyperparameters logged on the judge run, including where the data and labels came from, so the run explains itself months later.<\/figcaption><\/figure>\n\n\n\n<h3 id=\"h-4-version-the-training-data-as-an-artifact\" class=\"wp-block-heading\">4. Version the training data as an artifact<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The model is only as reproducible as its data. We logged the train, validation and test splits, plus the 3,600 exact prompts the model was trained on, as a versioned Comet Artifact linked to the experiment that consumed it. If the labels change or we add more turns, that becomes version 1.1.0, and every experiment records which version it saw.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"590\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-artifact-training-data.png\" alt=\"The ollie-judge-data artifact at version 1.0.0 in Comet, holding train, validation and test splits and the formatted training examples\" class=\"wp-image-20961\" srcset=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-artifact-training-data.png 1600w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-artifact-training-data-300x111.png 300w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-artifact-training-data-1024x378.png 1024w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-artifact-training-data-768x283.png 768w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-artifact-training-data-1536x566.png 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">The ollie-judge-data artifact, version 1.0.0: the three splits and the formatted training examples, linked to the source experiment.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">These are real user conversations, so the artifact and project are private. Keeping production-derived data inside the same governed workspace as the experiments, rather than in a bucket somewhere, is part of the point.<\/p>\n\n\n\n<h3 id=\"h-5-fine-tune\" class=\"wp-block-heading\">5. Fine-tune<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We trained a LoRA adapter (6.4 million trainable parameters, 0.37% of the model) on the three label questions, in both option orders, for one epoch: 450 optimizer steps. The whole run, training and evaluation included, took about 37 minutes on a MacBook. The loss falls sharply in the first few dozen steps and then flattens; the model learns the shape of these decisions quickly.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"720\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-train-loss.png\" alt=\"Smoothed training loss for the judge run in Comet, falling steeply in the first 25 steps and staying flat through step 450\" class=\"wp-image-20962\" srcset=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-train-loss.png 1600w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-train-loss-300x135.png 300w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-train-loss-1024x461.png 1024w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-train-loss-768x346.png 768w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-train-loss-1536x691.png 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">Training loss for the judge run in Comet, smoothed. Most of the learning happens in the first ~25 steps.<\/figcaption><\/figure>\n\n\n\n<h3 id=\"h-6-compare-against-the-judges-you-already-have\" class=\"wp-block-heading\">6. Compare against the judges you already have<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We scored the 300 held-out turns with our model and pulled Jev&#8217;s and gpt-4o-mini&#8217;s existing scores for the same turns. Each judge&#8217;s decision threshold was chosen on the validation set, not the test set, so no judge got an unfair cut-off. All of the comparison metrics were logged back to the same Comet experiment.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1600\" height=\"1000\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-declined-metrics.png\" alt=\"Metrics tab in Comet with test accuracy, AUC and kappa on the declined label for the fine-tuned judge, Jev 1.13 and gpt-4o-mini\" class=\"wp-image-20963\" srcset=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-declined-metrics.png 1600w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-declined-metrics-300x188.png 300w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-declined-metrics-1024x640.png 1024w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-declined-metrics-768x480.png 768w, https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-judge-declined-metrics-1536x960.png 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><figcaption class=\"wp-element-caption\">Held-out results for &#8220;did Ollie decline?&#8221; logged in Comet: our judge (dm-judge) against Jev 1.13 and gpt-4o-mini. LLM-J has no AUC because it only returns yes\/no.<\/figcaption><\/figure>\n\n\n\n<h2 id=\"h-what-we-learned\" class=\"wp-block-heading\">What we learned<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>An off-the-shelf small model is not a judge.<\/strong> Before any fine-tuning, Qwen3-1.7B said &#8220;yes, the reply addressed the question&#8221; to 97% of turns. Its accuracy looked respectable only because most turns are in fact answered; its agreement with the labels beyond chance was 0.05. A decision model gives you a clean probability. It does not give you judgment.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Fine-tuning on the wrong data doesn&#8217;t help.<\/strong> Our first attempt fine-tuned the model on CommonsenseQA, a general reasoning benchmark. It worked on its own terms, taking accuracy on that benchmark from 0.597 to 0.784, and it did nothing for judging. On Ollie turns it said &#8220;yes&#8221; to 100% of them, and on the on-topic question its accuracy fell from 0.77 to 0.33. Getting better at a general task is not the same as getting better at yours. Because both runs sat side by side in the same Comet project, that was a two-minute diagnosis instead of a mystery.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>On its own decisions, the trained model wins clearly.<\/strong> Trained on 600 of our labelled turns, the same 1.7B model became the best judge we tested for the questions it was trained on:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Held-out test set (n=300)<\/th><th>Before fine-tuning (AUC)<\/th><th>After fine-tuning (AUC)<\/th><th>After fine-tuning (accuracy \/ \u03ba)<\/th><\/tr><\/thead><tbody><tr><td>Declined<\/td><td>0.638<\/td><td><strong>0.982<\/strong><\/td><td>0.933 \/ 0.842<\/td><\/tr><tr><td>On topic<\/td><td>0.772<\/td><td><strong>0.927<\/strong><\/td><td>0.893 \/ 0.740<\/td><\/tr><tr><td>Answer sound<\/td><td>0.678<\/td><td>0.771<\/td><td>0.703 \/ 0.337<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Even when we asked our model the exact question Jev and gpt-4o-mini were asked, &#8220;does the reply address the question the user asked?&#8221;, it separated declines from attempts better than either: accuracy 0.913 against 0.857 and 0.883, \u03ba 0.790 against 0.651 and 0.720. These decisions are pattern recognition over one product&#8217;s conversations, and that is exactly what a small model learns well from a few hundred examples.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>The trained model does not win everywhere, and where it loses tells you why.<\/strong> On answer soundness the headline numbers look close: our model scores 0.703 accuracy against Jev&#8217;s 0.683. But most of that comes from spotting declines, which the label also counts as &#8220;not sound.&#8221; Restrict the test to the 213 turns where Ollie actually tried to answer, where the label really means <em>was this answer correct?<\/em>, and Jev pulls ahead: AUC 0.724 to our 0.681. Of our model&#8217;s 43 wrong &#8220;sound&#8221; calls, 35 were on-topic requests whose answers the reference judge marked as unsound.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That&#8217;s not a training problem; it&#8217;s a knowledge problem. Deciding whether an answer about Opik&#8217;s SDK is <em>correct<\/em> requires knowing Opik&#8217;s SDK. Six hundred examples of what good answers look like don&#8217;t teach a 1.7B model the facts behind them. A larger general model brings that knowledge for free.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What the off-the-shelf judges still give you.<\/strong> Neither Jev nor gpt-4o-mini needed a single label or a minute of training, and either can be pointed at a new question this afternoon. gpt-4o-mini also writes down its reasoning, which matters when a human reads the verdict. Our trained judge knows three questions and nothing else. Asking it a fourth means labelling data and training again.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Caveats worth stating.<\/strong> Our labels came from a stronger model, not from humans, so what we measured is how closely each judge reproduces gpt-5.5. Our model was trained on that labeller&#8217;s output, while Jev and gpt-4o-mini were not. That is the realistic setup for a team distilling an expensive judge into a cheap one, but it isn&#8217;t a measure of absolute correctness. This is one training run with a 300-turn test set, so differences of \u00b10.03 in accuracy are within noise. And the fine-tuned model&#8217;s raw probabilities are poorly calibrated (its best thresholds sat near 0.03 for two of the labels), so the ranking is trustworthy, but the raw number shouldn&#8217;t be read as a confidence until it&#8217;s recalibrated on validation data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The calibration caveat has a known cause. We trained on one hard yes\/no label per question, and <a href=\"https:\/\/www.comet.com\/site\/blog\/train-a-decision-model\/\">Abby&#8217;s experiments<\/a> found that this style of training leaves a model overconfident (ECE 0.12 on that post&#8217;s test set), while training on a distribution of labels brought it to 0.02.<\/p>\n\n\n\n<h2 id=\"h-when-to-train-your-own-judge\" class=\"wp-block-heading\">When to train your own judge<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Train your own judge when\u2026<\/th><th>Use an off-the-shelf judge when\u2026<\/th><\/tr><\/thead><tbody><tr><td>The question is about the shape of the interaction: declines, scope, format, tone, policy<\/td><td>The question needs world or domain knowledge: is this answer correct?<\/td><\/tr><tr><td>You&#8217;ll ask the same few questions on a lot of traffic<\/td><td>Your questions change often, or you&#8217;re still discovering them<\/td><\/tr><tr><td>You have a few hundred trustworthy labels, or a stronger model you can distil from<\/td><td>You have no labels and no budget to make them<\/td><\/tr><tr><td>A score is the whole output<\/td><td>A human needs to read why<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 id=\"h-summary\" class=\"wp-block-heading\">Summary<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Label your own decisions first.<\/strong> Without labels, you can compare judges but not grade them. An Opik online evaluation rule with a strong model is a quick way to start.<\/li>\n\n\n\n<li><strong>Train on your task, not a proxy.<\/strong> General reasoning benchmarks made our model better at benchmarks and no better at judging. A few hundred examples of the real decision did more.<\/li>\n\n\n\n<li><strong>Expect a trained judge to win on pattern decisions and lose on knowledge decisions.<\/strong> Declines and scope went to the 1.7B model; answer correctness went to Jev.<\/li>\n\n\n\n<li><strong>Keep the work reproducible.<\/strong> Version the data as a Comet Artifact, log every run as an experiment, and keep the dead ends. They are how you know the winner is not a fluke.<\/li>\n<\/ul>\n\n\n\n<h2 id=\"h-get-started\" class=\"wp-block-heading\">Get started<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.comet.com\/site\/products\/ml-experiment-tracking\/\">Comet Experiment Management<\/a>: track fine-tuning runs, version training data as artifacts, and compare runs side by side.<\/li>\n\n\n\n<li><a href=\"https:\/\/www.comet.com\/docs\/opik\/production\/online-evaluation\/rules\">Opik online evaluation<\/a>: score production traces with Jev, an LLM-as-a-Judge, or a labelling model.<\/li>\n\n\n\n<li><a href=\"https:\/\/www.comet.com\/site\/blog\/jev-model-vs-llm-as-a-judge\/\">Jev vs. LLM-as-a-Judge<\/a>: the experiment this one continues.<\/li>\n\n\n\n<li><a href=\"https:\/\/www.comet.com\/site\/blog\/train-a-decision-model\/\">Train your own decision model<\/a>: how decision models work, with a full build, calibration tests and public code.<\/li>\n\n\n\n<li><a href=\"https:\/\/nishtahir.com\/build-your-own-decision-model\/\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Nish Tahir&#8217;s constrained decoding guide:<\/a> the constrained-decoding recipe this post builds on.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>We labelled 1,000 production turns, fine-tuned a 1.7B open model on 600 of them, and tested it against Jev and a gpt-4o-mini LLM-as-a-Judge. It won on declines and scope, not on answer correctness.<\/p>\n","protected":false},"author":132,"featured_media":20964,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[65,7],"tags":[],"coauthors":[374],"class_list":["post-20965","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-llmops","category-tutorials"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn&#039;t) - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn&#039;t)\" \/>\n<meta property=\"og:description\" content=\"We labelled 1,000 production turns, fine-tuned a 1.7B open model on 600 of them, and tested it against Jev and a gpt-4o-mini LLM-as-a-Judge. It won on declines and scope, not on answer correctness.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-19T02:06:38+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-11-19T02:06:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-llm-judge-featured.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"720\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Aswin Thiyagarajan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Aswin Thiyagarajan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"13 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn't) - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/","og_locale":"en_US","og_type":"article","og_title":"Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn't)","og_description":"We labelled 1,000 production turns, fine-tuned a 1.7B open model on 600 of them, and tested it against Jev and a gpt-4o-mini LLM-as-a-Judge. It won on declines and scope, not on answer correctness.","og_url":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2026-11-19T02:06:38+00:00","article_modified_time":"2026-11-19T02:06:39+00:00","og_image":[{"width":1280,"height":720,"url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-llm-judge-featured.jpg","type":"image\/jpeg"}],"author":"Aswin Thiyagarajan","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Aswin Thiyagarajan","Est. reading time":"13 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/"},"author":{"name":"Mike Ranellone","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/b0df8d0db9a521af425e33f561b39c6a"},"headline":"Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn&#8217;t)","datePublished":"2026-11-19T02:06:38+00:00","dateModified":"2026-11-19T02:06:39+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/"},"wordCount":2456,"commentCount":0,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-llm-judge-featured.jpg","articleSection":["LLMOps","Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/","url":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/","name":"Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn't) - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-llm-judge-featured.jpg","datePublished":"2026-11-19T02:06:38+00:00","dateModified":"2026-11-19T02:06:39+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-llm-judge-featured.jpg","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-llm-judge-featured.jpg","width":1280,"height":720,"caption":"Build Your Own LLM Judge: a fine-tuned 1.7B model scores 0.933 accuracy on declines against 0.883 for gpt-4o-mini and 0.857 for Jev, and 0.681 AUC on answer soundness against 0.724 for Jev"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/finetune-judge-vs-jev\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn&#8217;t)"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/b0df8d0db9a521af425e33f561b39c6a","name":"Mike Ranellone","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/47e0209bd037ec57787bae2b580d796f","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/06\/cropped-mike-ranellone-96x96.jpg","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/06\/cropped-mike-ranellone-96x96.jpg","caption":"Mike Ranellone"},"sameAs":["https:\/\/www.comet.com\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/mikercomet-com\/"}]}},"jetpack_featured_media_url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/build-your-own-llm-judge-featured.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/20965","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/132"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=20965"}],"version-history":[{"count":3,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/20965\/revisions"}],"predecessor-version":[{"id":20969,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/20965\/revisions\/20969"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/20964"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=20965"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=20965"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=20965"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=20965"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}