Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn’t)

TLDR: We took 1,000 real conversations with Ollie, the AI assistant inside Opik, labelled each one with three yes/no questions, and used 600 of them to fine-tune a 1.7B-parameter open model into a judge. On a held-out set of 300 turns, that judge beat both Jev and a gpt-4o-mini LLM-as-a-Judge at the questions it was trained on: whether Ollie declined a request (0.93 accuracy, 0.98 AUC) and whether the request was in scope. On the one question that depends on knowing the right answer, whether Ollie’s reply was factually sound, Jev stayed ahead. The takeaway is a rule of thumb: train your own judge for decisions about the shape of an interaction; keep an off-the-shelf judge for decisions that need knowledge.

Every Ollie reply is itself a trace in Opik, which makes Ollie a convenient test subject. We already had two online evaluation rules scoring every reply, and all the data needed to compare them. This post is about what happened when we added a third rule to label that data, then trained our own judge on the labels.

Off-the-shelf judge, or your own decision model?

Our previous post ran two off-the-shelf judges over 1,000 real Ollie turns, each asked whether the reply addressed the user’s question: Jev, a small model that answers typed yes/no questions with a probability, and gpt-4o-mini as an LLM-as-a-Judge. They agreed 89.7% of the time, and Jev was 3.5× cheaper. What we could not say was which one was right, because we had no labels.

Labels change the question. Once you have a few hundred labelled decisions from your own traffic, an off-the-shelf judge is no longer the only option: you can train a small decision model on those labels instead. This post does that and asks whether it was worth it. We build the judge step by step, with every run tracked in Comet Experiment Management, then score it against Jev and gpt-4o-mini on turns none of them had seen.

For what a decision model is, how one is built on a transformer backbone and how to test whether its probabilities can be trusted, see Abby Morgan’s Train your own decision model. This post covers one applied case: a judge for a single product, trained on that product’s traffic and compared with the judges we were already running.

How we went about it

  • Data. The same 1,000 Ollie turns from the previous post. Each turn is one user message and one Ollie reply, with both existing judges’ scores already attached.
  • Reference labels. Three yes/no questions per turn, answered by a strong model (gpt-5.5) through an Opik online evaluation rule: is the request on topic for Ollie, did Ollie decline, and (treated as an attempt to answer) is the reply sound. Across the 1,000 turns, 68% were on topic, 29% were declines, and 67% were judged sound.
  • Split. 600 turns for training, 100 for validation, 300 for the final test, stratified on the soundness label with a fixed seed. The test split was used once.
  • Judges compared. Jev 1.13 as shipped, with no task-specific training, asked “does the reply address the question the user asked?”. gpt-4o-mini as an LLM-as-a-Judge, asked the same question. And our model, which is a separate decision model and not Jev: Qwen3-1.7B with a LoRA adapter trained on the 600 turns.
  • Scoring. Each judge’s yes/no threshold was chosen on the validation set, so no judge got a lucky cut-off. We report accuracy, agreement with the labels beyond chance (Cohen’s κ), and AUC for judges that return a probability.
  • Tracking. Every training run is a Comet Experiment Management experiment, and the training data is a versioned Comet Artifact.
Held-out test set (300 Ollie turns)Our fine-tuned 1.7B judgeJev 1.13LLM-J (gpt-4o-mini)
Did Ollie decline? Accuracy0.9330.8570.883
Did Ollie decline? Agreement with labels (Cohen’s κ)0.8420.6510.720
Did Ollie decline? AUC0.9820.914n/a (yes/no only)
Was an attempted answer sound? AUC0.6810.724n/a (yes/no only)
Training needed600 labelled turns, about 37 minutes on a laptopNoneNone

Building the judge, step by step

1. Label production decisions with an Opik online evaluation rule

An online evaluation rule in Opik is a scorer attached to a project. Every new trace that lands in the project is passed to a model together with a question, and the model’s answer is stored on the trace as a feedback score. The Ollie project from the previous post already had two such rules: one running Jev and one running gpt-4o-mini, both asking “does the assistant’s reply address the question the user asked?”.

To get reference labels, we added a third rule. It sends each turn to gpt-5.5 with three strictly worded questions and stores the three answers as feedback scores on the same trace:

  • on_topic: is the user’s request within Ollie’s scope (Opik, Comet, LLM observability, evaluation, tracing, or the product itself)?
  • declined: did Ollie decline or redirect instead of attempting a substantive answer?
  • answer_sound: treated as an attempt to answer, is the reply responsive and factually sound for what was asked? A decline counts as not sound.

These three questions are also the scope policy the previous post said was missing. Its single question, whether the reply addressed the user, scored a polite refusal as a failure. Three labels separate a request Ollie should refuse, a refusal and a wrong answer.

Once the rule had run over the 1,000 turns, every turn carried five scores: Jev’s, gpt-4o-mini’s, and the three reference labels. We exported them with the Opik SDK, split them 600 / 100 / 300 as described above, and did not look at the test split again until the final table.

2. Start from a decision-model recipe

We started from Nish Tahir’s recipe for emulating a decision model with an ordinary LLM. Take a small open LLM (we used Qwen3-1.7B), show it a multiple-choice question, and read the probability of each option letter from a single forward pass. Nothing is generated, so there is no output to parse.

We framed each judge question as Yes/No and asked it twice, once with Yes as option A and once with Yes as option B, then averaged the two probabilities. Language models prefer whichever letter comes first, and they lean toward agreeing in general, so this reduces the position bias rather than removing it. Averaging log-odds instead of probabilities would be a small further improvement that we have not made yet.

Our post on training your own decision model measures the same bias on its own models: with the options listed in reverse, an untrained model changed its top answer on 30% of questions, and trained models still did on 5 to 9%.

def p_yes(model, tokenizer, turn, question):
    probs = []
    for options in [("Yes", "No"), ("No", "Yes")]:
        prompt = format_prompt(turn, question, options)   # INPUT / OUTPUT / statement / A. B. / Answer:
        logits = model(**tokenizer(prompt, return_tensors="pt")).logits[0, -1]
        p = torch.softmax(logits[[A_ID, B_ID]], dim=-1)   # constrained to the two option letters
        probs.append(p[options.index("Yes")])
    return sum(probs) / 2   # reduces, does not remove, position bias

3. Track every run in Comet Experiment Management

Every training run is a Comet experiment: hyperparameters, a live loss curve, test metrics, the code that produced it and the resulting LoRA adapter, all in one place. That made it easy to compare a dead end against the run that worked. There was a dead end, and it is covered below.

Two fine-tuning runs listed in a Comet project: the Ollie judge and an earlier CommonsenseQA run, with batch size, LoRA settings and step counts
Both fine-tuning runs in the Comet decision model project: the Ollie judge and the earlier CommonsenseQA run, with their batch sizes, LoRA settings and step counts side by side.
Hyperparameters tab of the judge run in Comet, including batch size, LoRA rank and a labels_from field naming the gpt-5.5 online evaluation rule
Hyperparameters logged on the judge run, including where the data and labels came from, so the run explains itself months later.

4. Version the training data as an artifact

The model is only as reproducible as its data. We logged the train, validation and test splits, plus the 3,600 exact prompts the model was trained on, as a versioned Comet Artifact linked to the experiment that consumed it. If the labels change or we add more turns, that becomes version 1.1.0, and every experiment records which version it saw.

The ollie-judge-data artifact at version 1.0.0 in Comet, holding train, validation and test splits and the formatted training examples
The ollie-judge-data artifact, version 1.0.0: the three splits and the formatted training examples, linked to the source experiment.

These are real user conversations, so the artifact and project are private. Keeping production-derived data inside the same governed workspace as the experiments, rather than in a bucket somewhere, is part of the point.

5. Fine-tune

We trained a LoRA adapter (6.4 million trainable parameters, 0.37% of the model) on the three label questions, in both option orders, for one epoch: 450 optimizer steps. The whole run, training and evaluation included, took about 37 minutes on a MacBook. The loss falls sharply in the first few dozen steps and then flattens; the model learns the shape of these decisions quickly.

Smoothed training loss for the judge run in Comet, falling steeply in the first 25 steps and staying flat through step 450
Training loss for the judge run in Comet, smoothed. Most of the learning happens in the first ~25 steps.

6. Compare against the judges you already have

We scored the 300 held-out turns with our model and pulled Jev’s and gpt-4o-mini’s existing scores for the same turns. Each judge’s decision threshold was chosen on the validation set, not the test set, so no judge got an unfair cut-off. All of the comparison metrics were logged back to the same Comet experiment.

Metrics tab in Comet with test accuracy, AUC and kappa on the declined label for the fine-tuned judge, Jev 1.13 and gpt-4o-mini
Held-out results for “did Ollie decline?” logged in Comet: our judge (dm-judge) against Jev 1.13 and gpt-4o-mini. LLM-J has no AUC because it only returns yes/no.

What we learned

An off-the-shelf small model is not a judge. Before any fine-tuning, Qwen3-1.7B said “yes, the reply addressed the question” to 97% of turns. Its accuracy looked respectable only because most turns are in fact answered; its agreement with the labels beyond chance was 0.05. A decision model gives you a clean probability. It does not give you judgment.

Fine-tuning on the wrong data doesn’t help. Our first attempt fine-tuned the model on CommonsenseQA, a general reasoning benchmark. It worked on its own terms, taking accuracy on that benchmark from 0.597 to 0.784, and it did nothing for judging. On Ollie turns it said “yes” to 100% of them, and on the on-topic question its accuracy fell from 0.77 to 0.33. Getting better at a general task is not the same as getting better at yours. Because both runs sat side by side in the same Comet project, that was a two-minute diagnosis instead of a mystery.

On its own decisions, the trained model wins clearly. Trained on 600 of our labelled turns, the same 1.7B model became the best judge we tested for the questions it was trained on:

Held-out test set (n=300)Before fine-tuning (AUC)After fine-tuning (AUC)After fine-tuning (accuracy / κ)
Declined0.6380.9820.933 / 0.842
On topic0.7720.9270.893 / 0.740
Answer sound0.6780.7710.703 / 0.337

Even when we asked our model the exact question Jev and gpt-4o-mini were asked, “does the reply address the question the user asked?”, it separated declines from attempts better than either: accuracy 0.913 against 0.857 and 0.883, κ 0.790 against 0.651 and 0.720. These decisions are pattern recognition over one product’s conversations, and that is exactly what a small model learns well from a few hundred examples.

The trained model does not win everywhere, and where it loses tells you why. On answer soundness the headline numbers look close: our model scores 0.703 accuracy against Jev’s 0.683. But most of that comes from spotting declines, which the label also counts as “not sound.” Restrict the test to the 213 turns where Ollie actually tried to answer, where the label really means was this answer correct?, and Jev pulls ahead: AUC 0.724 to our 0.681. Of our model’s 43 wrong “sound” calls, 35 were on-topic requests whose answers the reference judge marked as unsound.

That’s not a training problem; it’s a knowledge problem. Deciding whether an answer about Opik’s SDK is correct requires knowing Opik’s SDK. Six hundred examples of what good answers look like don’t teach a 1.7B model the facts behind them. A larger general model brings that knowledge for free.

What the off-the-shelf judges still give you. Neither Jev nor gpt-4o-mini needed a single label or a minute of training, and either can be pointed at a new question this afternoon. gpt-4o-mini also writes down its reasoning, which matters when a human reads the verdict. Our trained judge knows three questions and nothing else. Asking it a fourth means labelling data and training again.

Caveats worth stating. Our labels came from a stronger model, not from humans, so what we measured is how closely each judge reproduces gpt-5.5. Our model was trained on that labeller’s output, while Jev and gpt-4o-mini were not. That is the realistic setup for a team distilling an expensive judge into a cheap one, but it isn’t a measure of absolute correctness. This is one training run with a 300-turn test set, so differences of ±0.03 in accuracy are within noise. And the fine-tuned model’s raw probabilities are poorly calibrated (its best thresholds sat near 0.03 for two of the labels), so the ranking is trustworthy, but the raw number shouldn’t be read as a confidence until it’s recalibrated on validation data.

The calibration caveat has a known cause. We trained on one hard yes/no label per question, and Abby’s experiments found that this style of training leaves a model overconfident (ECE 0.12 on that post’s test set), while training on a distribution of labels brought it to 0.02.

When to train your own judge

Train your own judge when…Use an off-the-shelf judge when…
The question is about the shape of the interaction: declines, scope, format, tone, policyThe question needs world or domain knowledge: is this answer correct?
You’ll ask the same few questions on a lot of trafficYour questions change often, or you’re still discovering them
You have a few hundred trustworthy labels, or a stronger model you can distil fromYou have no labels and no budget to make them
A score is the whole outputA human needs to read why

Summary

  • Label your own decisions first. Without labels, you can compare judges but not grade them. An Opik online evaluation rule with a strong model is a quick way to start.
  • Train on your task, not a proxy. General reasoning benchmarks made our model better at benchmarks and no better at judging. A few hundred examples of the real decision did more.
  • Expect a trained judge to win on pattern decisions and lose on knowledge decisions. Declines and scope went to the 1.7B model; answer correctness went to Jev.
  • Keep the work reproducible. Version the data as a Comet Artifact, log every run as an experiment, and keep the dead ends. They are how you know the winner is not a fluke.

Get started

Aswin Thiyagarajan Avatar

Aswin Thiyagarajan

Aswin Thiyagarajan leads product for Opik at Comet. He started as a software engineer, and over 16 years he’s built developer tools and technical products, from APIs and analytics platforms to the workflows developers rely on every day. He’s shipped AI products in e-commerce, energy, and healthcare, where he learned that every domain breaks in its own particular way. At Opik, he takes hard AI infrastructure problems and turns them into tools developers find simple and useful.