TLDR: We took 1,000 real conversations with Ollie, the AI assistant inside Opik, labelled each one with three yes/no questions, and used 600 of them to fine-tune a 1.7B-parameter open model into a judge. On a held-out set of 300 turns, that judge beat both Jev and a gpt-4o-mini LLM-as-a-Judge at the questions it was trained on: whether Ollie declined a request (0.93 accuracy, 0.98 AUC) and whether the request was in scope. On the one question that depends on knowing the right answer, whether Ollie’s reply was factually sound, Jev stayed ahead. The takeaway is a rule of thumb: train your own judge for decisions about the shape of an interaction; keep an off-the-shelf judge for decisions that need knowledge.
Every Ollie reply is itself a trace in Opik, which makes Ollie a convenient test subject. We already had two online evaluation rules scoring every reply, and all the data needed to compare them. This post is about what happened when we added a third rule to label that data, then trained our own judge on the labels.
Off-the-shelf judge, or your own decision model?
Our previous post ran two off-the-shelf judges over 1,000 real Ollie turns, each asked whether the reply addressed the user’s question: Jev, a small model that answers typed yes/no questions with a probability, and gpt-4o-mini as an LLM-as-a-Judge. They agreed 89.7% of the time, and Jev was 3.5× cheaper. What we could not say was which one was right, because we had no labels.
Labels change the question. Once you have a few hundred labelled decisions from your own traffic, an off-the-shelf judge is no longer the only option: you can train a small decision model on those labels instead. This post does that and asks whether it was worth it. We build the judge step by step, with every run tracked in Comet Experiment Management, then score it against Jev and gpt-4o-mini on turns none of them had seen.
For what a decision model is, how one is built on a transformer backbone and how to test whether its probabilities can be trusted, see Abby Morgan’s Train your own decision model. This post covers one applied case: a judge for a single product, trained on that product’s traffic and compared with the judges we were already running.
How we went about it
- Data. The same 1,000 Ollie turns from the previous post. Each turn is one user message and one Ollie reply, with both existing judges’ scores already attached.
- Reference labels. Three yes/no questions per turn, answered by a strong model (gpt-5.5) through an Opik online evaluation rule: is the request on topic for Ollie, did Ollie decline, and (treated as an attempt to answer) is the reply sound. Across the 1,000 turns, 68% were on topic, 29% were declines, and 67% were judged sound.
- Split. 600 turns for training, 100 for validation, 300 for the final test, stratified on the soundness label with a fixed seed. The test split was used once.
- Judges compared. Jev 1.13 as shipped, with no task-specific training, asked “does the reply address the question the user asked?”. gpt-4o-mini as an LLM-as-a-Judge, asked the same question. And our model, which is a separate decision model and not Jev: Qwen3-1.7B with a LoRA adapter trained on the 600 turns.
- Scoring. Each judge’s yes/no threshold was chosen on the validation set, so no judge got a lucky cut-off. We report accuracy, agreement with the labels beyond chance (Cohen’s κ), and AUC for judges that return a probability.
- Tracking. Every training run is a Comet Experiment Management experiment, and the training data is a versioned Comet Artifact.
| Held-out test set (300 Ollie turns) | Our fine-tuned 1.7B judge | Jev 1.13 | LLM-J (gpt-4o-mini) |
|---|---|---|---|
| Did Ollie decline? Accuracy | 0.933 | 0.857 | 0.883 |
| Did Ollie decline? Agreement with labels (Cohen’s κ) | 0.842 | 0.651 | 0.720 |
| Did Ollie decline? AUC | 0.982 | 0.914 | n/a (yes/no only) |
| Was an attempted answer sound? AUC | 0.681 | 0.724 | n/a (yes/no only) |
| Training needed | 600 labelled turns, about 37 minutes on a laptop | None | None |
Building the judge, step by step
1. Label production decisions with an Opik online evaluation rule
An online evaluation rule in Opik is a scorer attached to a project. Every new trace that lands in the project is passed to a model together with a question, and the model’s answer is stored on the trace as a feedback score. The Ollie project from the previous post already had two such rules: one running Jev and one running gpt-4o-mini, both asking “does the assistant’s reply address the question the user asked?”.
To get reference labels, we added a third rule. It sends each turn to gpt-5.5 with three strictly worded questions and stores the three answers as feedback scores on the same trace:
on_topic: is the user’s request within Ollie’s scope (Opik, Comet, LLM observability, evaluation, tracing, or the product itself)?declined: did Ollie decline or redirect instead of attempting a substantive answer?answer_sound: treated as an attempt to answer, is the reply responsive and factually sound for what was asked? A decline counts as not sound.
These three questions are also the scope policy the previous post said was missing. Its single question, whether the reply addressed the user, scored a polite refusal as a failure. Three labels separate a request Ollie should refuse, a refusal and a wrong answer.
Once the rule had run over the 1,000 turns, every turn carried five scores: Jev’s, gpt-4o-mini’s, and the three reference labels. We exported them with the Opik SDK, split them 600 / 100 / 300 as described above, and did not look at the test split again until the final table.
2. Start from a decision-model recipe
We started from Nish Tahir’s recipe for emulating a decision model with an ordinary LLM. Take a small open LLM (we used Qwen3-1.7B), show it a multiple-choice question, and read the probability of each option letter from a single forward pass. Nothing is generated, so there is no output to parse.
We framed each judge question as Yes/No and asked it twice, once with Yes as option A and once with Yes as option B, then averaged the two probabilities. Language models prefer whichever letter comes first, and they lean toward agreeing in general, so this reduces the position bias rather than removing it. Averaging log-odds instead of probabilities would be a small further improvement that we have not made yet.
Our post on training your own decision model measures the same bias on its own models: with the options listed in reverse, an untrained model changed its top answer on 30% of questions, and trained models still did on 5 to 9%.
def p_yes(model, tokenizer, turn, question):
probs = []
for options in [("Yes", "No"), ("No", "Yes")]:
prompt = format_prompt(turn, question, options) # INPUT / OUTPUT / statement / A. B. / Answer:
logits = model(**tokenizer(prompt, return_tensors="pt")).logits[0, -1]
p = torch.softmax(logits[[A_ID, B_ID]], dim=-1) # constrained to the two option letters
probs.append(p[options.index("Yes")])
return sum(probs) / 2 # reduces, does not remove, position bias
3. Track every run in Comet Experiment Management
Every training run is a Comet experiment: hyperparameters, a live loss curve, test metrics, the code that produced it and the resulting LoRA adapter, all in one place. That made it easy to compare a dead end against the run that worked. There was a dead end, and it is covered below.


4. Version the training data as an artifact
The model is only as reproducible as its data. We logged the train, validation and test splits, plus the 3,600 exact prompts the model was trained on, as a versioned Comet Artifact linked to the experiment that consumed it. If the labels change or we add more turns, that becomes version 1.1.0, and every experiment records which version it saw.

These are real user conversations, so the artifact and project are private. Keeping production-derived data inside the same governed workspace as the experiments, rather than in a bucket somewhere, is part of the point.
5. Fine-tune
We trained a LoRA adapter (6.4 million trainable parameters, 0.37% of the model) on the three label questions, in both option orders, for one epoch: 450 optimizer steps. The whole run, training and evaluation included, took about 37 minutes on a MacBook. The loss falls sharply in the first few dozen steps and then flattens; the model learns the shape of these decisions quickly.

6. Compare against the judges you already have
We scored the 300 held-out turns with our model and pulled Jev’s and gpt-4o-mini’s existing scores for the same turns. Each judge’s decision threshold was chosen on the validation set, not the test set, so no judge got an unfair cut-off. All of the comparison metrics were logged back to the same Comet experiment.

What we learned
An off-the-shelf small model is not a judge. Before any fine-tuning, Qwen3-1.7B said “yes, the reply addressed the question” to 97% of turns. Its accuracy looked respectable only because most turns are in fact answered; its agreement with the labels beyond chance was 0.05. A decision model gives you a clean probability. It does not give you judgment.
Fine-tuning on the wrong data doesn’t help. Our first attempt fine-tuned the model on CommonsenseQA, a general reasoning benchmark. It worked on its own terms, taking accuracy on that benchmark from 0.597 to 0.784, and it did nothing for judging. On Ollie turns it said “yes” to 100% of them, and on the on-topic question its accuracy fell from 0.77 to 0.33. Getting better at a general task is not the same as getting better at yours. Because both runs sat side by side in the same Comet project, that was a two-minute diagnosis instead of a mystery.
On its own decisions, the trained model wins clearly. Trained on 600 of our labelled turns, the same 1.7B model became the best judge we tested for the questions it was trained on:
| Held-out test set (n=300) | Before fine-tuning (AUC) | After fine-tuning (AUC) | After fine-tuning (accuracy / κ) |
|---|---|---|---|
| Declined | 0.638 | 0.982 | 0.933 / 0.842 |
| On topic | 0.772 | 0.927 | 0.893 / 0.740 |
| Answer sound | 0.678 | 0.771 | 0.703 / 0.337 |
Even when we asked our model the exact question Jev and gpt-4o-mini were asked, “does the reply address the question the user asked?”, it separated declines from attempts better than either: accuracy 0.913 against 0.857 and 0.883, κ 0.790 against 0.651 and 0.720. These decisions are pattern recognition over one product’s conversations, and that is exactly what a small model learns well from a few hundred examples.
The trained model does not win everywhere, and where it loses tells you why. On answer soundness the headline numbers look close: our model scores 0.703 accuracy against Jev’s 0.683. But most of that comes from spotting declines, which the label also counts as “not sound.” Restrict the test to the 213 turns where Ollie actually tried to answer, where the label really means was this answer correct?, and Jev pulls ahead: AUC 0.724 to our 0.681. Of our model’s 43 wrong “sound” calls, 35 were on-topic requests whose answers the reference judge marked as unsound.
That’s not a training problem; it’s a knowledge problem. Deciding whether an answer about Opik’s SDK is correct requires knowing Opik’s SDK. Six hundred examples of what good answers look like don’t teach a 1.7B model the facts behind them. A larger general model brings that knowledge for free.
What the off-the-shelf judges still give you. Neither Jev nor gpt-4o-mini needed a single label or a minute of training, and either can be pointed at a new question this afternoon. gpt-4o-mini also writes down its reasoning, which matters when a human reads the verdict. Our trained judge knows three questions and nothing else. Asking it a fourth means labelling data and training again.
Caveats worth stating. Our labels came from a stronger model, not from humans, so what we measured is how closely each judge reproduces gpt-5.5. Our model was trained on that labeller’s output, while Jev and gpt-4o-mini were not. That is the realistic setup for a team distilling an expensive judge into a cheap one, but it isn’t a measure of absolute correctness. This is one training run with a 300-turn test set, so differences of ±0.03 in accuracy are within noise. And the fine-tuned model’s raw probabilities are poorly calibrated (its best thresholds sat near 0.03 for two of the labels), so the ranking is trustworthy, but the raw number shouldn’t be read as a confidence until it’s recalibrated on validation data.
The calibration caveat has a known cause. We trained on one hard yes/no label per question, and Abby’s experiments found that this style of training leaves a model overconfident (ECE 0.12 on that post’s test set), while training on a distribution of labels brought it to 0.02.
When to train your own judge
| Train your own judge when… | Use an off-the-shelf judge when… |
|---|---|
| The question is about the shape of the interaction: declines, scope, format, tone, policy | The question needs world or domain knowledge: is this answer correct? |
| You’ll ask the same few questions on a lot of traffic | Your questions change often, or you’re still discovering them |
| You have a few hundred trustworthy labels, or a stronger model you can distil from | You have no labels and no budget to make them |
| A score is the whole output | A human needs to read why |
Summary
- Label your own decisions first. Without labels, you can compare judges but not grade them. An Opik online evaluation rule with a strong model is a quick way to start.
- Train on your task, not a proxy. General reasoning benchmarks made our model better at benchmarks and no better at judging. A few hundred examples of the real decision did more.
- Expect a trained judge to win on pattern decisions and lose on knowledge decisions. Declines and scope went to the 1.7B model; answer correctness went to Jev.
- Keep the work reproducible. Version the data as a Comet Artifact, log every run as an experiment, and keep the dead ends. They are how you know the winner is not a fluke.
Get started
- Comet Experiment Management: track fine-tuning runs, version training data as artifacts, and compare runs side by side.
- Opik online evaluation: score production traces with Jev, an LLM-as-a-Judge, or a labelling model.
- Jev vs. LLM-as-a-Judge: the experiment this one continues.
- Train your own decision model: how decision models work, with a full build, calibration tests and public code.
- Nish Tahir’s constrained decoding guide: the constrained-decoding recipe this post builds on.
