{"id":20803,"date":"2026-10-26T20:07:13","date_gmt":"2026-10-26T20:07:13","guid":{"rendered":"https:\/\/www.comet.com\/site\/?p=20803"},"modified":"2026-10-26T20:17:25","modified_gmt":"2026-10-26T20:17:25","slug":"train-a-decision-model","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/train-a-decision-model\/","title":{"rendered":"Train your own decision model"},"content":{"rendered":"\n
\"Train<\/figure>\n\n\n\n
\n
GitHub Repo<\/a><\/div>\n\n\n\n
Comet Project<\/a><\/div>\n\n\n\n
Opik Project<\/a><\/div>\n<\/div>\n\n\n\n

Decision models drew wide attention in September, when TypeSafe AI launched Jev<\/a>, a model that returns probabilities over a fixed set of answers instead of generating text. Open-source alternatives followed almost immediately: Latent.Space counted six in two days.<\/p>\n\n\n\n

Most writing about decision models since then defines them by their interface, which uses some sort of schema or structured output. But an LLM with structured output produces something almost identical. What makes decision models unique is how they generate this output and what their probabilities are trained to mean.<\/p>\n\n\n\n

This article explains that difference and what it changes in practice. It defines what a decision model is, builds a small one on a transformer backbone, tracks its training in Comet, and tests the trained model in Opik to check whether it has the properties that decision models promise.<\/p>\n\n\n\n

What is a decision model?<\/h2>\n\n\n\n

A decision model takes an input, along with one or more questions whose possible answers are declared in advance, and returns a probability for each answer. The clearest way to see what that changes is to compare it with how an LLM produces structured output.<\/p>\n\n\n\n

A structured-output LLM generates an answer token by token and constrains those tokens to fit the provided schema. Each output token costs another forward pass, later answers in the same response are conditioned on earlier ones, and the only probabilities available describe the next token rather than whether the answer is right.<\/p>\n\n\n\n

By contrast, a decision model reads a probability for every allowed answer from a single pass, and those probabilities are trained to match how often each answer turns out to be right. This means cost and latency depend on the input rather than the length of the answer, several questions about the same input can be answered without influencing each other, and a 0.9 is meant to be right about 90% of the time.<\/p>\n\n\n\n

Decision models vs. classifiers<\/h3>\n\n\n\n

If that sounds like a classifier, it\u2019s because a decision model is much closer to one than to a chatbot. Transformer models come in two broad families. BERT-style encoders read the whole input at once, with every token able to attend to every other token in both directions, and they\u2019re pretrained by predicting hidden words from the context on either side. With a small head on top, they\u2019ve been the standard tool for classifying sentiment, spam, or intent since BERT came out in 2018. GPT-style decoders read left to right and are trained to predict the next token. They\u2019re built to generate.<\/p>\n\n\n\n

A decision model does the encoder\u2019s job: read the input and return a distribution over answers. What\u2019s changed is where the task comes from. A classic BERT classifier fixes its labels at training time, so adding a label means retraining the head. A decision model receives its questions and options with each request. A common summary of Jev is that it\u2019s BERT-like, but with the data, compute, and training recipe of modern LLMs. TypeSafe hasn\u2019t published Jev\u2019s backbone, so that describes the approach rather than a confirmed architecture.<\/p>\n\n\n\n

Calibration<\/h3>\n\n\n\n

The difference in what the probabilities mean comes from training. LLMs are post-trained toward answers people prefer (RLHF) or answers that pass an automatic check (RLVR), and neither directly rewards a model for saying how likely it is to be right. TypeSafe trains Jev, for example, with a method it calls Reinforcement Learning for Calibrated Decisions (RLCD). The goal is calibration, or probabilities that match how often each answer turns out to be right.<\/p>\n\n\n\n

Calibration is important because not every wrong decision costs the same. Missing an urgent ticket might cost far more than escalating one that could have waited, so the right cutoff isn\u2019t always 0.5; it depends on what each mistake costs. Calibrated probabilities let you set that cutoff in code: escalate a ticket above 0.2, auto-approve a refund above 0.95, and send the uncertain cases to a person. The same model can serve all of those decisions, and when costs change, you move the threshold instead of retraining.<\/p>\n\n\n\n

But \u2018calibrated\u2019 isn\u2019t the same as \u2018accurate.\u2019 A model that says 0.6 on every item and is right 60% of the time is perfectly calibrated, but it tells you nothing about which items to trust.<\/p>\n\n\n\n

Accuracy measures how often the top answer is right. Calibration measures whether the probabilities match how often answers turn out right, so a model can be accurate and overconfident, or calibrated and barely better than guessing, and a useful decision model needs both<\/strong>.<\/p>\n\n\n\n

When to use a decision model<\/h3>\n\n\n\n

Decision models don\u2019t replace LLMs. LLMs remain the right tool for open-ended work like chat, coding agents, and anything that produces text a person reads. Decision models fit tasks where the possible answers can be listed in advance: classifying, routing, scoring, gating, and judging other models\u2019 outputs, often at high volume or under tight latency limits.<\/p>\n\n\n\n

<\/td>Generative LLMs<\/strong><\/td>Decision models<\/strong><\/td><\/tr>
Returns<\/strong><\/td>Text<\/td>A probability for each allowed answer<\/td><\/tr>
Answer space<\/strong><\/td>Open. Output must be parsed and validated unless a schema constrains it.<\/td>Declared in advance; output is always one of the declared answers<\/td><\/tr>
How the output is produced<\/strong><\/td>One token at a time, each conditioned on the ones before<\/td>Read out in a single pass<\/td><\/tr>
What the probabilities mean<\/strong><\/td>Likelihood of the next token<\/td>Trained to match how often each answer turns out right<\/td><\/tr>
Cost and latency<\/strong><\/td>Grow with input and output length<\/td>Grow with input length only<\/td><\/tr>
Several questions on one input<\/strong><\/td>Later answers are conditioned on earlier ones<\/td>Can share one pass over the input while keeping questions independent<\/td><\/tr>
Explains its answer<\/strong><\/td>Yes, it can write out its reasoning<\/td>No, it returns only the probabilities<\/td><\/tr>
Best fit<\/strong><\/td>Open-ended work: chat, writing, code, agents<\/td>Tasks with answers you can list in advance: classifying, routing, scoring, gating, judging<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Build a small decision model<\/h2>\n\n\n\n

The clearest way to understand what makes a decision model unique is to build one. In this section, we\u2019ll use Qwen3-0.6B-Base with LoRA adapters and a decision head trained from scratch on typed-decisions<\/a>, a public benchmark on Hugging Face (Apache-2.0). Each case pairs one input, which Jev calls the state<\/a> (for example, a summary of an agent\u2019s run), with five typed questions about it. Every gold answer is a probability distribution rather than a single label. <\/p>\n\n\n\n

We\u2019ll break this down into four pieces: the data format, packing one state and several questions into a single sequence, the head that turns hidden states into probabilities, and the loss that trains those probabilities to mean something. The full code, including training, evaluation, and Comet logging, is in the companion repo<\/a>. The excerpts below are trimmed, and some variable names are changed for readability.<\/p>\n\n\n\n

The data<\/h3>\n\n\n\n

The typed-decisions benchmark covers four workflows:<\/p>\n\n\n\n