A new kind of AI model answers in a third of a second and attaches a confidence number. I tested what that number means, alongside a regular chat model and plain code, on the same contracts with the same answer key: ask again, reword it, and cut the evidence out.
Confidence scores are quietly becoming product logic. “Above 90%, act automatically; below, ask a person” is now a line in a lot of code. Decision models from OpenAI and TypeSafe make it easy: every answer comes back as a probability, not a paragraph.
But a probability is only useful if you know what it measures. Is it how often the model is right? How stable the answer is? How much evidence it found? Teams are setting thresholds on a number nobody has explained to them.
If a 90% answer is right only 90% of the time, that’s a policy you can write. If it means something else, every threshold built on it is a guess.
Decision models sit in a new spot between a chat model and code: fast and cheap like code, flexible like a model. In Second Opinion I built on their probabilities. Before recommending that to anyone, I wanted to know what those numbers are made of.
The ideaRun one task at every level, from prose to probability to plain code, measure what each one’s confidence actually tracks, and use each level only for what it’s good at.
Sources are listed at the end of the page.
The same request, sent five or six times to each level. Each dot is one run, placed by the chance it gave that the clause is present. Rules and Decisions never move; the chat model can change its mind.
Answers grouped by how sure each level said it was. On the dashed line, a level is right exactly as often as it claims. Below the line, it’s overconfident.
Rules and prose answers carry no confidence, so every answer counts as “sure.”
Neither vendor explains how its probability is produced. Here is what each one publishes, and what a test can add.
| What the docs say | What they don’t say | |
|---|---|---|
| OpenAI Decisions | A predicate returns “an estimate from 0 to 1 that the condition is true.” Choice and score return a distribution plus a confidence field. Set thresholds from your own labeled examples, weighing the cost of each kind of mistake. | How the estimate is made, whether it’s calibrated, what “confidence” means, whether repeat calls match. |
| TypeSafe Jev | A yes/no question returns “the probability the answer is yes.” Confidence is a published formula over the probabilities: for yes/no it equals |2p − 1|. It summarizes how peaked the answer is and is “not the only” reasonable summary. Act on high, confirm medium, route low to a person. | How the probability is produced, what the model is, whether it’s calibrated or deterministic. |
| Chat models (Luna) | With reasoning off, the API can return the probability of each output word. That is the raw signal a yes/no probability can be built from. | Stated confidence (“I’m 95% sure”) is just more generated text, not a measurement. |
The same 240 answers, every level. Speed is the median for all 12 questions on one contract; cost is what the calls actually billed. Luna’s word probability needs one call per question, so it costs more. The clause-cut row isn’t ranked: rules miss many clauses to begin with.
This is the shape Second Opinion uses, measured end to end.
A confidence number is a policy input, not a fact. Someone has to own what it means.
Debating whether to trust “the AI” goes away; each level has a measured job. Choosing what a mistake costs, setting the line, and deciding the uncertain cases stay with people.
Pick a level and the confidence above which you’d let it act on its own. See how much it would handle, and how often it would be wrong.
# Luna through the regular API: reasoning off, one word, and the probability of that word POST /v1/responses { "model": "gpt-6-luna", "reasoning": { "effort": "none" }, "input": "CONTRACT: … QUESTION: Is consent required to assign this contract? Answer with one word: Yes or No.", "include": ["message.output_text.logprobs"], "top_logprobs": 5 } # → "Yes", logprob −0.137 → P(yes) ≈ e^−0.137 ≈ 0.87
One task and one dataset. Repeat and prose tests use 20 contracts (240 answers); calibration for the Luna levels uses 40; the clause-removal test uses 35 clauses, and the labels can miss a second mention of a clause, so some “still yes” answers may be right. Luna’s word probability comes from the top few word alternatives the API returns. The claim that Decisions scores answers the way Luna’s word probability does is an inference from behavior, not a documented fact. Vendors can change models and defaults at any time, so these numbers are a snapshot from October 2026.