Most AI products show you the model’s first answer. On 900 contract questions checked against lawyers, that answer was wrong about 1 in 9 times, and it sounded just as sure when it was. This build spends a few cents checking the work, and measures what each check buys.
Every commercial contract gets read for the same handful of questions. Can they end this if we’re acquired? Is our liability capped? Does it renew on its own? A paralegal or junior lawyer reads the whole agreement to answer them, and a single missed clause can cost far more than the reading did.
AI can answer those questions in seconds. The catch is that it answers every one in the same confident voice, so the wrong answers look exactly like the right ones.
A missed change-of-control clause or an uncapped liability is exactly the kind of leak that number describes. Reading slowly is the everyday cost. Trusting a fast answer that happens to be wrong is the expensive one.
Two new “decision models” launched this fall: TypeSafe’s Jev and OpenAI’s Decisions API. Both return probabilities instead of prose, which makes something possible that chat answers don’t: measuring how sure a model is, and spending more only where it isn’t. Contracts are a good test because the stakes are real and lawyers have already labeled the answers.
The ideaDon’t trust the first answer. Ask two cheap decision models, calibrate them per question, deliver only what they’re sure of, make every answer prove itself with a quote from the contract, escalate the rest, and hand whatever is still uncertain to a person, with the evidence attached.
Sources are listed at the end of the page.
75 contracts the system never saw, 12 clause types each. Every square is one answer. Step through the ladder, or tap a square for its paper trail.
Every number below follows where you stop the ladder. Public figures are linked; assumptions are yours to change.
Each point is one step; the vertical lines are 95% ranges from resampling the contracts. Lower is fewer silent errors; further right is more review. A team picks the point that matches its risk.
Today, a reviewer reads every agreement end to end. With a second opinion, they confirm the answers the system wasn’t sure of, each with the passages and votes attached.
Reading boilerplate to find out an agreement has no non-compete goes away. Deciding what a clause means for the deal, negotiating it, and signing off on anything the system was unsure of stay with people. The answers it still gets wrong are why someone should skim the rest:
| Clause | Lawyers | Final |
|---|
# One request per contract: 12 questions share one read of the text (input billed once) POST /v1/decisions { "model": "gpt-6-luna", "input": "<the whole contract>", "questions": [ { "type": "predicate", "name": "change_of_control", "instructions": "Does one party have the right to terminate, or is consent or notice required, if the counterparty undergoes a change of control?" }, … 11 more clause types … ] } # → { "answers": [ { "name": "change_of_control", "probability": 0.94 }, … ] }
| Step | Wrong · to a lawyer | Cost | What happened |
|---|
One dataset, 150 contracts, and contracts under about 12,000 tokens, which may be easier than the long ones. The lawyers’ answer key has gaps too: some “invented clauses” are real clauses the labels missed (contract 075 can be ended without penalty on 60 days’ notice but isn’t labeled as termination for convenience), so the error counts are conservative. The escalation rule was chosen on only 60 training answers, and Astra reuses it rather than being validated separately. On the hardest calls the disagreement is often about the definition itself: both large models read broad indemnities as “uncapped liability” and the answer key doesn’t, which is exactly the kind of call a lawyer should own. And the Decisions API is in public beta; its behavior may change.