Every call to a modern model now has a dial for how long it thinks before it answers. I turned it through every setting on the same 20 problems, graded by code, and measured what each notch buys: accuracy, time and money.
Reasoning models think in hidden tokens before they answer. You choose how much with a single setting, from “none” to “max.” Thinking costs money and time, and you can’t see it.
Most teams pick a level once and forget it. Turn it up and the bill and the wait grow; leave it down and quality might suffer. Few have measured where the returns stop.
OpenAI’s own advice is to use the top settings “only when evals show a clear benefit.” This is that eval, small and public.
Effort is the cheapest decision in an AI product and one of the least examined. It sets latency for every user and a multiplier on every bill. I wanted my own numbers before recommending a default.
The ideaWrite fresh problems a model can’t have seen, grade them with code, and run the same prompt through every thinking setting and two model sizes. Then pick settings per job, not per company.
Sources are listed at the end of the page.
Each circle is one run of one problem. Its size is how much the model thought; its color is whether the answer was right. Luna ran every problem twice.
Two runs of 20 problems at each setting. The light band shows the two runs separately.
Median thinking tokens per run, GPT-6 Luna. Shading is relative to the largest value.
Measured on this problem set at published prices. Your tasks will differ; the shape is what carries over.
Thinking level becomes a per-task policy, owned like any other product setting.
Guessing which setting is “safe” goes away; a small eval answers it in an afternoon for under a dollar. Deciding which mistakes matter, and what a user will wait for, stays with people.
Describe your workload as a mix of these problem types, and see what each setting would cost and get right on it.
# One call. Only "model" and "effort" change between runs. POST /v1/responses { "model": "gpt-6-luna", "reasoning": { "effort": "medium" }, "instructions": "Solve the problem. … end with one final line: ANSWER: <answer>", "input": "How many 4-digit positive integers have all different digits, …", "max_output_tokens": 32000, "background": true } # → usage.output_tokens_details.reasoning_tokens: the hidden thinking you pay for
Twenty problems in five types, two runs per Luna setting, one run per Sol setting, and Sol only at low, medium and high (plus medium on the four hardest). That is enough to see the shape, not to rank close settings: high scoring below medium here is within run-to-run noise. The problems are mine and lean toward exact answers; open-ended writing or coding could respond to effort differently. Two Sol answers on the hardest problems were code instead of a number and count as wrong, which is what a user would get.