Ask HN: How are thinking efforts implemented?

ssimianwords 4 days ago 19 comments

HI version is available. Content is displayed in original English for accuracy.

Claude and ChatGPT have thinking efforts where you can tune the amount of thinking allowed.

Like low, medium, high, xhigh and so on.

But are they different models underneath? Or same model with different parameter?

The reason I ask is because, if I change the effort param mid conversation in Claude code, I get a warning suggesting I’m breaking the cache.

I don’t think this happens in Codex because when I change the effort, the responses are still quick.

⚡ Community Insights

Discussion Sentiment

80% Positive

Analyzed from 577 words in the discussion.

Discussion (19 Comments)Read Original on HackerNews

pyentropy•4 days ago

Take a look at the harmony repo which specifies the internal OpenAI format - the effort level is specified in the context after the <|start|> tag - https://github.com/openai/harmony

Note that inference libs also have parsers that put hard limits on reasoning tokens with separate counters (similar to how you can put a limit on token generation per completion versus waiting for an <eos>). For that, take a look at vllm reasoning docs.

pyentropy•4 days ago

Examples with inference of different reasoning effort levels is in the OpenAI docs as well - https://developers.openai.com/cookbook/articles/openai-harmo...

https://docs.vllm.ai/en/latest/features/reasoning_outputs/#a...

https://developers.openai.com/api/docs/guides/reasoning

simianwords•4 days ago

I think you have the right answer but I'm struggling to understand: does changing the effort change the prompt at the start of the conversation? I wonder why come up with this way at all? Why not just add a parameter at the end or something? At least it won't break cache.

Maybe like: add a secret suffix to your chat in the conversation to think more like

   conversation....

   Hey please help
   [think more]

pyentropy•4 days ago

I'm considering the possibility that it's good to break the prefix and cache because the LLM itself was rewarded (during post-training) with different prefixes/system prompts, each containing reasoning traces of the correct size.

I might be very very wrong though and LLMs disagree with me, insisting that cache is preserved and the system message doesn't have to change (even though it often contains effort level in context) if effort level changes across turns, and that all you have to do is tell the inference lib that parses think tags to early-close think tags that are too long.

aabdi•4 days ago

Different models do slight variants.

Usually it’s done in post training to enforce behavior based on prompt. Ie. System prompt with thinking:max or low or wtv.

Enforcement then goes via constrained decoding, checking for think token start and end with max lengths, or other variations

bjourne•4 days ago

LLMs work by generating the most likely continuation to a prompt. But they can also generate multiple likely continuations. This create multiple branches which in turn can generate even more branches. The LLM can then evaluate the branches, prune the unpromising ones, and merge the best ones. More branches means more tokens, means more effort.

simianwords•4 days ago

this has nothing to do with the thinking effort however

bjourne•4 days ago

Yes, it does. Breadth of search is exactly what the effort setting controls.

pyentropy•4 days ago

LLM-judge/parallel branching ≠ multi-token prediction ≠ reasoning effort.

See https://developers.openai.com/cookbook/articles/openai-harmo... and src/openai/types/shared/reasoning_effort.py

FergusArgyll•about 2 hours ago

I think you may be confusing the openai "pro" series models with thinking. Thos are rumored to be multi "branched"

__patchbit__•4 days ago

At a guess. May be associated with token length context window. Down selecting is consistent with warning message, forcing cutoff to context window. The technical term cache being a synonym. Increasing the headroom for more "thinking" should allow the implementation to access more resources without warning about the cache breaking.

sometimelurker•4 days ago

they use multitoken prediction behind the scenes, that might interact with the CoT in a strange way. maybe for different thinking modes they have different MTP models? if so thats interesting

pyentropy•4 days ago

The number of tokens you predict at time (multi or not) has nothing to do with whether the model wants to emit any, some or a lot of reasoning tokens in reasoning tag -- similar to how branch prediction will not really change the for loop iteration count.

sometimelurker•4 days ago

no it might. a high reasoning task is probably harder than a low reasoning task, so the same MTP LLM will predict more correct tokens on the low reasoning task. to compensate for this, big labs likely have different MTP LLMs for different cases. it would make sense for them to do this

Yahyaaa•3 days ago

Usually it’s not a different model, it’s the same model with different inference-time settings. “Thinking effort” typically changes the compute budget and decoding behavior (how many steps, how much exploration, sometimes internal planning loops).

Some stacks also tie it to orchestration layers or system/prompt signals, which is why it can look inconsistent across products