What happens behind the scenes when we change effort for same LLM models?

ttbharath about 3 hours ago 6 comments

FR version is available. Content is displayed in original English for accuracy.

What really happens behind the scenes when we select high effort vs medium effort for the same LLM models?

⚡ Community Insights

Discussion Sentiment

83% Positive

Analyzed from 378 words in the discussion.

Discussion (6 Comments)Read Original on HackerNews

neongreen•42 minutes ago

This is a very good question and the answers are different for different models.

---

Deepseek v4 Pro has three modes: no thinking, thinking ("high"), very good thinking ("max"). Max is implemented as follows:

https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...

> When reasoning_effort="max" is set, a special prefix is prepended at the very beginning of the prompt (before the system message) to instruct the model to maximize its reasoning depth: > > Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios. Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.

---

Inkling, apparently, trained their model to treat effort level as "how cheap is one reasoning token":

https://magazine.sebastianraschka.com/p/controlling-reasonin...

> During large-scale RL, they did two things for each sample: > > 1. Specified the desired effort level in the system message. > 2. Adjusted the cost assigned to each generated token. > [...] > > Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly.

bellowsgulch•34 minutes ago

I wish I could link it, but I had also read another strategy for reasoning effort, perhaps the most naive implementation, described as whether or not you ignored or accepted the end of sequence token, n-times, n being some count of subsequent ignores.

Now how true that is, I don't know. You could reasonably implement this locally with an inference provider, but I don't know any that do, or I haven't used them at least.

I may also have the technical detail incorrect, but it did come down to essentially "continuing inference beyond a natural termination."

boveyking•about 2 hours ago

You have a restaurant and have 2 types of chef.

One is the sous chef , who knows recipes and cooks fast, handle most dishes as quick as possible. He does not think much. That is your medium effort llm in charge.

One is the head chef, who is inspecting, pause, consider edge case. Slower but catches what sous chef missed. That is your high effort llm in work.

tbharath•about 2 hours ago

But I was thinking Chef is the LLM model. So I was wondering if it is the same model, what actually changes the effort? Is that part of the inference?

OKRainbowKid•about 2 hours ago

So are you saying that effort level isn't a parameter to an LLM, but that each level actually corresponds to a slightly differently tuned LLM?

cyanydeez•about 1 hour ago

llamacpp has the basics: reasoning-budget limits thinking token output and reasoning-message thats injected when budget is exhausted. client can set these per request so dynamics are possible.

models dont seem to care if you cut them off.