DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
57% Positive
Analyzed from 2505 words in the discussion.
Trending Topics
#context#models#prompt#don#less#using#llm#prompts#more#local
Discussion (74 Comments)Read Original on HackerNews
At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.
I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject.
If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt.
The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.
Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.
How do you all keep your context so low? Complex tasks just balloon it in my experience.
To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too).
To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices.
To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time.
When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).
I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has.
Lately I haven't been saying go to X file and change Y, I say you have SSH keys to the embedded hardware: plan, deploy, and test latency improvements to the X stage of the image processing pipeline, parity test output images to avoid regressions, and target a 5ms latency improvement, ideas to look at are M, N, and P image processing techniques, I think latency is in this domain is mostly caused by Y, but verify, and let me know when you have a plan ready to implement.
Then it churns for 4 hours, it builds and runs it's own test harness and profiler, and autonomously works until it has a changeset ready to go.
That's how I get to 600k, and it seems to work fine? It got us 4.8ms today, through SIMD improvements and using a technique I hadn't thought of in place of a more expensive operation.
By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps
You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc...
I could see this scale to multiple kiltobytes of metadata in the prompt easily.
RAG was supposed to be the way out on that, and ended up being mostly abandoned.
Unless you're a contracting agency!
A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do.
I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on part with the SOTA a few months ago, and my opinion, it likely is around the pinnacle of advancement without high inefficiency in preparing with the current LLM recipes. Further, I'm of the opinion America's SOTA is hitting the real cost-sigmoid and there's no singularity in site. These things will hack the planet if you put them in a group of agents and tell them to do it; but as context grows, the probability that they can answer "how many r's are in strawberry" goes down. No amount of parameters it going to erradicate that.
But I digress, my new stage of working with LLMs is figuring out how to use Qwen3.6-35B-A3B as the entry point to collect the context, and then use the big boys Qwen3.8 to make the edits, then degrade back and forth to complete changes. There's no harness yet for this, but there's clearly an intelligent way to setup a engineering harness.
And I do understand people have codebases that simply can't easily live in the smaller (~128k-256k) context windows, but instead of porting your codebase to another novel language, breaking it into context-aware components would make it closer to what these things can do.
And I'll repeat: I don't think we're approaching the singularity of self-recursion primarily because the LLMs will duplicate errors, context poison, and whatever else they encounter and there's no human who can sit around correcting it constantly. The American AI apparatus should cut their models like the Chinese down and work on real problems and stop imbibing the singularity-watts-are-all-we-need drugs.
I agree, prompt adherence seems to get worse when operating on large inputs. Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)
Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster).
If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.
And there's only so much attention to go around.
This was certainly true when I first tried the new models with a 1M context. After 200k things got weird pretty fast. I haven’t had that problem since Opus 4.8. I’m regularly bumping against 800k tokens in “lazy” adhoc sessions. “Lazy” in that I ought to do as you suggest, in the way that I ought to refactor this code, I ought to factor out the meat of this session, but in the moment it’s still producing useful output! Tool harness is a force multiplier too: tools that put all tool use in subagents are incredibly frugal with the main chat session.
I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.
These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash
I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.
So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
I have to observe:
Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools.
Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though.
I settled on 64k tokens because of a recommendation from an LLM. I now see that that guidance is associated with similar sounding (but not the same) configuration constraints. So this weekend I’ll try turning it to 11 (250k tokens?). Anticipating some hard locks over here.
I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do?
(I kind of hate to abandon Claude Code, as it seems to be the most capable coding assistant of the limited set of tools I've tried. But that horrendous context bloat is really painful!)
> Anthropic forbids its use on their subscriptions
This! How are these companies even allowed to do this while they anyway charge for either API access or limit usage in the generic plans. It's blatantly just "I don't want you to spend less per generic task!".
It unfortunately feels like we will be stuck waiting for a burst bubble before local hardware can be reasonably acquired for personal LLM usage.
Using frontier providers to manage prompt tuning experiments has been very exciting. Might be something there.
I thought the interesting cyber stuff is really at the edge of frontier
Even for $10K you get mediocre performance.
Comes down to how much of the ambiguity we expect out of the model.