DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
67% Positive
Analyzed from 4478 words in the discussion.
Trending Topics
#context#models#more#agents#llm#model#rules#long#things#prompt
Discussion (83 Comments)Read Original on HackerNews
Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue.
Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) In those cases, the model's bounded amount of "high-quality" attention (inherited from a short-context base model) can easily attend to the prompt (which, for added ease-of-recognition, usually appears either at the very beginning or very end of the context); while the additional "low-quality" attention trained in during long-context training can attend "well-enough" to the data to perform the task the model needs to perform.
Anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.
And trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window.
Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"
I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)
After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ...
... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something.
Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001.
All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context.
But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context.
The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes.
Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.
Ah yes, magic that costs the same as a new car.
If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain.
If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results.
Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense.
And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
The coding harness is what's getting it to continually align to your current instructions.
This is very obvious with local models.
You need to let go of the idea that this is something LLM's can do. They can't. At best, they can bias towards rules conformance with more or less likelihood, but coverage and conformance both go down super-linearly as you accumulate more rules, more context, and more output in a session. That's simply the nature of how these tools work and you need to engineer your workflows around it if you want to use them.
If you absolutely need some rules enforced, you need to adopt some framework for validating those rules that then rejects, reprocesses, or repeats any session that fails to satisfy them. In the best case scenario, this is some traditional deterministic validator (like a linter, compiler, analyzer, exhaustive test suite, etc in coding) but if you need to process in stochastic space because its something rich and ambiguous like natural language itself, then you want to dispatch a swarm very narrow, task-focused subagents that each validate against a very constrained subset. (And prepare yourself to have those to fail sometimes too. LLM's are noisy and cannot deliver strict rule enforcement on their own.)
In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.
Then there is the "hard control", which is the inviolable that the LLM cannot bypass.
You need to move as much as is technically possible to either hard or (failing that) advisory controls.
A good programming harness is basically a stack. A good stack keeps building each layer. You _cannot_ pull things off the bottom of the stack because that's an expensive cache hit; but you can pull things off the top. So what your harness should be doing: <tools> <rules> <user content> onto each request _then_, when you get to the next request or result, pulling that out if there's some change.
So you can see it's the cache that's either exponentially growing or having to cache bust to keep it fresh.
Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness.
What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs.
It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.
And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases
With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month
How many arbitrary rules can you memorize? Do you even remember them two years later? If you do, then we can get to your statement.
And your statement is true. Humans do not run every action through a memorized list of rules, to see if any of them block the action. We don't. We're not going to, either, no matter how badly the policy manual writers want us to.
The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.
Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.
Specifically, having agent implementation drift from the Spec.
I've done a ton of testing and the model doesn't matter, fable or sol still miss a ton of detail and drift.
I'm building http://engine.build which closes the gap and makes sure the implementation matches the spec.
It's not the same as the satisfaction you get when solving complex problems with code yourself but writing clear specs and thinking through the problem is still very satisfying to me.
How many times have you told an agent not to do something then had to correct it?
You must always flip the frame. Objective analysis is way better with llms than steering via skills.
This is just a small example of why "loops" became popular for a minute and now it is "graphs"
If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want.
There's a reason the main agentic task LLMs excel at are coding tasks, it's the way of working of the creators, and they understand intimately the flow and can train for it.
I believe the true way will be able to easily fine tune models on your agentic use cases, but it would require a big company to compile a huge dataset on it's way of working and I don't think anyone wants to be the first.
In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
If people spent more time studying about AI and how it works, they would realize that the default should be to one shot prompt your task with a big, cached system prmopt, with an user prompt that is just dynamic data, specified to the cheapest model that can do the job.
Unless you really can't do this given your problem, you should try to make a graph of well defined, step by step oneshot prompts, and THEN if your problem still can't be solved with that, then you start leveraging agents.
Despite this giving better results, and being more cost efficient, is evidently too much work then just letting the AI do all the work.
"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers
Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.
Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicators and unknowns -> LLM call estimates deposit potential and confidence -> database lookups inject regional ore demand, nearby deposits, infrastructure and historical yield data -> LLM call combines geological evidence with business context -> LLM call generates an investment recommendation and rationale.
That's what I mean by graph. Every step is a separate LLM call with a well-defined responsibility, consuming the output of the previous node. Each node can be tested, benchmarked, retrained, replaced, or monitored independently. Why would you use an agent here? You can cache every single system prompt on each call making your total token output much cheaper than having a full 'output' only token generation workflow which is what happens with agents.
There is nothing to discover. The workflow is already known. The company already knows how geologists evaluate prospects. The company already knows what data sources matter. The company already knows what the final output should look like. You don't want the model deciding which tools to call, which reasoning path to take, or which pieces of information are important every single run. You want the exact same process applied to every report so results are consistent, measurable, auditable and debuggable. My default is: One-shot prompt -> if not enough -> graph of LLM calls -> if not enough -> agent. A surprising amount of enterprise AI is really just: Unstructured input -> extraction -> classification -> enrichment from databases -> decision support. Not: Unstructured input -> autonomous agent spends 20 steps deciding what to do next.
Agents make sense when the workflow itself is unknown.
If the workflow is already understood, a graph is usually cheaper, more reliable, easier to evaluate, easier to debug, and less dependent on whatever synthetic "agentic" behaviors happened to get reinforced during post-training. I am sure people default to agents mostly because it's less engineering work than explicitly modeling the process.
i thought it was well known that claude code got good at coding because anthropic bought tons of coding data from companies like mercor.
If the stakes are high, e.g, proceeding in ignorance could lead to prison time, people will favor inaction, even if the policy technically permits a corner case. If the stakes are low, people will completely override policy for the path of least resistance.
I feel like “discretion” parties missing. Do we need some kind of special discretion model?
From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"
At some point between the model i was using and the previous version of it, Claude started inserting massive comments with references to tickets and other tasks. All this while having specific directives on the CLAUDE.md
Since then I resorted to developing my crapware as if I was the floor manager of a vehicle assembly line, and I have a few highly-specialized sub-agents running errands around the main session, but only ever taking care of a single concern. The main session builds with the knowledge contained in things like CLAUDE.md but the sub agents make sure things like the no/low-comments directives are either enforced, or factored into the final product.
When "performance" breaks down over long lengths, one could attribute it to a lapse in attention, but one could equally suspect inconsistent instructions.
The fewer instructions and conditions that need to be simultaneously met the easier it is to comply, but with more and more instructions one is bound to introduce internal inconsistencies within the instructions.
[0] https://github.com/deepclause/deepclause-sdk
Glad to be able to put some numbers on it.
Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.
I'm probably going to need to enable branch protection on my personal projects... What a pain.
It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over
https://venturebeat.com/technology/cursors-composer-2-was-se...
It does what its has been trained to do. So find out what its trained to do and just use it to do that. This is not general intelligence.
Plus the model's capacity to take more context into account and actually integrate it to the output is simply limited by the number of activated parameters. If you give it a playbook, you are forcing to choose it between attending to the playbook and the task at hand.
If you want to force it to work step-by-step, you need to present the steps one-by-one. Ideally with rules for the current step at hand and maybe relevant input again, depending on overall task size.
Why did you think models love to re-read files before editing them? It increases recall quality and thus edit precision and thus benchmarks.
not sure I got it?
Separately, the frontier labs are kinda pushing us into that behaviour by releasing models with ever-larger context windows.
Multiple rounds of generating small contacts documents that grow from the original idea , trying to keep each slice small enough to process for a human to approve/disprove .
Eventually it leads to a long list of tasks grouped by functionality. You start a new context and the orchestrator agent dispatches tasks to sub agents with a limited amount of information provided to each sub agent.
Also should have adversarial review and approval gates with other agents and roles.
Dont try and handle the entire project in context.
Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!).
Then work on small-medium tasks at a time with a fresh context.
At the end of your task, ask the agent if there are any key points about the project layout or architecture it would want to add to memory - audit those manually and amend your AGENTS.md accordingly.
If your code is well structured and you keep your tasks localised, you can get away with seemingly minuscule context windows.
It's also worth noting that high effort models love to slurp up whatever context they can. You almost never need/want high effort for non cross-cutting tasks.
The prompts reflect the actual jobs enterprise workers perform every day. Each task drops an AI agent into a live company environment, requiring them to cross-reference an extensive, multi-section handbook against a cluttered inbox, a multi-channel Slack workspace, Jira queues, and a stack of files (spreadsheets, PDFs), and working out both what to do and what the handbook forbids.
https://github.com/surge-ai/handbook/tree/main