HI version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
78% Positive
Analyzed from 3466 words in the discussion.
Trending Topics
#next#token#llms#llm#word#training#tokens#predictor#predicting#prediction
Discussion (74 Comments)Read Original on HackerNews
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co.
There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
Both systems have emergent behavior that goes beyond what would naively be expected.
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
Also known as predicting.
An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
But when we talk about humans, we're not talking about the chemicals involved in those humans.
When we talk about LLMs, the tokens are the valuable thing they produce for us. We want LLMs because they give us sequences of tokens.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
I don't think you intended this, but the word choice here gave me a chortle.
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
The output of the LLM is literally a probability distribution of what the most likely next token is.
Recursively invoked.
With carefully selected context.
And massive investment in RL to tune token selection.
And the ability to use cli tools on other folks' machines.
That's a powerful system built around a conceptually simple technology: Next token predictors.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
A pure next-token language model won't be able to give detailed instructions to an ensemble of motors, mimicking a human body, to do a wide variety of tasks our human brain is excellent at doing, for example, inserting keys into a car, opening the door, sitting down, starting the car, putting the car in reverse, and exit a parking lot, being careful not to hit anything.
They still can't do code accurately. The fact that you use this as a defense of your position greatly undermines the credibility of your claim.
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
https://youtu.be/GlYgs6v2YfU?is=IxVMhoCCE4N4WRVK
(Start at 15:30 for the LLM specific parts)
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence) 2. Cow (85% confidence) 3. Bag (75% confidence) 4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
this sentence above would made a longer article if I bothered to so blog as is being blogged here
Because no, post training doesn't change that.
People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.