FR version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
84% Positive
Analyzed from 4188 words in the discussion.
Trending Topics
#notation#attention#more#math#need#don#every#something#https#why
Discussion (123 Comments)Read Original on HackerNews
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
It's like saying "I never understood people who prefer to use functions instead of inlining everything". Adding a bunch of visual noise to a statement doesn't improve comprehension.
https://www.mathjax.org/
than like this?
The heiroglyphs let me offload some of the work to my vision system. It feels like there's some kind of hierarchical attention organization mechanism that works better in visual space. If I'm not pushing to "maximum math," both representations clear the hurdle of "good enough" (CIF does) but if I'm trying to chew through something genuinely gnarly I want all the help I can get.Remember that the oft-used e^x is actually an infinite series, even writing it out in summation form would be quite verbose given its frequency in many equations.
But, for someone just starting out (or even an expert who just wandered into a new area), this becomes a barrier to understanding. Sometimes we need the mathematical summary, what tells us the immediately answer we are interested in, perhaps even if it loses the deeper nuances that are beneficial for the experts more fluent in the language.
(And yes, all math notation is human created, so it makes my point quite a bit messier in practice.)
Trying to read any math paper is basically like trying to read CodeGolf.
So you and all your peers agree to call that procedure "ẽ"
Also, letters avoid having to name them, naming being a hard problem and all.
They explain one particular aspect of the notation but never define the variables used. What is k? q? S?
It's obvious if you've studied machine learning before, and for some of them you can make an educated guess, but it makes the article mostly opaque if you don't already have some domain-specific background knowledge.
Creating or combining to have something new, that does not already exist is actually freaking hard!
The moment its presented and people go "o, that is not that difficult", "i was able to also do that", or some nonsense like that. Everything looks simply the moment somebody did the hard work.
We have all been there was developers. Thinking we invented something new, and ... then you discover somebody already made it in the 70's and its everywhere. But because it never cross your path, you never realized it existed.
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
https://x.com/waterloo_intern/article/2081762065392541951
and in particular this image which clarifies the key essential difference between liner attention and delta network by examining the case of two tokens with same key but different value
https://pbs.twimg.com/media/HOPCc7BaEAAQDtO.jpg?format=jpg&n...
I think for comparison it would also have been good to have how original quadratic attention handles it: since both keys are identical, the attention would be "evenly divided" between both values so the final output would be the average of both values, as opposed to the latest value
(Side notes: I have physics PhD and mild dyslexia)
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner-friendly setting with hyper-strict control on the quality of media and source material they get access to, just like we've been doing it with frontier models. Think of it like a monastery but for kids, while teaching them all the latest advances in our understanding of reality through mathematics, engineering, computer science, deep learning, and whatnot.
What I'm getting at it is that we might still need super smart people to push the boundaries of knowledge while using super-advanced AI tools, and anyone who says AI will "completely replace" humans are just misguided. We will always need super smart people with largely unadulterated thinking.
I don't see how you get to that conclusion. We're just atoms. We evolved over a long time and something selected for a particular configuration of atoms that does "something" (see neurons firing in a network) that then somehow produces what you call "novel ideas".
But if one particular set of atoms can do it, so can another. I don't see why we can't eventually invent/evolve/train a bunch of computer atoms to come up with more novel ideas that we can.
"And softmax of course."
https://en.wikipedia.org/wiki/Bra-ket_notation
Now way I could have come up with Kimi Delta Attention.
I would have liked some refresher on some variables though (like d_k in quadratic attention).
Raising and lowering operators for summation notation are the beginner tools for covariant derivatives of the metric tensor.
Christoffel symbols are where it's at, if you need to write out the Ricci tensor. The more constrained the space the more concise the notation can be.
Note that MechE tensor notation has an even more compact (eigen) form for principal stresses.
It is what has separated me from being able to code just about anything on a GPU and being known for some of that work and coming up with a better way to run ab initio quantum chemistry on them.
It truly has been my Waterloo for many years. So make me wrong.
What? Little old me! Well, then, let's have a look...
> (First paragraph)
> A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣ q ⟩ ∣q⟩ is a column vector, ⟨ k ∣ ⟨k∣ is a row vector, ⟨ k ∣ q ⟩ ⟨k∣q⟩ is a number, and ∣ v ⟩ ⟨ k ∣ ∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.
Hmm... Guess not!
Huh?
If your aim is to truly 'get started' with ML then hardware is absolutely not a bottleneck (either local or cloud).
Remember that ML is much more than LLMs. Even modern day LLMs can be quantized to a point where they can run on local hardware although their capabilities won't be as impressive.
I would recommend looking into some of Andrej Karpathy's videos if you want a grasp of the basics.
Filippo Brunelleschi said he could build the large dome for the church that had stood unfinished for a century. Skeptical, other's demanded he'd explain how. He refused. Instead he challenged everyone to balance an egg on its tip. Nobody could do it. He then demonstrated by lightly tapping the egg on the table, flattening the tip, making it stand. "Anyone could've done that! You never said we could break the egg!". And that's the point. Anyone could've done it. But nobody did. Nobody thought 'outside the box'. And likewise, his solution to building the dome is as simple, and as ingenious.
It's called Egg of Columbus. (there's a similar story about Columbus that's more famous, but apparently fictitious). It teaches us that hindsight is 20/20.
The fact that invention is hard and actually you probably couldn't have invented a bunch of hard stuff is not really that, um, relevant.
https://xkcd.com/2501/
Is it really one big computation f(g(h(x)))?
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
You know, I haven’t kept up with MoE and etc where there’s a bit of selection going on, so I should probably be more humble. I think new work has only added different “paths down the same hill” though (no recurrence, just select different matrices), but could be wrong there. I don’t think I’m wrong on my general intuition, just want to be epistemically honest!
Is it all one big computation? Its turtles all the way down.
Original attention involves (very crudely) an approach of scanning how every token (roughly a word) relates every other token and training a classic neural network on related tokens - to get either language translation or next word prediction (and next word prediction is what "seems intelligent" in LLMs). [1]
The problem is that since original attention is "everything to everything else" it scales quadratically (O(n^2)) with the size of the train set (or train set window) and so basically even the largest data center can use that once a truly vast training set is accumulated. Which is to say that "dirty little secret" of LLMs following the "Attention Is All You Need" paper don't actually scale. That model (in my crude, amateur understanding) is elegant for allowing every word's connection to every other word to be weighed and still brute-force for not starting with or achieving "understanding" of the words [3 give only some background but also why "full" attention is powerful].
Linear attention is a way around the quadratic quality of original attention so everyone is naturally using clever approaches to make it work. Simplifying terribly - you're trying to determine the value of word before you see in context. But my intuition is that since (Everything X Everything) is inherently a quadratic relationship, none of these can capture their expanded data set in the way original LLMs did - not they are worse but all the models seem likely to hit diminishing returns in terms of blindly capturing meaning from all-the-world's text (and data).
Background and notes: [1] https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... [2] Linear Transformers Are Secretly Fast Weight Programmers: https://proceedings.mlr.press/v139/schlag21a/schlag21a.pdf [3] Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines: https://arxiv.org/pdf/2106.01506
I guess that's probably why you still need some MLA layers in there.
Yep! I know some of these words.
The rest of us reading this are like, holy batman, what the fuck was that?!
A piece like this which uses it in a headline but in no way makes an average reader feel like they could have come up with it is just badly misjudging how good of an explanation it is.
“You Could Have Invented Parser Combinators”
“You Could Have Invented Container Runtimes”
“You Could Have Invented Git”
Given the references to “mathematicians”, I think this reaction is more about an unfamiliarity with the concept of applied mathematics, which is ironic for practitioners in a field containing so much that is (or should be) regarded that way.
Software used to be all about “discrete math”, logic, maybe a bit of queuing theory…and now suddenly linear algebra and statistics became important. Don’t panic, it’s just another textbook on the shelf.
You could have your own hacker news, it's just a textbox, a bunch of tables and headings! Once you add these, it'll be abundantly clear that you also need a database. It should now be obvious that you also need a user system and it can be easily shown that needs a backend. Admin tools, tests, statistics, performance checks and so on can easily be derived from such backend.
I often find people get annoyed at mathy stuff because they seem to think that they should be able to read it like a (comparatively low information dense) newspaper article or something similar.
Math isn't like that, it usually has high information density and you need to parse every single symbol. And also people make this mistake where they gloss over stuff they don't get because they think they'll just understand things from context. Works great in normal literature - but math ain't like that. If you don't understand something, go back to the definitions.
Writing a detailed technical post behind the news of Kimi K3 and KDA algorithm with an audacious title like "You Could Have Invent Breakthrough It too" they are pre-filtering out the ones who couldn't comprehend with quick read (myself included) and attracting the ones who agreed with the blog post. At the end with a strong CTA to promoting their 10x cheapter open weight model AI inference and hiring too.
Good job Doubleword, I see what you are doing there.
We're just a group of guys and gals who like inference!
Nothing complex
"New algorithm, or fmadd?"
"... fmadd."
"You could have come up with Kimi Delta Attention, but you didn't, did you."
I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it.
It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.
What do you mean by this? Like concatenating all token embeddings into one large vector?
It's just one way to increase the expressiveness of the whole network. In general one could say that coming up with ways to greatly increase the expressiveness of neural nets is not hard. Or it's hard because it blows up compute. Meanwhile the human mind demonstrates that you can drop compute by 5-6 orders of magnitude without losing expressiveness to save power.