ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
69% Positive
Analyzed from 2414 words in the discussion.
Trending Topics
#kimi#model#attention#open#weights#training#layers#distillation#source#architecture
Discussion (98 Comments)Read Original on HackerNews
(As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the data anyway (because of randomness!))
They could give you the random seeds? (Assuming you carefully train in such a way to remove other sources of randomness, like concurrent execution.)
> You can't rebuild it from scratch
There's is extremely clear and misunderstanding-free.
Open weights is not open source.
> It's entirely reproducible from the available documentation
You have a very interesting understanding of "reproducibility", I'll give you that :)
But even with that, there are plenty of technical details (especially in regards to the training process) missing from the tech report that leads to these weights not being reproducible in any sense of that word.
He was talking about two different things, hence the parentheses. The architecture is reproducible, not the model weights.
Reproducing the exact training run, however, is basically impossible without the original dataset and training pipeline (here meaning all of the code + infra involved in actually executing the pre and post training loops). Also, it would be exorbitantly expensive to do if you weren't also a lab trying to train a similar model.
But you can still scale the architecture down and experiment as a solo researcher using the published research. There are probably some open source implementations already on GitHub for any given big open model release.
I share this expectation. But this is an interesting empiric question that deserves study; even if just to confirm what 'everyone knows'.
Transformers are very "mendable" in that you can permute the architecture in crazy or random ways, and still basically always end up with get a coherent LLM. The difference comes down to training efficiency, inference efficiency, and usually minor differences in performance.
Hyperparams and stuff, I mean it's standard to do a sweep anyway.
The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss.
Interesting what other labs comes up with from this.
My only doubts are around Linear Attention instead of DSA as this is inherently lossy. You are kind of banking on that your query is inherently in the embedding space of the model already and can be lossy.
Source: my bank balance
Something is wrong at moonshot.
It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?
"Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."
And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.
Temporarily (maybe permanently) using it as my daily driver but it's wild how comparable it is to Opus 4.7/4.8 (what's been my go to for a bit now—wrote a quick post on what I found today [1]).
[1] https://graybearding.bearblog.dev/kimi-k3-is-insane/
Plus I can actually see the thinking traces and I’m not getting downgraded/denied by anthropic constsntly…
that's a weird way of describing a near frontier open weights un-crippled useful coding buddy
do you work for OpenAI or Anthropic per chance?
Like the distillation “attacks” Anthropic conducted against literally millions and millions of different creators?
There's been some distillation.
Just like how Elon said in court they distill to make Grok.
Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.
Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.
RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.
>Curious to see if it holds up at frontier scale.
I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
> Seems to benchmark
yes, but in human usage the differences show up
SWA has a similar issue. Unless you keep the entire KV prefix lying around (which is not unreasonable: you retain flop + bandwidth benefits but lose capacity benefits), you need to start 1 window back from the rollback point, in order to refill the sliding window before going into normal prefill.