RU version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
75% Positive
Analyzed from 1800 words in the discussion.
Trending Topics
#models#more#model#compute#gpus#nvidia#moonshot#having#gpu#huawei
Discussion (40 Comments)Read Original on HackerNews
I guess at least partly a reflection of all the optimizations in the Kimi 3 architecture.
In the recent leaked DeepSeek investor meeting, they also mentioned only having a 20K GPU cluster (unclear if NVIDIA, or Huawei).
The goal will be to develop smaller models with more efficient architectures, that have similar or even better performance than larger models.
Continual learning tends to imply individualized models, else there is no data privacy (the secrets learned on the job at your company now being available to your competition), which really turns the current AI business model of a single centralized model served to everyone on it's head. If every customer has a different model that essentially means the end of batch processing with the same weights loaded into the GPU.
The direction this suggests is a move away from centrally served common models to locally served individual ones, which generally requires them to be smaller, even if some larger companies may be willing to invest in beefier hardware.
I think this is at least in part why the AI companies are trying NOT to implement true continual learning and see if they can instead finesse it continual compacted(?) memorization instead, since then it's "just" additional context that needs to be recalled and fed into every request, not weights that need feeding into the GPU. I don't think memorization is any substitute for learning, especially learning of practiced skills, but since it's far easier to implement, and non-disruptive to the cloud-based API business model, this is what we will see first.
The recent news of NVIDIA' investment in Sutskever's SSI has a tiny hint of this also, talking about SSI advising on NVIDIA's future architectural direction - apparently pushing it in a different direction than current (cloud-based, pre-trained) models. NVIDIA may be quite happy to see a move towards local models.
The frontier labs would be well served in carving out 20k sub-clusters and giving research teams carte blanche in building things with radically different architectures - with full permission to distill whatever they want from the flagship models. We'd expect to see more product lines that feel "different" from the flagship models if this were already being done.
I am adding multi-modality to https://github.com/guilt/TinyToT, and I see that dis-aggregating capabilities, very similar to how our own sensory organs work, seems to be paying off quite well.
A VAX 11/780 was good, but an 80386 was a lot better, since the latter could run on 3 AA batteries and the former needed 6,000 watts of 3 phase.
> A person familiar with Moonshot’s procurement strategy confirmed that the company does indeed have a channel for accessing Blackwell processors via Southeast Asia. They didn’t specify whether this was a rental channel — which is legal, in most cases — or direct purchases, which are a breach of US regulations. The Information reported this week that Moonshot is seeking additional Blackwell processors to train its next model.
> The 20,000 chips Moonshot accesses via Alibaba, meanwhile, are from Nvidia’s earlier generation of Hopper products, the people familiar with the agreement said.
So the 20k GPUs from Alibaba is only a lower bound on how many you need to train a model like Kimi K3.
The advantage of having more GPUs in any case is not so much that you can train bigger models, but that the turnaround time is faster, so you can run more experiments to dial in training choices. It's entirely possible that Musk has more than enough compute, but can't hire the talent to run all those experiments. (That would also explain why he has excess capacity he can rent to Google.)
A few quotes from the transcript:
> Our current computing capacity is approximately 20,000 H-equivalent units, most of which have just arrived within the past month or two
> Regarding the Huawei 950, Huawei currently provides us with 16,000 SIM cards
> A Huawei 950 [cluster] with 16,000 cards is equivalent to only a B-series card [cluster] with 4,000 cards.
the only way is down for the massive valuations and 'a.i' revenue projections.
Its all a fucking capitalistic farce to display to other rich elite that "Look at how much clout I have! I can make these peons dance around and do my bidding! Im a slave-owner!"
https://infosec.exchange/@david_chisnall/116991627711001827
He noted that that Silicon Valley doesnt really want to SOLVE problems. They want to find already-solved problems with problem matching. And of course, we just throw more people and more compute instead of optimization and understanding.
The Chinese are being actively constrained with bullshit politics around a second Red Scare moment. And, well, they're winning. A lot.
Are you sure they are using all of their compute on training? Didn't they rent out a ton to other AI companies?
Moreover, it’s very plausible (and expected) to use multiple clusters and GPU types for RL rollouts which could very well not be included in this count.
No part of this pipeline is fixed in stone.
Source?
Someone will find a way to make cheaper compute, and since nothing fundamental changed there (LLM didn't change how silicon were made), that's bound to happen.
If Alibaba can't bring the chips into China, but can buy a whole bunch in Thailand or Singapore or whereever (or secure exclusive rights via JV partners where relevant) and then just provide them as a service to its customers in China - what is the point? I'm sure many customers would actually prefer such an arrangement.
I wonder how much spending is motivated by the phenomenon of sudden emergent performance in LLMs. Clearly some people who are smarter than me expect something like emergent AGI, or at least they think the odds justify spending whatever it takes to see if that would happen.
That leaves a lot of room for efficient aggressive followers.
Either they have Blackwell with native 4-bit floating math, or they use have Chinese domestic NPU that support mxfp4 natively.
The article’s statement does not make sense.
Personally, I think 20K nvidias is a stop gap solution because they really don't have the capacity to serve their models to earn any money right now.
From the article:
A person familiar with Moonshot’s procurement strategy confirmed that the company does indeed have a channel for accessing Blackwell processors via Southeast Asia. They didn’t specify whether this was a rental channel...
Probably either the author or the people they interviewed got the specific GPU details mixed up or wrong. Maybe it was B200s.
https://archive.is/AvWWX