DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
72% Positive
Analyzed from 3457 words in the discussion.
Trending Topics
#more#water#tokens#nvidia#chip#same#chips#cheaper#models#produce
Discussion (105 Comments)Read Original on HackerNews
For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough.
While 2 years ago nothing was useful more than 1 year long, there are many older models in use now (e.g. Haiku 4.5, GPT-OSS 120b), and I expect this trend to continue.
I know this is what Taalas was doing (acquired by AMD), here was their demo, https://chatjimmy.ai/ which is based on Llama 3.1 8B. It feels like this should start to happen soon.
Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a month, start to finish, for the physical processing.
Maybe once LLM improvements asymptote further?
The bigger issue seems to be that these chips can’t hold that many weights at the moment.
(I’m curious if chips with large weights in them would be more tolerant or less to yield issues. If you flip a few bits in the weights, does it really matter at scale?)
Yeah, those guys aren't biased at all.
I mean, previously you could have said something much the same except substitute "frat boys".
I don’t see how tokens can’t produce speech or track metabolic needs. You can talk to chatgpt can’t you? Or do you mean literally talking? Because that’s not a brain function, that’s the mouth, vocal chords, and lungs.
It probably could, but the point is this would require additional tokens, blowing up the comparison. The token output of LLMs and "token output" of speech are simply at different abstraction levels. Hence my comparison to the LLM brain driving the robot harness to produce speech etc. This would be more comparable, and also look significantly worse than "only" the 22x less efficient number.
In short, better hardware will drive down token cost in the near-term, but will drive up the demand for tokens as it gets cheap enough for other sectors to start to use it heavily.
It comes from steam engines where economists originally thought that coal demand would plummet with more efficient engines, but it actually just meant that we found more uses for steam engines.
If you make the thing more accessible, more people are going to use it. If it consumes a resource, the use of that resource will increase in relation to the increased adoption.
Hydrogen engines use hydrogen. Making hydrogen engines cheaper will increase adoption. Increased adoption will increase consumption of hydrogen.
Like, who'd have ever thought "oh wow, we've gotten to the point that people can have a computer in their own home, surely electricity use will plummet." or "oh wow, more than 50% of the population can now feasibly purchase an internal combustion engine, surely fuel demand will plummet."
In the original context they'd decreased the cost and complexity of steam engines. Anyone who'd seen the amount of money people were making with the old steam engines would be clearly incentivized now that they have the same economic opportunity available for less capital up front. Therefore more steam engines, therefore more fuel demand. Who in their right mind would really be surprised that resource consumption went up when people could and did build more machines?
And you can say of course, it's so obvious, how could a dumdum not see that! But then there are lots of examples of things where increased efficiency results in less usage overall, because demand is inelastic, etc. Jevon's paradox doesn't apply to everything.
I don't think we know yet what is going to happen as software development gets much cheaper. If in ten years we can produce software 1000x more cost effectively, will we need fewer software engineers, the same, or more? Guess we'll see!
Codex keeps doing these usage resets. What do I do? Burn even more tokens than ever before. I know I'm not the only one.
This doesn't mean the concept is BS, but one single concept cannot explain away everything in such a system.
Token prices plummeting is probably a good thing, but not without the regulatory backstops that prevent these effectively industrial facilities from being operated with no regard for the externalities they impose on people who live near them.
If we can’t even build data centers, the least disruptive industrial use possible, there’s no hope to reindustrialize the US or anywhere outside of China.
how much pollution do you believe modern gas-turbine engines to produce?
>Not to mention the water use controversy.
what percentage of US water usage do you believe is by AI data centers?
you really don't get it?
So as much as i agree “more profits to stakeholders screw the customer”, i think its more of an emergency to get to profitability before the music stops.
what makes you think this?
If the chips weren't this compelling they would have something different to announce.
These are paperclip maximizers who just happen to wear human skin - there is no underlying premise nor ideological goal.
One thing not on that comparison table: die size. If I'm understanding that correctly, it's about the same as the Rubin, but at 1/3 the number of NVFP4 PFLOPs. (The text disagrees with the table, I'm taking the table as truth, perhaps that's wrong...)
Maybe the money will still flow into this industry after all
I remember when Tesla was building its own inference chips, and after about 2 years and billions spent, the whole effort was scuttled b/c they simply could not keep up with the iteration and R&D cycles of dedicated chip companies. I suspect the same will be the case with OpenAI vs Cerebras + Nvidia/Groq
Even in nvidia land rubin + LPU does a similar thing.
It has its downsides of course - if your traffic swings prefill heavy to decode heavy, you can't suddenly use your lpu for prefill. With GPUs they're totally interchangeable. Tradeoffs.
Hes got a press release.
The issue is, baking something to silicon requires discipline and about 2 years.
This isn't something you can just change your mind on halfway through. Trust me, I know. You need a clear vision of what you want to support, why and what bits of a chip you need to achieve that.
That sounds quite like...nonsense?
Chip companies work on years-long cycles. They know today what are they launching 4-5 years from now.
Or at least Nvidia GPUs will become slightly cheaper for regular consumers again
There is probably a ~50% chance that the next Dem candidate for presidency runs on a national datacenter moratorium or something equally as crippling.
Once models are more or less interchangeable, the price of LLMs will drop to essentially the price of energy required to run them, and the big labs will be able to run them cheaper than anyone else.
With competition we will actually have the fair split, whatever that is, and thus much lower prices.
At the moment, to have a big AI firm, or really AI firm at all, you need to be blessed by NVIDIA, in the form of receiving circular financing for your compute. They know that their prices aren't fair, or competitive.
Commoditization of inference is the end of that. The end of the mega-premium on inference hardware, and it's good not only for people who like running their LLMs, but it's the first step towards commoditization of training.
If what you said were true, you would hardly see people complaining about the quality of Opus 5 or good writing from Sol. But people do.
They are all converging.
One objective of the project might be simply to provide credible negotiating leverage when dealing with existing suppliers like NVidia. You don't have to deploy at scale for that to work, but you do have to look like you could if pushed hard enough.
But then that means you have no actual moat against the behemot, right? Your competitor can move into the market as soon as they want to, at much better cost (so at slightly better price)... and Nvidia certainly can adapt much faster around hard hardware specs innovation than a new entrant ever could.
The other side of the conversation is the idea that large models in DCs on custom silicon is the future. Maybe for enterprise? But consumers will eventually (10 yrs) have affordable hardware designed to run crazy-good local models (more RAM + higher bandwidth). That will take pressure off of datacenters, but also reduce AI profits, and move that money to consumer chip/device makers. Apple is once again the biggest winner. Nvidia consumer chips might get cheaper, but nerfed, to encourage datacenter use where they make more money. I'm hoping AMD can stop being terrible at software so that when we finally have their better hardware we can actually use it.
They should GTFO with this crap.
Create barriers to computing for ordinary people while milking businesses for tokens.
There's a reason why Micron and Nvidia are the crown jewels of American technology right now and for the foreseeable future.
To give some context, Intel started making DRAM, I think they were actually the company that came up with modern memory techniques. They exited the market and pursued a more lucrative moat with CPUs.
If somehow you get around the patent issues, you're now faced with huge research and development costs, fabs to build, processes to sort out and all of that has very high failure rates.
Last time I checked Micron was the largest patent holder in the world and even for them this is a hard area where they are number 3 in the market.
If the rich decided to buy all drinking water, you would probably be saying that's okay, making water is difficult, shortly before dying.