Ways to think about token pricing

mips_avatar•about 2 hours ago

I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.

Analemma_•about 1 hour ago

You are waaaaaay too optimistic about RAM prices. This is not a one-quarter crisis and you need to settle in for the long haul: the era of "hardware always comes down in price" is over, definitely for the next 3 years and possibly the next 5-8. The new DRAM fabs aren't even coming online until 2028 and there's not much reason to think they'll make much of a dent in the supply-demand mismatch even when they do.

Legend2440•about 1 hour ago

DRAM prices have been highly cyclical for a long time. They'll come down again.

draygonia•about 1 hour ago

From what I'm reading on the shortage, they are very aware of this fact (prices being highly cyclical) and they're trying to avoid overproducing RAM to avoid the oversupply glut. That's probably going to hurt the chance of prices coming down anytime soon.

cyanydeez•about 2 hours ago

sure, if there werent capitalists digging their moat. in 1 year local inferwnce doubled cost from ram alone because cloud vendors cornered markets on their circular cash flow expectations.

mips_avatar•about 2 hours ago

Yeah but Samsung/Micron/SK hynix are building new fabs, and Chinese Ram will be able to fully supply the lower end ddr5 ram market within 2 years. The crunch is temporary and on the other side of it there will be incredible hardware at the $10-20k price point.

nekusar•about 3 hours ago

1. Slot machine

You put in minutes of time combined with token costs. You pull the lever and HOPE something good comes back.

And it gets tantalizingly close, but for 100% gotta pull the arm again!

shshsjsj•10 minutes ago

that is way too accurate and ofc hn would downvote something funny and true.

Ways to think about token pricing

⚡ Community Insights

Discussion (10 Comments)Read Original on HackerNews