Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

CyLith•12 minutes ago

I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.

zamadatix•9 minutes ago

That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.

ianmurrays•8 minutes ago

I guess you have to know which experts to keep “hot” in ram, which you can’t know beforehand, so there wouldn’t be much gain.

dghlsakjg•about 1 hour ago

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.

People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.

Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

arjie•38 minutes ago

Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.

hedora•31 minutes ago

AMD already demonstrated 1T on strix halo clusters. << $10K at original MSRP.

AHASIC•about 1 hour ago

I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.

Mistletoe•39 minutes ago

What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.

CircuitSeuss•2 minutes ago

A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.

Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.

bobbylarrybobby•17 minutes ago

The models themselves have far from plateaued. Maybe someone finds a way to get a really capable model down to, say, 12GB of ram. Then we'd be in business.

sudo_cowsay•24 minutes ago

It could be on software side too. OpenAI has certainly not plateaued.

jbird99•about 1 hour ago

At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.

kennywinker•about 1 hour ago

It says very prominently in the post: 4.5-5t/s for 80b on an M5

wat10000•about 1 hour ago

Isn’t it only writes that kill drives?

sudo_cowsay•23 minutes ago

Yeah, that's why most of these comments seem weird to me.

Alpha3031•27 minutes ago

Yes for NAND, and I suppose nobody is using mechanical hard drives for this.

brrrrrm•about 1 hour ago

this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.

half an hour to process 10k tokens on an M5 seems... not great

kennywinker•about 1 hour ago

Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

⚡ Community Insights

Discussion (15 Comments)Read Original on HackerNews