Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
68% Positive
Analyzed from 2276 words in the discussion.
Trending Topics
#run#model#models#same#tok#don#speed#mac#hardware#tokens
Discussion Sentiment
Analyzed from 2276 words in the discussion.
Trending Topics
Discussion (50 Comments)Read Original on HackerNews
Curious if other folks are also surprised by this apparent discrepancy or have a ready explanation.
I can't find the post though.
Edit: a quick search from a server refurbisher nearby gives me a dual Xeon Gold 6330 28-core machine with 512GB and a 16GB V100 GPU. The memory is spread across 32 slots, for about ÂŁ6,470.
Jealous of the folks with 5090s running ninfer and getting >100tok/s. At those speeds it's a true frontier replacement IMO.
The author has an M3.
Here's reality, MLX on the software layer will not magically place hardware matrix multiplication units in your GPU cores.
Newer Macs are always just gonna smoke anything earlier than an M5.
Not even sure why this person is trying to get this stuff to run on hardware that wasn't designed for AI?
Everyone is pointing him to newer hardware precisely because you need the newer stuff to get models to be performant. You can go with AMD, NVidia or Apple, but you're gonna be using stuff designed well after the M3 if you want to push >100tok/s.
Posturing/overclaiming like this shades rather than illuminates, there are no worlds in which the "M3...wasn't designed for AI". My M4 Max 64 GB gets the same speed.
Current prices in USD:
128 GB 1 TB M5 Mac Studio: $5399
128 GB 1 TB GMKtec EVO-X2: $3499
128 GB 1 TB Framework Desktop: $3748
https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B
I'll stick with my old AMD datacenter cards thanks. An Instinct MI50 32GB would cost around $550 or so. It has 1TB/s vram bandwidth thanks to its HBM2.
Tensor parallel in llama.cpp using RCCL (disabled by default in llama.cpp for some reason). Surprisingly, for these cards HIP is actually faster than Vulkan, unlike the 9070 XT where Vulkan still wins.
ROCm nightlies do actually support these old cards, just not the ROCm stable releases.
The large RAM Macs are unusable for inference of dense models as of now. Token generation is too slow.
What you get with local LLM options: 1. Download the inference engine app from the web; 2. Download the model; 3. Configure MTP / DFlash / DSpark whatever; 4. Configure your Pi / OpenCode harness to point to this local LLM inference engine. 5. Configure tools for these harness to be effective. 6. Switching between Ollama, LM Studio, llama.cpp (DwarfStar4), oMLX, MTPLX, to see which one is fastest for your workload. 7. Again switching between different quants of the same model to see which one is less dumb.
To be honest, llama.cpp probably the closest to deliver on "just use it, don't worry about speed" if your focus is about a pure LLM inference engine.
Am I getting these results because I picked the wrong model? Or I need to improve my prompt? Or the tool just can’t do what I’m trying to do? How current are these model recommendations? Have they been superseded by something newer?
As a true beginner to AI at the time, even the sizes and bits were meaningless to me. And I don’t remember having any context as to what I should be attempting to run on my mac.
So I think you need to include steps 4, 5, and 6 of swapping between different models, quants, and prompts. And step 7 is probably wading through the complicated UI, full of jargon that most people don’t know.
Don’t get me wrong, I recognize that it’s a powerful tool, and the steep learning curve exists because it exposes quite a few power-user features. But for someone graduating from commercial AI image generators that take a text prompt and maybe a choice of couple models, it’s not easy.
I would like to see "guarantee" for that app. You get full refound and it is free, right?
What would be incredible is the 3.8 35B MoE version too, I can run 3.6 with 60 tok/s which is a really, really nice speed.
You will get better information cruising r/localllama for about 10 minutes.
Crawling through Reddit or forums to find the right incantation to run a model is frustrating.
Why is anyone trying to run these on an M3?
I thought it was common knowledge that, if you insist on using Apple, only M5 processors and higher have matrix multiplication units in the cores?
I've seen the same thing with people buying NVidia cards with 16GB of ram and wondering why they aren't getting 100tok/s?
Guys, please, be reasonable. You'll have to get the hardware if you want to run these things fast. If you want to experiment, the slow stuff is fine. But try not to get on HN and ask why your M3 can't get 100tok/s. You're kind of out-ing yourself.
Even if Google, OpenAI, Anthropic whatever promise not to use it, how sure can I be of that? Data is gold anyway. And we're in the middle of a massive gold rush. And they've already been caught scraping sites they had no business to, and pirating books. Clearly their promises and the law mean nothing to them. It's just something you pay off in a settlement if you get caught, a cost of doing business.
With local models besides the speed you also lose a lot of inference quality but it helps to mitigate that. For example making sure your RAG inputs are properly prepared and categorised so the model can find them easily without having to wade through a bunch of misdirected crap.
For example what I do with my bookmarks and chats (the latter are recorded per day), is before I enter them into a RAG corpus I run a small LLM over it to summarise what's being discussed or what the webpage is about. That really helped retrieval quality, and doing this is a batch task that can run asynchronously so speed is not very relevant. This way I get a lot closer to SOTA-model retrieval quality (like with Office Copilot 365 looking for conversations in Teams).
PS: I wouldn't be surprised if Microsoft runs something similar on their end :)
But yes the Mac Studio is outrageously priced especially for running such a small model as qwen 27b. For that price you can use something much much cheaper. It only shines for models that are much bigger, because there simply is not much hardware that can address fast 512GB banks.
Which is a regulation constraint on many professions, BTW. Some people simply can't give some data to say, ChatGPT, without comprehensive guarantees written in Sam Altman's blood.
There need to be second order consequences to those who delegate their responsibility to companies that behave like this. They know the contract is worthless, but then proceed to use it as defense for their own gross negligence.
The new M5 Ultra should deliver ~50% faster token generation (1.2TB/s mem bandwidth); and extrapolating from my M5 Max (since the M5 Ultra is literally just 2x Maxes), probably ~3x faster PP.
But I don't think it's fair to look at this only from monetary ROI vs API. With local models, you get privacy and ownership.
I do not trust _any_ API provider with my most personal information; such as for example, all my messages, emails, daily journals spanning a decade+, all my photos and videos, etc. So it unlocks new use cases that I simply don't feel comfortable with via API.
And a personal assistant with ALL my context and data, locally, has been incredibly useful for me :) Zero outages either, zero "overloaded", etc. Nearly-zero refusals too (I don't run abliterated models; thinking prefill has worked for anything I've wanted to do)
I realize the price of NVIDIA has gone up but there are plenty of GPU options from others like AMD and Intel with reasonable performance.
>~14 tokens/s
For anyone reading that has never ran local llms, please understand that anything under 100 tok/sec is worthless. You are faster typing stuff into Gemini free version that you get with a google account and copy/pasting it in (and you can easily build browser automation with playwright or any other js runtime to have this available in a chat window)
https://openrouter.ai/anthropic/claude-opus-5 is it worthless because its 65 tps?
re: gemini
https://openrouter.ai/google/gemini-3.7-flash worthless as well?
That being said, 14 tok/s is pretty slow.
I agree that 14t/s is pretty tedious for interactive use, yes. But 50-60tk/s is faster than I can read. 100tk/s is outright fast. Don't forget there is a limit entering content into meatspace.
Also, Gemini may be free but what if I don't want to give all my data to Google? This is precisely why I have a lot of stuff locally.
And will it remain free? How are they going to make back all those trillions of investment?
But yeah I would kinda balk at 14tk/s too that's why I use old datacenter/workstation-class GPUs.
Still, as you said - for day to day, 50-60 tokens / s is a good baseline.