ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
62% Positive
Analyzed from 1289 words in the discussion.
Trending Topics
#ollama#llama#running#model#cpp#using#lot#better#qwen#pro
Discussion (41 Comments)Read Original on HackerNews
I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.
But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.
It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?
I have an m5 max 64gb and I think anything slower would be quite painful.
Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this.
Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed.
- find a free GPU droplet on digital ocean
- fire it up
- pull in a snapshot of the model + extra files/packages etc
- set up a ssh tunnel so that the localhost:8000 routes to the above
Then I just configured OpenCode to use the above and was off to the races.
Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)
Using oh-my-pi in a prebuilt environment that I let Qwen build too.
Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.
Genuine question : is there something fundamentally wrong with Ollama ?
I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).
I thought the main advantage of VLLM was better concurrency management (better batching).
But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.
For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.
And it did in that case make a significant difference.
Yes https://news.ycombinator.com/item?id=47788385
I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.
Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).
I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.
This is the version I want to read :)
I assume it is unpleasant in spite of the math, not because of it?
Using opencode and it built a old fashioned arcade vertical shooter with no issues.
Images are ok'ish, just had grok create updated images, and it came out great.