Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
86% Positive
Analyzed from 1770 words in the discussion.
Trending Topics
#model#models#run#qwen#open#context#https#bit#qat#quant
Discussion Sentiment
Analyzed from 1770 words in the discussion.
Trending Topics
Discussion (60 Comments)Read Original on HackerNews
[1] https://www.reddit.com/r/LocalLLaMA/comments/1vmi0fg/deepsee...
In terms of what you get for what you pay for, it's incredible - probably by far the best.
But unless I'm reading things wrong, it does not appear to be top-of-the-line.
At this price range $0.87per 1M they will get a lot of usage of people trying it out. Given the benchmark numbers, for many people and many use cases this will become their primary driver. There are people and use cases where Fable, Sol will work better but those are likely not the target of DeepSeek anyway.
In terms of performance and price pareto curve I don't think any model can beat this today (though openAI is doing some exciting recent work in efficiency) - which is a remarkable feat for the DeepSeek team.
Either way, what a time for consumers of these models :)
https://openrouter.ai/deepseek/deepseek-v4-pro-0813
License pretty similar to k3 with some caveats. Free to use for internal or <50M$ revenue / year. Limitations above that threshold for serving the model or services targeting coding / productivity agents.
Benchmarks are looking good, trading blows w/ opus4.8 and sol, generally 10-20p under fable. But that's neither here nor there w/ qwen, their benchmark to real world usage correlation has been iffy in the past.
The local model 3.8-27B announced for Friday, same time so ~48 hours from now. That'll be a bit more exciting for a lot more people, since 3.6 was quite good for local inference, and their 3.7-max -> 3.8-max shows a lot of improvement.
Maybe I’m misreading this or some other post, I thought QWEN was stepping away from releasing these models for local consumption
The 1bit quant model is at an astonishing 397GB with 95B active per MOE. This literally puts Opus 4.5 performance level into a machine a normal person could buy, and still gets usable tokens/second.
The full lossless model BF16 is clocking at 4.9TB. The model card claims the model to be between Opus 4.8 and Fable 5. Again that's astonishing as getting a machine with 7TB RAM (with context + KV cache) is still within the realm of medium size companies.
Bad things: The open source version has its vision capability removed, and the context capped at 250k . I expect someone to bolt a Kimi 2.6 vision tower to it to restore the vision capability (at less performance of course). For context, I played around with extending the context to 600k for Qwen 3.5 397b, and the context remained stable up to around 480k. It'd be interesting to see if the same can be done to Q3.8 .
Also no out of the box DSpark/DFlash support. MTP is present so we should at least get some boost in TP speed.
Honestly this model people at home can tinker with, if you have a big enough Mac. Maybe 4 Strix Halo/DGX Spark, and then at 1 bit quant? Nah.
Use the right sized model, for your hardware. You'll get better results.
Extremely large models don't suffer as much from quantization due to its weight topology also contains encoded information, so the loss of info from any one weight is somewhat mitigated.
I wouldn't pick up 400gb of hardware to run in that mode. I might try it for fun, but even then you are looking at handling a 95GB active parameter set.
This is NOT a model for most home labs. I'm sure some can and will use it. But most, should steer clear.
Suppose I have 100GB of unified memory, how should I know which model suits it best? I understand how a 2.4T model wouldn't fit, but I don't understand the impact of quantization and whether I should use a 200G model quantised to fit say 90GB of memory, or a non-quantised 90G model.
The old rule of thumb was that a lower quant of a larger model > higher quant of a smaller model. That being said, for some things going lower than fp8 will see a lot of degradation in generation quality. Except if the model comes with QAT 4bit quants. Then there's also nvfp4 w/ calibration data, which also can improve things. So it's really not easy to tell "at a glance" you'd have to test them yourself on your hardware.
Anything below that, and especially 1.58b - is typically complete garbage, and you're much better off running a model 100x smaller at regular precision (compared to one 7x smaller quantized into complete garbage).
If the model was designed specifically to quantize down to 1.58b, then it's different.
AFAIK, there's no large models designed for this yet.
at this kind of quantization is it useful though?
That is unfortunate, that the open weight model doesn't have vision support or the 1M context length...
[1] https://old.reddit.com/r/LocalLLaMA/comments/1vl6ior/i_gave_...
[2] https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4
for example, they already have qwen3.8-max
https://openrouter.ai/discover?model=qwen/qwen3.8-max
note that they add some fee ontop of things (maybe 10% of spend?). it isn't htat big of a deal for general experimentation, but if you end up wanting to use a single model in a higher-volume way, it likely makes sense to cut them out of your stack.
On your 5090 you could easily run a smaller model like Qwen 3.6 27B: https://huggingface.co/collections/Qwen/qwen36 or Gemma 4 etc., or as mentioned there's a Qwen 3.8 27B coming out in a few days.
Apparently the ~30B variant will be released on Friday?
Read the room, Qwen. It's not a good time to hobble your releases.