Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU

ppich about 4 hours ago 33 commentsRead Article on piszczek.pl

FR version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

63% Positive

Analyzed from 908 words in the discussion.

Discussion (33 Comments)Read Original on HackerNews

simonw•about 2 hours ago

"Combining them into one heroic speedup would make a better headline and a worse benchmark."

"The machine immediately taught me that capacity estimates are just admission tickets."

"Useful in production, poison in a kernel comparison."

Please don't publish writing like this, it's exhausting to read. You can edit that stuff out.

The best part of this is the "Five xhigh artifacts from the finished model" section at the bottom, I suggest either moving that up or at least prominently promoting it at the top of the article.

PeterStuer•about 2 hours ago

It has gotten so much worse over the last month. The default writing style of the Claude 5 model series in Claude Code is some sort of jiberish jargon.

dannyw•about 2 hours ago

I find the ‘explanatory’ output style of Claude to be a bit more tolerable, but yes.

Claude seems to speak and write more in Claude-speak with every release.

cedws•about 2 hours ago

Apparently we've blown way past the Turing test and approaching AGI and yet LLM-generated text still sticks out like a sore thumb. Maybe LLMs aren't that good at writing after all.

Groxx•about 2 hours ago

I think it's fair to say they're better at a paragraph or so than most humans. And have been for quite some time, which is probably why their use in writing has exploded.

Long form though? Still pretty bad. Probably getting worse in practice, as people have them write larger and larger chunks of text without paying any more attention to the result.

wgd•about 1 hour ago

LLMs are great at writing, it's The Assistant who is a terrible writer. Sadly that one persona is all you get these days.

mhaberl•about 2 hours ago

lot of tech folks think their writing is boring and ask AI to make it more interesting or whatever

llm gets them some of that sweet_llm_padding and readers smell the slop and attack the writing :)

people, go write "boring stuff" - its more interesting!

supermatt•about 3 hours ago

Can you please try and see how many tokens you get with some form of concurrency. Pretty much ALL the benchmarks I've seen on the more accessible cards are just single request.

jermaustin1•about 2 hours ago

Because concurrency with a single "accessible" card quickly diminishes. I have dual 3090s, and on Qwen 3.6 35B A3B at 80k a single card with max concurrent set to 4 will get 70-80tps single request, 50-60 TOTAL tps with 2, 45-50 with 3, and around 40tps with all 4 going.

I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.

petu•about 1 hour ago

I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other.

As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.

With dense Qwen 27B on 3090/llama.cpp I get:

  - no MTP: 1x42, 2x33, 3x24, 4x19 t/s  
  - MTP: 1x50, 2x30, 3x33, 4x30 t/s

jermaustin1•23 minutes ago

That is interesting. I'll have to test that theory out today.

supermatt•about 2 hours ago

I haven’t tried any larger models, but a 12B model on my Ampere A5000 gets around 4-5x the aggregate throughput with concurrency. I have the maximum context configured to 32k, but my actual requests are usually around 2-4k tokens.

No idea how that compares to running a larger model and context though.

dannyw•about 2 hours ago

Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp?

I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.

mhitza•about 1 hour ago

What do you use instead of llama.cpp? With vllm for example most models don't seem to be supported out of the box.

Tostino•about 2 hours ago

You have something misconfigured then. Concurrency has never lowered my overall TPS. Also have dual 3090s. Generally use vllm though.

jermaustin1•about 2 hours ago

I've had some rough time getting LM-Studio properly configured for multi-card. It exists, but I feel like it is kind of buggy. I will disable a card and it will still load the model into it. Sometimes it will split the model even though there is loads of room available.

I might need to finally make the switch away from it, but it is so convenient, especially as a chat interface for system prompt experimentation.

pich•about 2 hours ago

vLLM is probably the key difference there… its scheduler is built around batching/concurrency, while this setup is heavily optimized llama.cpp for single-stream latency

Tepix•about 2 hours ago

Always put the quantisation in the title!

pich•about 2 hours ago

Its not quite that simple here. The iMatrix-guided hybrid uses different quantization levels per tensor/layer, so there isnt one honest Q4/Q5/NVFP4 label I can put in the title

nodja•about 3 hours ago

The whole site looks like and reads like AI slop. The outcomes also don't make any sense and don't feel rigorously tested (no, having claude test for you doesn't count as rigorous).

genxy•about 2 hours ago

The person is having a AI induced manic episode, we have all been there.

IncreasePosts•about 2 hours ago

Please stop making this comment. The war is lost. Instead, you should be commenting that it looks like a human wrote this when you come across the rare brain-produced writing

nubg•about 3 hours ago

quantization level?

MaxikCZ•about 3 hours ago

Its egregious the quant level isnt disclosed along the "Qwen" string. Everyone knows theres huge difference in speed/quality along the quant axis, I now attribute the ommision of such to deliberate choice to not curb the hype of the tittle.

pich•about 2 hours ago

5.01 BPW custom hybrid: bulk NVFP4, selected Q5_K/Q6_K tensors from an iMatrix, Q6_K embeddings and Q8_0 lm_head. The iMatrix was built from 5,472 messages across 296 real Hermes sessions

ramon156•about 2 hours ago

Glad to read theyre not 296 fake Hermes sessions /s

tingletech•about 3 hours ago

they talk about the quants they tried in the article and settle on a Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf which they calibrated on their own session traces and they pulled in 5 different llama.cpp pull requests to their local llama-server.

metadat•about 3 hours ago

The article mentions Q4, Q5, Q8, and NVFP4. It's total AI slop though, tough read.

In my testing I got 150 tokens/sec with a single 5090 RTX.

sleepyeldrazi•about 2 hours ago

yeah, I was confused during the whole thing, i get 70 t/s on a 3090, which evens out around 50 t/s at 128k+ , have been running 3.6 and now 3.8 (both iq4_nl at 256k q4 kv) on the 3090 for months. I am confused as to what we 'discovered' here, it's a common config. and at less than 1/2 the price of the gpu (and double the bandwidth, though no fp4 cores to be fair).

pich•about 2 hours ago

A 3090 has 936 GB/s of bandwidth vs 432 GB/s on the 70W RTX PRO 4000 SFF, so 70 t/s there is not surprising. The interesting constraint here was fitting a workload-tuned 5.01 BPW quant + 256K + MTP into 24 GB while working with less than half the memory bandwidth

Foobar8568•about 2 hours ago

Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.

iv42•about 2 hours ago

If you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).