Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them (https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run variation (https://link.springer.com/article/10.1007/s10654-016-0149-3 misconception 22).

It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the task (benchmark or otherwise).

So as long as the thinking (albeit longer) is sound, this leads to the same success rate (as shown in the article) but potentially at the cost of more tokens and hence more time.

I think it'll be further useful to chart each quantization's used tokens as well, in addition to the success rate.

Thanks for doing and sharing the research!

I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there.

I also heard that KV cache quantization matters more with longer contexts. It may be interesting to benchmark this too: what the quality looks like on different combinations of model quantization × KV cache quantization × context size.

Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),

But on the other hand, the result that I'm assuming the author means to communicate ("on these evals, generation quality seems fairly good") sounds worthwhile to share?

Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?

To my own surprise, Q6_K_XL (from unsloth) comes up with a solution, anything Q5 doesn’t. To further surprise me, so far only the XL Q6 variant managed to solve it.

The problem, at least as stated, seems to be right on the edge of what the Q6 quantization can do.

Unfortunately even a successful run is rather long, so I don’t have a whole lot of data.

But the whole thing sure made me doubt the common idea that you wouldn’t perceive a difference until crossing past 4 bits quantization.

Err… can someone explain to me what is meant here? Surely the model wouldn’t consistently “guess wrong” compared to randomly, as that would be better. I guess some things like general coherency (i.e. is it even readable or gibberish) factor into that score?

But is there any model that actually works in a decent way at quantization of 1?

Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work that gets otherwise rejected.

       /data/llm/llama.cpp/build/bin/llama-server
        --threads 4
        --threads-batch 8
        --batch-size 4096
        --ubatch-size 256
        --port 9999
        --temp "1.0"
        --top-p "0.95"
        --top-k "20"
        --min-p "0.0"
        --presence-penalty "0.0"
        --reasoning auto
        --reasoning-preserve
        --reasoning-budget 4096
        --gpu-layers-draft all
        --spec-type draft-mtp,ngram-map-k4v,ngram-mod
        --spec-draft-n-max 3
        --spec-draft-p-min 0.75
        --spec-ngram-mod-n-match 24
        --spec-ngram-mod-n-min 4
        --spec-ngram-mod-n-max 16
        --spec-ngram-map-k4v-size-n 8
        --spec-ngram-map-k4v-size-m 16
        --spec-ngram-map-k4v-min-hits 1
        --n-gpu-layers all
        --ctx-size 131072
        --repeat-penalty 1.0
        --jinja
        --metrics
        --model /data/llm/models/unsloth/Qwen3.8-27B-UD-IQ3_S.gguf
        --chat-template-file /data/llm/models/qwen3.6-chat-template.jinja
        --fit off
        --flash-attn on
        --cors-origins localhost
        --mmproj /data/llm/models/unsloth/Qwen3.8/mmproj-BF16.gguf
        --no-mmproj-offload
        --parallel 1
        --kv-unified
        --cache-type-k q4_0
        --cache-type-v q4_0
        --cache-type-k-draft q4_0
        --cache-type-v-draft q4_0