Qwen3.8 27B scores 52 on Artificial Analysis

184

aanana_ about 3 hours ago 94 commentsRead Article on artificialanalysis.ai

⚡ Community Insights

Discussion Sentiment

94% Positive

Analyzed from 2839 words in the discussion.

Discussion (94 Comments)Read Original on HackerNews

beltsazar•about 2 hours ago

As a comparison, Qwen3.6 27B scores 38, which was the highest in its small model category (4B–40B).

Qwen3.8 27B beats all medium models (40B–150B). It has the same score as DeepSeek V4 Flash 0731, which ranks #5 in large model category (> 150B).

Sources:

- https://artificialanalysis.ai/models/open-source/small

- https://artificialanalysis.ai/models/open-source/medium

- https://artificialanalysis.ai/models/open-source/large

phsource•about 2 hours ago

Simon Willison's post about this gives a good context on why exactly this is happening. While it doesn't mention this in the Artificial Analysis page, this is likely with Max reasoning, which has extremely long reasoning traces:

https://simonwillison.net/2026/Aug/16/qwen-38-27b/

It seems like the token usage is 2.3x GPT Luna Max and almost 2x Kimi K3!

https://imgur.com/a/dDSyhr2

I'm curious if they can make up for this with insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)

kees99•about 1 hour ago

Qwen models are slower in tokens/s, compared to similarly sized gemma4 and others, and they use more tokens per task, in part thanks to that xhigh default.

On the other hand, there are some of us who are stuck with hardware that has plenty of compute, but limited (V)RAM. The new 27B is just perfect for that.

petu•about 1 hour ago

> Qwen models are slower in tokens/s, compared to similarly sized gemma4 and others

No? Gemma 31B and Qwen 27B are about the same speed. Gemma 26B-A4B and Qwen 35B-A3B are about the same speed.

skohan•about 2 hours ago

I'm running 3.8 27B locally, and the results from the past few days have been excellent. I find raw speed is less of an issue when you can trust the model more to reach the right result.

stymaar•about 1 hour ago

> insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)

It's a dense model so it will use all of its parameters per token. 37B active parameters isn't tiny at all, it's almost what Deepseek R1 had, and it's 2/3 of what Kimi k3 uses, so it's not going to be “insanely high” tps: it's going to be three times slower than Deepseek Flash (Prefil speed is going to be quite high though, but not token generation).

ArvidSu•about 2 hours ago

A ThinkingCap variant of Qwen 3.8 27b would be extremely interesting.

https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B

And then a Bonsai ternary on top of that model.

kees99•about 1 hour ago

Re: bonsai - unsloth's quants have Q2 (UD-IQ2) variants, which are more or less same in size.

...or did Prism do something special with their "bonsai" releases? I didn't notice anything like QAT being mentioned.

drob518•about 1 hour ago

It’s still going to chew up context quickly. Surely, some of the added tokens are helping the model, but does it require as many as it generates? What happens on long, multi step tasks as it pushes old tokens out of context? I’m not sure we know the answers to those.

kzrdude•about 1 hour ago

It's fun to see "test time" scaling work out so well, maybe the best example of all.

hrmon•5 minutes ago

I want to highlight its (1-hallucation rate) at 70%. BRAVO! For me, this is its most wonderful score. GPT-5.6-Sol sits at 8%.

x313•22 minutes ago

I used this a lot over the weekend, and it's a really intelligent and strange model.

It gets really agentic at the higher reasoning levels. It does the basics like goal tracking and tool calling well, but more than that, it gets obsessed with solving problems and will do insane/unusual things to get to the solution. It actually reminds me of GPT-5.6-Sol-max which is similarly obsessive.

It doesn't surprise me at all that it outscores Opus 4.6. Opus had way better world knowledge but was more "human" with agent stuff - sort of lazy and uncreative, basically giving up once the obvious solutions failed. These newer models work more like magic, they are so creative and persistent at finding ways to get to the solution.

K0IN•about 1 hour ago

I used Qwen 3.6 27B extensively (>1B tokens) and DeepSeek V4 Flash (the older one also 2B+ tokens).

And I just can't fathom that the new 3.8 beats the new DeepSeek V4 Flash (which, in my eyes, is one of the best everyday coding models).

What an insane release, and convenient size to use every day/locally.

but i will test this model extensivly.

drob518•about 1 hour ago

I’ve been using v4 Flash 0731 a lot lately and you can’t beat the price performance. That said, it sometimes takes my prompts as more of a suggestion than a directive. I’ve found that introducing a reviewer subagent (even with the same model) helps push it back to what I’ve asked for. But makes every coding session a back and forth: “do X” -> “use a reviewer subagent to analyze whether you really did X as I asked”.

Saris•33 minutes ago

What model do you normally run the subagent on? You mentioned flash as well for that, but I wonder if a more 'strict' model would do a better job at pushing the main back on track.

drob518•14 minutes ago

For cost reasons, I’ve been using Flash for the reviewer, too, but I plan on trying to use Pro for that. Thus far, however, Flash has been doing well at reviewing. I’m cheap as I’m paying for all the tokens myself.

algo_trader•25 minutes ago

> I used Qwen 3.6 27B extensively (>1B tokens) and DeepSeek V4 Flash

Were your opinions effected by the harness ?

DS is an amazing combo. It probably could only happen in China, not in current USA or EU (for different reasons)

ignoramous•14 minutes ago

> It probably could only happen in China, not in current USA or EU (for different reasons)

Per Artificial Analysis benchmarks, Meta's Muse Glimmer 30b (open weight) holds its own (for agentic code workloads) against models 5x to 10x its size, too.

f311a•about 1 hour ago

How is the general knowledge of Qwen 3.6? Do you need to explain things outside of algorithms to it? Since the size is so small, I guess you need more explanations to it. General knowledge helps with coding when your don't specify a lot of details and ask for big changes.

JacobAsmuth•about 1 hour ago

It has double the active params.

K0IN•28 minutes ago

yeah but using the rule of thumb (I see floating in the Internet) which is sqrt(#params * #active params) which would give sqrt(284B*13B) = 60B, so deepseek should perform better.

apitman•17 minutes ago

Seems like that equation really should weight total params and active params differently

padolsey•about 1 hour ago

The smaller these frontier-nearing models get, the more I'm reminded of https://en.wikipedia.org/wiki/Lottery_ticket_hypothesis

keeganpoppen•about 1 hour ago

i think there definitely is some truth to this in terms of embeddings spaces, which is why i believe they are implemented by OpenAI/Anthropic in roughly highest import => least import bit order-- an overwhelming majority of the variance is in the first few hundred vector bits. i haven't actually tested this myself by manually truncating vectors, but it is my understanding that they generally speaking have this property.

kmike84•about 1 hour ago

I have an internal automated benchmark, which roughly follows my workflow, and I've been testing various models on it, local and cloud. Qwen 3.8 27B did awesome. Its understanding is correct, research is better than e.g. glm's (and I like glm), and implementation is good and careful.

Qwen 3.8 27B doesn't look benchmaxxed. These "52 AA score" numbers feel real, which is surprising. I've been using it locally for a few days for other tasks as well. If not the speed, I'd be totally happy to use it as a daily driver instead of cloud models, it is that good.

--- (benchmark, to get an idea):

1. First, initial prompt which is not super precise - similar to how I'd write a task when talking e.g. to Opus. I'm describing an idea, and asking model to come up with some plan, and also to criticize the approach. Task is about implementing a particular pi extension. I'm checking if a model actually understands what I'm asking.

2. Then, as a follow-up, I ask to research alternative implementations, research UX of similar extensions, etc. It needs to do web searches, inspect open source codebases, read articles and papers, etc. I don't prompt to do this exactly, but I expect good models to figure out they need to do it.

3. Then, implementation.

Also, one finding: Q4 and Q8 seem to have very different behavior in this benchmark. Q4 produces 2-3x thinking in the end, and makes more turns - it seems it makes more mistakes, and needs effort to recover from them, while Q8 gets more things right in a first try. In the end, quality is roughly similar, but Q8 gets there much faster, especially the implementation (tried it several times). Could be a difference between concrete artifacts, or between runtimes, I don't know, but be careful - it seems the real-world experience with qwen 3.8 27B can be vastly different, depending on how it's set up.

Regarding DeepSeek 0731 vs Qwen 3.8 27B. On this benchmark, Qwen understand my intent better, it's better at research, and I also liked its implementation more. But: if you're more precise in what you ask, 0731 is also very good, and it's quite a lot faster on mac; raw speed is better, and it needs less thinking to get there. So, I'd say it's a tie in practice, both are awesome :)

algo_trader•18 minutes ago

> So, I'd say it's a tie in practice, both are awesome :)

Which harness for the benchmark ?

You have previously commented on using OC/GLM. R u going to stock with it?

kmike84•6 minutes ago

> Which harness for the benchmark ?

pi, with a plugin to do web search / web fetch.

> You have previously commented on using OC/GLM. R u going to stock with it?

For personal use - probably yes, z.ai + kimi + opencode go subscriptions, with some share of local models now. For work - claude code, codex.

ignoramous•13 minutes ago

> I have an internal automated benchmark ... I've been testing various models on it, local and cloud

Once you send your benchmark to "cloud", I don't think you can rely on it being secret/private any longer.

kmike84•5 minutes ago

Heh, a good point.

f311a•about 1 hour ago

Why is it so small, but expensive?

Open Router

Input /M $0.45

Output /M $3.20

Cache read /M $0.05

Throughput 27 tps

It would be a very nice model at 200-300 tps and if it was dirt cheap. What's the limiting factor of optimizing speed and price for inference providers?

AgentLemon•37 minutes ago

It's a dense model, so 27B active parameters to compute. Compare that to DeepSeek V4 Flash, which has only 13B active parameters (MoE).

freakynit•about 1 hour ago

I read it somewhere recently that it's architecture does not allow serving as many concurrent requests as the deepseek models allow. Maybe that's why.

FuckButtons•38 minutes ago

From first principles, 27b dense vs 13b moe, means you spend ~2x more memory bandwidth per request amortized over the whole server (ie, you assume all experts are being concurrently used by some user during the forward pass, then on average the bandwidth required for one forward pass for any individual request is just the size of one expert). deepseek also have some innovations around kv cache and compressed attention which allow for further reductions in memory bandwidth which means that the thing that’s actually bottlenecking inference, (memory bandwidth) is significantly lower than for qwen 3.8, which has been optimized for running 1 instance ~= 1 user.

theanonymousone•about 1 hour ago

That's my questions as well. DeepSeek v4 0731 is served dirt cheap and it needs 10 times more RAM.

kmike84•about 1 hour ago

DeepSeek needs more RAM for weights, Qwen requires more compute.

Also, DeepSeek's KV cache requires less RAM than Qwen's. In concurrent situations (on servers) you load model weights once, but you have different context in each parallel session. So, it can also need less RAM than Qwen to serve, even if it's a larger model.

petu•about 1 hour ago

> and it needs 10 times more RAM.

More like 3-6.

Qwen 27B full quality is FP16. So 54GB. In practice most inference providers would serve FP8, so 27GB.

DeepSeek V4 Flash in full quality is mostly FP4. ~167GB official release.

So Deepseek has 140GB model size overhead... which is shared between 100s of users single inference node serves, so not even a gigabyte of VRAM per user.

Memory required for 200K of context per user:

  V4 Flash: 1GB. 
  Qwen 27B: 13GB.

thenews•about 1 hour ago

more profits i guess, https://news.ycombinator.com/item?id=49323381

anana_•about 3 hours ago

For more context, this puts it on par with models like GLM 5.2 and GPT 5.6 Luna, which are far larger

anana_•about 2 hours ago

And to read the tea leaves a little:

3.8 actually performs slightly worse than 3.6 on AA-Omniscience Accuracy, which could imply that they traded out world knowledge for capability in other areas.

It also produces nearly twice as many tokens per task as 3.6 (and by extension, time), which may be a tradeoff required to achieve correctness at this parameter size.

skohan•about 2 hours ago

Imo it makes sense for things to move in the direction of small, focused models that excel in one area. I use LLMs for technical work 99% of the time, I could care less about general world knowledge, or if the model is good at creative writing.

With good orchestration and delegation you can get surprisingly far with small models running on consumer hardware.

tancop•about 1 hour ago

The biggest untapped market is pure agentic models that are built for tool calling and non hallucination instead of memorizing facts. You need some world knowledge (as in common sense) to build a useful model, but I don't think perfect recall on general QA is a good use of space when you have web search and structured knowledge in Wikidata or Wolfram Alpha.

Training should focus on tasks that require real intelligence instead of memory. Creative writing is actually good for this if you score it on coherence instead of getting random real life details right. Basic level of coding (simple prompt to code, don't need to one shot complex projects) is also great because writing a small script is more efficient than 20 separate tool calls.

anana_•about 1 hour ago

Agreed. Luckily, this model also scores high in AA non-hallucination, so it knows what it doesn't know -- perfect for situations where it can just tool call a web search.

drob518•44 minutes ago

Yep, exactly. I keep saying that I want the “coding expert” extracted from these multi-T parameter models to run locally on reasonable hardware (large laptops, not servers). Yea, I know there’s no single “coding expert” that you can actually extract in these models, but you get what I mean. Like you said, when I’m coding, I don’t care about world knowledge, and I’m fine with consulting another model when I need that.

bertili•about 2 hours ago

And more context:

Same score as the latest DeepSeek Flash 0731 which has 284B parameters! (13B active)

Its also the second best Qwen model, much better than Qwen 3.7 Max, but significantly below Qwen 3.8 Max.

anthonypasq•about 1 hour ago

isnt the active parameter count more relevant than the total? qwen is a dense model no?

kzrdude•15 minutes ago

For some tasks yes, and we don't know how many active parameters Luna is using..probably less than 27B

nsingh2•about 2 hours ago

Also with Qwen 3.8 being more token hungry than Luna, using around 2.3x tokens. Which hurts for local deployment.

sottol•about 2 hours ago

I'm torn on this - on the one hand performance matters, on the other so does capability.

I could run Qwen 3.6 27B on my laptop, but at 5 tok/s it was too slow even without overthinking - I never used it. OTOH, Qwen 3.6 35B A3B ran at 20 tok/s but it just could not get done what I asked of it. It sort of got close but you had to repeat and retry so much that it might have been faster to run 27B dense... maybe?

So that said, I might take a much better model that runs 2-3x slower (total time per task) but that's more capable over a faster, less capable one.

I'd also like to try a proper "plan-then-execute" type execution where thinking is entirely disabled (or low) during the execution stage but enabled/max during the planning stage.

I will definitely give 3.8 27B a better shot than 3.6 though.

skohan•about 1 hour ago

Depends on your use-case. Over the past couple days, I've found 192k context more than enough for coding. There's more thinking for sure compared to comparably sized models (running on xhigh), but I've found the results are so much better that the entire session consumes less tokens on average since weaker models need more review passes.

johnnyApplePRNG•about 2 hours ago

We don't actually know how large they are, actually.

halJordan•about 2 hours ago

Well, ackshually. E do know exactly how big glm 5.2 is. And there's more than enough data to draw conclusions about luna. Or are you one of the guys who says "big bang is just a theory"?

knicholes•about 1 hour ago

The big bang is a theory. It's not JUST a theory, however.

catigula•37 minutes ago

Which should tell you how useful these benchmarks are.

sp1982•about 1 hour ago

Perhaps model size and reasoning length trade off to some extent, similar to CPU vs. RAM. A smaller model with a longer reasoning trace has more intermediate structure to latch onto and build on.

deflator•about 1 hour ago

Makes sense to me.

We will see, since if true then it is likely the other makers of small, dense models will copy it and include high reasoning by default.

If that also makes the other dense open source models better, then you are probably correct.

colingauvin•about 2 hours ago

It's 7th (!!!) overall on the agentic index, above Terra.

hadlock•about 2 hours ago

Strangely Qwen 3.8 Max isn't on their list, at all.

colingauvin•about 2 hours ago

It's 3rd at 58.

JV00•about 1 hour ago

Why is it not included in the Pareto line intelligence/cost chart?

leprials•about 1 hour ago

Theres no official API yet. So theres nothing to price against.

ignoramous•6 minutes ago

> So theres nothing to price against.

To my surprise, providers on OpenRouter (io/akash/chutes) are serving Qwen3.8 27B at ~ $0.4 (in) / $3 (out) / $0.25 (cache), more expensive than DeepSeek v4 Flash.

https://openrouter.ai/qwen/qwen3.8-27b / https://archive.vn/RrDGO

WithinReason•37 minutes ago

Just count the cost of electricity then :)

culi•about 1 hour ago

Most are running it locally

prakashbuilds•about 1 hour ago

Interesting to see where local models are going to be in the coming days. I am already starting to believe open source models are the way to go in the coming days. With Qwen 3.8 Max, Kimi K3 etx already delivering at part perf with frontier models, the future is going to be exciting.

sottol•about 2 hours ago

A lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to get at least an idea on relative performance of models, is that still warranted?

What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?

drob518•31 minutes ago

IMO, ELO rating from The Intelligence company and arena.ai are more representative of rankings since they use humans to judge a head-to-head comparison between a couple models at a time. https://www.intelligence.ai/ http://arena.ai

re5i5tor•about 2 hours ago

Have you tried it? I’d recommend doing so, it’s impressive in real use cases.

aqme28•about 1 hour ago

Do you have any evidence that this model is bench-maxxed? I know that's particularly difficult to quantify. If there is an indicator of bench-maxxing, that just becomes the new benchmark to benchmax.

deaux•about 1 hour ago

Here, filtered down for you. [0] Look at the individual benchmarks, not the combined one. You can tell that this model is much more benchmaxxed as its relative ranking swings between benchmarks is much larger. This is a hallmark.

[0] https://artificialanalysis.ai/models/qwen3-8-27b?models=deep...

throwa356262•28 minutes ago

I dont think this is benchmaxing.

They have simply decided to not train the model in some areas such as world physics

achrono•40 minutes ago

Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzzle, which you then ask the model to solve mathematically, all wrapped as an ask to generate ASCII art. Yes, all these elements exist in some form in the evals but the key is in how utterly unconventional the elements are that you pick and in how you combine them.

I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.

skohan•31 minutes ago

At that point aren't you just edge-case testing?

Surely most of your use-cases are not novel tasks that combine obscure domains.

It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.

deaux•about 1 hour ago

This one is very benchmaxxed, and you can tell from this page alone. Look at the huge variance in ranking per benchmark. Most models, including at that size, are much more consistent.

culi•about 1 hour ago

This website aggregates 82 different benchmarks. It seems to do pretty consistently well across the ones its been tested on

https://benchlm.ai/models/qwen3-5-27b#ledger

Iolaum•about 2 hours ago

Yea and we are reaching the point where this benchmaxing is visible in the model's reported overthinking.

logicchains•about 2 hours ago

It's not overthinking, it's the right amount of thinking necessary for such a small model to get good results. The dumber the model, the more it has to think to be smart. There's no easy way to reduce the thinking without reducing the model quality.

zdragnar•about 1 hour ago

Qwen doom loops were amusing to watch the first time or two, but it's incredibly vexing to have it waffle over the same decision over and over and over and over again. I can get more done with a faster model by correcting it, and it feels better to babysit them than it does to babysit qwen to see if I need to intervene or if it will actually finish.

I do like the output from qwen when I get it, but honestly I haven't been impressed enough with it to put up with the downsides.

dethos•about 1 hour ago

I'm impressed with the score. This is a model that runs on a good, but still regular, desktop PC.

johnnyApplePRNG•about 2 hours ago

Unbelievable. Bravo Qwen team.

IronWolve•about 1 hour ago

Anyone try the 9B/2B distills yet? Wondering how they do for local tools

armcat•about 1 hour ago

So it's effectively on-par with GLM 5.2 and GPT 5.6 Luna?

bertili•about 2 hours ago

I can't shake this the existential feeling that this compact series of 27G bytes represent something profound and universal.

kzrdude•about 1 hour ago

Raw model size is around 27x2 GB since it's in BF16 format

apitman•about 3 hours ago

Very interesting. I was not expecting anything close to this.

cardboard9926•about 2 hours ago

Where's GLM 5.3 score?

jakswa•44 minutes ago

I'm waiting for this comparison too. I was impressed by a 1-shot GLM 5.3 did for me the other day.

matheusmoreira•about 1 hour ago

It tied with Luna/max. Simply incredible.

Lynnr•about 1 hour ago

Monster pure platinum cable for $100,000 per yard scored 3.1415927 on the audiophile benchmark!

manofmanysmiles•about 1 hour ago

Imagine this, and sucesor models on Cerebras or other silicon...

WASDx•12 minutes ago

That might actually compensate for the overthinking, if it can think really fast. Dense models are easier than MoE to put on silicon. https://chatjimmy.ai/ is getting 16k tps with an 8B model. Extrapolating that gives nearly 5k tps for 27B. And we're still early in this technology.

If tps is so high, a compaction step could be performed over every thinking turn to keep context size down.