State of Open Models: Summer 2026 Observations

cl42•about 15 hours ago

I recently read a few articles about how the harness is a bigger factor to successful LLM usage and wish they discussed this here.

I use GLM-5.3, Qwen3.8, Claude (all of 'em), GPT Sol/Luna/Terra across direct API calls + local models where I can (128GB Macbook Pro)... The harness and whether the model or underlying system prompts know how to make the best use of iterative LLM calls makes such a big difference...

For example: one-off articles on a news topic (e.g., "Update me on the US-Canada relations") yields very similar results across all models... But "run a web search, write a draft perspective from three points of view, and structure data around it" will make everything but Claude + GPT struggle.

james2doyle•about 14 hours ago

Funny enough, I have a simple test that I have been running on each new model that catches my eye on OpenRouter. It is just a short prompt that asks the model to research yesterdays news and summarize it in a specific format along with a critique of the article or a highlight of any bias: https://gist.github.com/james2doyle/6afb04ea6b18e1a36bc45258...

I have been doing this test for about a year now. I run it on different harnesses and apps as well just to give me an idea of what they differences between them might be. Since I have been running it for a while now, I have a good sense of the correlation between this output and how the model will be on the rest of the things I want it to do.

I would say that almost every model (just tested Mercury 2.5 Preview and IBM Granite 4.2-8b) are pretty much the same on this task. Some are more diligent with how many sources they go out and get, but for the most part they are all very close in quality and will follow the instructions very well.

So saying "everything struggles" with that has just simply not been my experience.

gwerbin•about 14 hours ago

Anecdotally it's a mix. Claude is so good in part because the models are clearly trained to use the harness, and the harness (despite questionable UX) is really best-in-class when it comes to its functionality.

> run a web search, write a draft perspective from three points of view, and structure data around it

Ironically that's not harness-heavy at all, is it? Apart from sterring via system prompt, that's largely relying on the model itself to reason through the task (what to search for, which links to follow) and then synthesize the information and present it in a way that meets the user's request. Seems like a good test of pure LLM capability to me.

I find it hard to believe that if GLM 5.3 struggled with that task in, say, Pi, it would do any better in OpenCode. Unless you're talking about some next-level research stuff to provide strong guidance/steering and context offloading.

jst1fthsdys•about 12 hours ago

I had OMP running GLM-5.3 download multiple CSV datasets for fantasy drafts, classify them according to a strategy I am following, and then design a local site I could use for my draft, that would dynamically account for other picks and give a useful overview of what was available for what I had at the time of pick. 4 inputs from me, two related to how I'd run it on draft day. Worked great!

anukin•about 13 hours ago

Which harnesses do you use?

I usually use pi or minimal harnesses when I am working with Anthropic or OpenAI models. With local llms they seem to work well with maximalist harnesses like omp, Hermes etc.

hgoel•about 15 hours ago

>But "run a web search, write a draft perspective from three points of view, and structure data around it" will make everything but Claude + GPT struggle.

Did you mean with some specific harness? GLM5.3-Flash handled a very similar test I ran in opencode (with Kagi MCP to let it search) pretty well, and coming from Claude it was incredibly refreshing to not have to decipher its absurd techbro-speak.

AnodicElegy•about 15 hours ago

The data and graphs are great, but it would have been a much higher quality report if the text and titles were written by a human.

john_rood•about 15 hours ago

The harness share is the most interesting new data here, but raw request volume can be misleading. A noisy agent with a wide tool loop may generate 10x the Hub calls of a more efficient one. I'd love to see successful outcomes per 1,000 agent-tagged calls, segmented by harness, task class, model, and tool-error rate. Otherwise this tells us which clients are busy, not which ones are effective.

State of Open Models: Summer 2026 Observations

⚡ Community Insights

Discussion (8 Comments)Read Original on HackerNews