Back to News
Advertisement
Advertisement

⚡ Community Insights

Discussion Sentiment

67% Positive

Analyzed from 2133 words in the discussion.

Trending Topics

#model#models#don#apps#https#llm#gpt#interesting#grok#tokens

Discussion (73 Comments)Read Original on HackerNews

smusamashah•about 2 hours ago
"One honest caveat", "no glitches, no color changes" good tests and I read it to the end but I wish it was written by a human.
ranyume•about 1 hour ago
You are absolutely right!
paxys•about 2 hours ago
> Separate question, separate table. This is our standard latency harness (three short prompts, five reps, 400-token cap), not the build tasks. tok/s is output tokens over wall-clock, uniform for all.

> so their tok/s is a ceiling, not a true decode rate. The clear read: the GPT-5.6 tiers are the snappiest models here on short prompts (Luna answers in about a second), Qwen is absurdly cheap and fast, and DeepSeek and GLM are the slowpokes

You put in a lot of good work, and kudos for that, but man, reading paragraphs like these just puts me off of the entire piece.

Like…how hard would it have been really to type these two sentences by hand, in your own natural voice?

fluidcruft•about 2 hours ago
Yeah, I can't figure out where this "voice" comes from and it is so impossible to get rid of. It is so grating.
jaggederest•about 2 hours ago
You can suggest a different one and it works pretty well, the bar is very low.

I asked for hemingway in a planning document one time and the result was highly amusing to everyone. "We will not wreck it with small greeds." was an all time favorite for me.

Chu4eeno•about 2 hours ago
The problem is that everyone (well, too many) is now using the same "voice" when writing.

So no matter how good or thoughtful the writing is it gets tiresome.

furyofantares•about 2 hours ago
That's not true at all, human tends to come through (in a way I didn't notice pre-LLM) with varied tone, with opinions injected, and with varying degrees of weight to different components, in all but the most egregious of examples (linkedin motivation pieces, apple marketing speak).

LLMs have a bunch of tricks for dressing up their infodumps, but they are almost purely infodumps, and no real opinion comes through. There's no sense of more importance to one statement or the other, it's all monotone (and usually over the top.)

zeven7•about 2 hours ago
Man invents text generation machine.

Man generates text.

sublinear•about 2 hours ago
It's the unholy trinity of pundits, marketing, and HR.

Their writing was already painted into this corner long before the LLM epoch and they continue to publish more than anyone else.

razighter777•about 2 hours ago
It's so telling and offputting.

Please write like a normal human and put the effort in to type what you want to say. Using AI to make your writing is not only lazy, it's bland, tiresome, and disrespectful of the reader's time.

TrackerFF•about 1 hour ago
For the arguments sake: What if that is the authors natural voice?
stickfigure•about 1 hour ago
Eh? I write that way sometimes. Long before LLMs.

I'm sick and tired of reading comments like these.

jakubmazanec•about 2 hours ago
> how hard would it have been really to type these two sentences by hand, in your own natural voice

On the other hand, do we have to complain about every seemingly AI written text?

paxys•about 2 hours ago
Yes? AI generated text is explicitly disallowed on HN, so it’s not crazy to expect that a similar standard be used for linked content.
kbelder•about 2 hours ago
But AI generated linked content was explicitly not disallowed. At least not at this time.
fluidcruft•about 2 hours ago
It is just horrible writing style. It doesn't particularly matter that AI wrote it.
platinumrad•about 2 hours ago
Maybe I'm a control freak, but asking agents to one-shot random apps is nothing like how I actually use AI in software engineering.
mikeocool•about 2 hours ago
Yeah, the models have all been really good at generating greenfield apps for a really long time (in the scope of LLM time).

I suppose it’s interesting to see how they make better greenfield apps. But I am much more interested in how they solve hard problems in existing gnarly codebases.

bhu8•about 2 hours ago
Absolutely yes, but that's how you become twitter/X famous
thsbrown•about 2 hours ago
Man I do ponder this all the time.
fragmede•about 2 hours ago
It's not, but it's trying to bring any level of objective measure in this realm, vs just going off of vibes.
thebigspacefuck•about 2 hours ago
(LM)Arena is basically this. IMO it’s the best benchmark that avoids benchmaxxing

Agent: https://arena.ai/leaderboard/agent

Web dev: https://arena.ai/leaderboard/code/webdev

Currently Fable and 5.6 are neck and neck on web dev which is basically the same finding as this.

tedsanders•about 2 hours ago
Arena can definitely be benchmaxxed a bit, if you try. The distribution of prompts there is very different than usage by regular coders. E.g., lots of requests for one-shot games from scratch. So if you fine-tuned your model to be great at making fun one-shot games from underspecified prompts, your coding model might look better than it is (on general tasks, at least).

I work at OpenAI, and am happy to say we don't try to juice our scores here, as doing so would be counterproductive and make Arena a worse signal for everyone.

Chu4eeno•about 2 hours ago
There's a ton of arenamaxxing going on (especially from facebook), though I don't disagree that it's one of the better actual benchmarks.

Always fun to ask them to recreate classic demoscene effects (sadly they're still pretty bad at generating music, though at least claude seems to create decent synths).

I keep trying to get them to recreate the fluid+particle stuff from Agenda Circling Forth etc., but even giving them the blog posts describing the implementation (and screenshots) they're still pretty bad.

small_model•about 2 hours ago
Doesn't have Grok 4.5 listed yet, wonder why 5.6 is, it was released later?
rbehrends•about 2 hours ago
My concern with most of these visual benchmarks, popular as they are, is that they are likely more indicative of knowledge (i.e. how comprehensive the training data is and how well it can be retrieved from the model) than of reasoning ability. I don't see in particular how a model would construct a CoT that mapped somehow to a representation of the cube geometry and its animations in latent space without a large chunk of that being pre-existing information.
nomel•about 2 hours ago
> without a large chunk of that being pre-existing information.

Is there any evidence that novel reasoning is present in LLM? I've never been able to make that work, and I believe Apple's paper some time ago was good evidence that it doesn't exist. In my experience, sparse latent spaces result in a complete, comical, failure in reasoning.

drivebyhooting•about 2 hours ago
See the new mathematical proof published by OpenAI.

I’m not very valiant to verify its veracity. But even if the math is merely derivative it merits mention.

nomel•about 1 hour ago
True, but that's an unknown internal model, without details of the architecture. We'll have to see if the LLM model, itself, was responsible for the "novel" bits, or if it was stuffs bolted onto the LLM that made it possible. I suppose "LLM" is maybe no longer sufficient to describe the systems that LLM are being integrated into, so maybe my point is pedantic/semantic.
didip•about 2 hours ago
I actually like this methodology of testing AI much better than all the other benchmark tests.

Real world is messy, other benchmarks are clearly gameable by the Chinese open models.

Great job! And I don’t care about the tone of the article, it’s readable just fine.

ricardobeat•about 2 hours ago
Obviously AI-written, but I'm confused with the results: Muse Spark has the best Rubik's cube by far, the only one properly animating, yet it gets a 2/5

(edit: seems to be an issue with inline videos)

rsstack•about 2 hours ago
2/5 isn't quality, it's consistency as written there. The full links are at the bottom. Most of Spark's attempts are failures:

https://d1md4c6gq9re9p.cloudfront.net/blog/gpt-5.6-buildoff/...

https://d1md4c6gq9re9p.cloudfront.net/blog/gpt-5.6-buildoff/...

https://d1md4c6gq9re9p.cloudfront.net/blog/gpt-5.6-buildoff/...

ricardobeat•about 2 hours ago
Ah, I missed that, and didn't click through the links. Most of the videos are not showing any animation for me, only Opus / Qwen / Muse, so Grok's attempt looked broken.
dang•about 2 hours ago
Recent and related:

We made Grok 4.5, GPT-5.5, and Claude build the same apps - https://news.ycombinator.com/item?id=48838772 - July 2026 (92 comments)

sgk284•about 3 hours ago
Similarly, we updated our model arena (52 apps each built by 26 models) to have GPT 5.6 Sol, Terra, and Luna today:

https://arena.logic.inc/

It's really interesting to see the Sol/Terra/Luna apps side-by-side.

I need to add these stats somewhere in the UI, but one interesting take away: Terra took 1/2 as much wall-clock time as Sol, but Luna took more wall-clock time than Sol (by about 23%). It's still much much cheaper, but it seems like Terra is likely a more optimal time/cost balance for most use cases.

The Terra quality is usually nearly as good as Sol, but much faster and cheaper. I do appreciate Sol's design sensibilities (see, for example, the audio sequencer). It's the first model in a while that is clearly distinct on that front. They'd all converged to very similar visuals for a while.

vitorsr•about 2 hours ago
What caught my eye was:

            Model  Lines of Code  File Size  Gzip Size 
      GPT-5.6 Sol          1,264    35.5 KB    10.0 KB 
    GPT-5.6 Terra            827    20.0 KB     6.7 KB
sgk284•about 2 hours ago
Yea, that's an interesting result as well. The Terra apps don't feel 35% less feature-rich. So it seems quite token efficient.
ianm218•about 3 hours ago
This does seem to validate the critique that models like GLM are benchmaxxed and not as close to the frontier as you’d think based on their numbers.
orliesaurus•about 2 hours ago
Missing the exact prompts - would love to replicate...but also curious how you prompted these: they could be a big reason why some models failed completely at rendering SVGs (ie. GLM 5.2)
Advertisement
dofm•about 1 hour ago
Luckily for the future of the industry we mainly need casual games…?

The lessons that should have been learned here, surely, include:

1) you probably should not one-shot apps like this unless you're really not that bothered with consistency

2) if you are remaining in control of the code you generate, Qwen 3.7 plus is pretty competitive with Fable.

My questions:

How is "good results when it worked" a 4/5 score?

And how can any of these really be considered indicators of performance for the "genuinely novel" when the results are all so similar?

konart•about 2 hours ago
Not sure what prompt was used for GLM 5.2 but here is mine:

> Draw a horse riding an astronaut in svg

https://www.svgviewer.dev/s/if4gi3e7

sangupta•about 2 hours ago
Sign-in via Google is broken - it redirects back to localhost from Supabase :)
NBJack•about 2 hours ago
Given this appears to completely exclude Google's models, I'm not surprised. Even Muse is in there. I guess they aren't fans.
orliesaurus•about 2 hours ago
Really nice breakdown, surprised by the results - especially the fact that OSS models were so behind on most task... (lol at the SVG of the moon without any sign of life by GLM-5.2)
joehabeebs•about 3 hours ago
Interesting tests being done but I can't help but think it limits testing innovation in some way given that the requested apps are essentially all clones of others
christophilus•about 3 hours ago
I hear this take a lot, but every app I’ve ever built was like 80% similar to every other app out there. The unique/ creative part of an app is not the bulk of it, and LLMs have been pretty good at helping me explore the 20%, too.
billyp-rva•about 2 hours ago
Calculator / Rubik's cube / game of life apps should be very close to 100% identical, right? I don't see the point of asking an AI for one of these when there are dozens (hundreds?) of repos that all have exactly what you want.
kibae•about 3 hours ago
The cost seems to be using the wrong symbol: ¢ vs $
delichon•about 2 hours ago
Nope, they're that cheap. E.g. Grok 4.5 is $.02 to $.06 per million tokens. A 400 token reply costs ~.002¢

https://www.tryai.dev/models/grok-4.5

Update: kibae above and below is correct and I'm not. They have fixed their blog post.

il•about 2 hours ago
Grok 4.5 is $2/$6 there's no model anywhere close to that cheap
delichon•about 2 hours ago
The numbers come from the tryai.dev link:

  How much does Grok 4.5 cost on TryAI?
  Grok 4.5 is Input: $0.02 / 1M tokens, Output: $0.06 / 1M tokens. There is no subscription — you pay only for what you use.
master_crab•about 2 hours ago
A lot of these are visual-heavy tests that often require first person sight to confirm results. Considering GLM isn’t multimodal, that might explain why it did better on the calculator question and not much else.
dinkleberg•about 2 hours ago
Is this how I learn that Bezos now has a beard? Interesting that it is a detail that all of the models chose to include (unless that was in the prompt and just not put in the post).
losvedir•about 2 hours ago
I think there's approximately zero value in seeing how a model can turn 100 tokens into a 100k. What workflow is that? It's not useful in the real world.

I want to know how well it can follow instructions, manage various potentially competing desires in the context, and so on. It's much more interesting how it can turn 100k tokens (e.g. a codebase and lots of tool calls) into 100 tokens.

CompoundEyes•about 2 hours ago
It’s interesting how all the model names and versions are like SKUS taking up space on a display shelf. I look forward to whatever Sagittarius A* does!
Advertisement
esafak•about 2 hours ago
Could you make the tables sortable?
throw310822•about 2 hours ago
"Elon and Bezos watch a Blue Origin landing" svgs are super cute, and incredibly like children's drawings. They also nail Bezos' features pretty well.
ttoinou•about 3 hours ago

   "This isn't objective." Correct, and we are not pretending it is. We are not handing down a scientific verdict. 

Actually, you are doing rational investigation in a fuzzy probabilistic new/emergent space, with open sharing to the world. I don’t understand why people downplay themselves and put on a pedestal others supposedly serious sciences.
minimaxir•about 2 hours ago
It's a preemptive defense against methodology cynicism seen often on sites including but not limited to Hacker News. I've been guilty of including such defenses myself over the years because I've gotten annoyed with receiving such cynicism.

Look at the top comment on their previous HN submission: https://news.ycombinator.com/item?id=48839886

dang•about 2 hours ago
Ugh, that is bad, and I'm sorry I didn't see it at the top of the thread yesterday.

Really, the deeper problem is the upvotes that cause such posts to rise to the top of a thread and stick there, drowning out curious conversation and giving people a bad impression of the entire community. Unfortunately, the upvote problem seems basically unsolvable - people don't do it consciously and it's very much a tragedy-of-the-commons problem. So we're stuck with moderation on the comments.

boondongle•about 2 hours ago
Ultimately advocates exist for models and there are incredible financial incentives for some to be advocates, so authors are guaranteed someone being mad if their horse doesn't perform well.

Given that type of reaction is inevitable, it just saves the conversation.

sixhobbits•about 2 hours ago
because if you don’t put this disclaimer the top comment is always "Acthually this isn't real science because you didn't publish your P value" so you can't win.

also the article itself is clearly LLM generated though

tshaddox•about 2 hours ago
Indeed, although it is important to note that science is a proper subset of "using reason to solve problems."
jakevoytko•about 2 hours ago
Because for its entire existence, the top HN comment on articles is typically a contrarian take or pointing out flaws. This goes double for a study, where people just hunt for some aspect of the methodology they dislike. If you don't address the flaws, then it looks like you never considered them, and the top comment will say that your entire methodology is suspect. It's super predictable to the point that you can harness this kind of reaction to get stuff on the frontpage if you really want to.
adammarples•about 2 hours ago
Because serious science is hard and valuable for its rigour, and shouldn't be compared with just poking at data to see what happens
chris_money202•about 2 hours ago
Don't be fooled, there is politics, opinions, and less rigor in science as well.
yieldcrv•about 2 hours ago
tower defense against pedantic autists who miss the social cue of “does it matter?”
CharlesW•about 2 hours ago
> We generated a big pile of artifacts, we are publishing all of them, and you can form your own opinion.

My opinion is that two gimmicky "one-shot prompting shootout" marketing pieces in two days smells like desperation. I'm not sure you understand what a turnoff this is for potential customers.

nomel•about 2 hours ago
Say you were interviewing a human, to see how capable they were. You are allowed to give them take home work. What kind of questions would you ask, or tasks would you give, to try to get a measure of their competence? If you gave them a task, would you iterate with them on the design, or would you see what they could produce on their own, without input?

Measuring "intelligence" is hard, but giving an "intelligent" entity tasks, and seeing what comes out, and then comparing the output with others, seems like a very reasonable, relative, way to do it.