TechPulseTechPulse
Back to News
Advertisement
Advertisement

⚡ Community Insights

Discussion Sentiment

50% Positive

Analyzed from 196 words in the discussion.

Trending Topics

#more#task#cost#per#cheaper#behavior#haiku#benchmarking#tasks#important

Discussion (5 Comments)Read Original on HackerNews

cpard•25 minutes ago
This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens.

I have experienced similar behavior between opus and haiku when benchmarking Dara engineering tasks. The “cheaper” model takes many more turns to figure out the task and this is without taking into account other important factors.

Another interesting behavior that I observed is that Haiku tended to cheat more maybe because it was having a harder time to find the root cause of the problem.

Benchmarking and evaluation of agentic systems is very interesting and if there’s one thing that someone should keep from the Databricks post is how important is for everyone to build and run their own.

yodon•about 1 hour ago
I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?
falaki•about 4 hours ago
1) Many models are now competitive at the top tier, including open source. 2) GLM 5.2 in particular was a major step forward in open source coding agent performance, 3) Harnesses make a huge difference in cost-performance. 4) Cheaper per-token does not imply cheaper per-task.
falaki•about 4 hours ago
Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench.
vegetablefinger•28 minutes ago
k;l