ES version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
62% Positive
Analyzed from 4255 words in the discussion.
Trending Topics
#more#models#databricks#code#using#software#cost#don#experience#companies
Discussion (129 Comments)Read Original on HackerNews
- Spend most time prioritizing/discussing what to do.
- Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign)
- Use Opus 5 or Sol Med to execute
- Auto-fix bugs and CI until green + thermonuclear review skill x3.
- Manual interrogation of change/nits
- Come up with QA plan and have Codex Computer Use execute on it
- Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)
I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.
When I personally want tight code, I have to spend a considerable amount of time adjusting it manually:
- It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating
- I need to consider whether there are solutions based on higher-level assumptions, that AIs typically miss
- I need to check whether there are off-the-shelf solutions - AIs like to reinvent the wheel
IMO, software production has become a mass-produced commodity in every sense - it's much more expensive to produce software manually, but the quality is not the same.
(1) https://simpsons.fandom.com/wiki/The_Homer
> software production has become a mass-produced commodity in every sense - it's much more expensive to produce software manually, but the quality is not the same.
Agreed.
You will really have to weigh the cost of making the software against the expected revenue.
For who?
The public? The public has never liked buying software at any price.
Businesses? Businesses need higher quality software when it's relevant to their core competencies, so they hire people instead. Buying competing SaaS or depending too much on AI is throwing the baby out with the bathwater.
[1]: https://pluralistic.net/2024/04/24/naming-names/#prabhakar-r...
Also just remember - minimalist code looks and feels great but customers do not read your code. I have caught myself many times providing "corrections" to abstractions that were already ~fine, just not perfect. The average SWE costs $200/hr. Careful you don't burn $50 worrying about code that will likely be rewritten or can be better abstracted when that's actually needed.
This is a pointless quibble but the hourly rate claim is not true--it's like ~$60 in the USA [0]. Maybe you meant at a specific Org but this is important context when comparing "pricing" between human and AI.
[0] https://www.salaryexpert.com/salary/job/software-developer/u...
"Will I benefit from this code being minimalist before [date]", where [date] is whenever you think the agent will be good enough to come back and make the corrections you would make today.
And this is how I find out I'm woefully underpaid.
How does a “small startup” end up with a multi million line “legacy” codebase? Something not mathing
And if you're saying (based on your other comments) that a 6 month window is enough to create a legacy codebase...that indicates a serious lack of experience or understanding as to what a legacy codebase is, or why they exist.
I have had some $3,000 token days - even without Fable. I don't see how this is sustainable.
My personal 20x plans get so much usage for so cheap. The consumer subsidies are crazy, but alas I can't use them for work.
- Suggest a better approach that makes the AI say, “That’s much simpler. And you’re right. My original plan was over-engineered.”
It actually gives me quite an uncanny feeling, bulldozing over years of human optimization work with a newer, "perfect" design. Like bringing an AK-47 back to the middle ages.
it's so hard to square such totalizing statements with my day to day experience with fable and sol, (every possible, really??) arguing with them/my colleagues' agents that no they have slowed down the system 200x with their terrible change, doing string operations on millions of db rows, trying to get it to understand that I don't care that it's calling it a "cache" if a cache hit is slower than what we had before.
These agents do let you learn codebases quickly, and produce code way faster. I don't look at IDEs all that often. But literally multiple times every single day I catch them doing something stupid.
I don't think its impossible that we could get better performance from the agents. I know ive tried all sorts of workflows and skills, few of which seem to have much effect on the things the models struggle with. I think a big part of it is encoding enough context for large codebases, and providing it with all the tools it needs to make it successful, things to automatically check its work, etc. But that's not automatic, in fact its generally a terrible judge of what it needs or what its bad at
Opus and I wrote a parallelized test harness and labeled groundtruth in around 2 hours.
In 2022 that would've likely been all I did for a couple sprints
Gotta love this loop, I have it running while I'm asleep all the time.
Are you at least conversational in the subject matter? You're gonna have a good time just by paying attention and adjusting your workflow. If you're getting a lot of back and forth with it, its asking a lot of planning type questions, stop, step back, rethink the whole feature, and start again from the beginning with everything more fleshed out.
If you are in a brand new field, there's no way to bridge that divide. The issue is you don't know what is good or bad, or whether what you have learned is good or bad. You're in a sports car and you don't know how to drive much less what's track and what's field.
You can spend a lot of effort getting good at prompting towards writing tests and E2E tests to at least verify your app does what you expect it to, regardless of experience.
I actually have no doubt that I could replace my Opus 5 Low/Medium subagent profiles with Grok 4.5/GLM 5.2/Deepseek v4 Flash and perf would probably be pretty similar.
On top of that - highly recommend adding accurate cost counters to your statusline. You can't improve what you don't measure! (Or even have any intuition about).
> I probably spend like $80 a day
This doesn’t sound like “unlimited”, I spend more than this out of pocket per day and I have a strict budget.
Possibly, but the output of a 2022 engineer is about 1/10th of the output of a 2010 engineer, so it's an extremely low bar.
I would love to see what these tools create but outside slop there's never: This works, is in production, here's the code.
Any day now.
One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)
On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.
Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.
1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage.
2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money).
3. Nobody, not even the model provider, knows what your request will cost before it returns. You're writing a blank check every time you hit enter.
4. When you run out you run out very suddenly and disruptively. It's very hard to tell a developer on the 28th of the month "sorry, code by hand until the 1st of next month" so you tend to grant exceptions.
5. The price is changing all the time. New models come in, old models come out, prices change, caching behavior changes, harnesses change, etc. The cost of doing a single task is not predictable even if the task does not change.
6. Basically no volume discounting. Anthropic offered us 2% off for committing to $1M+ per year at API rates.
I manage AI spend for my team at work and I try really hard to keep costs under control but it's absolutely herding cats. Much harder than any other spending I've ever had to manage at work.
In addition, we're seeing people applying AI to more and more use cases, so token growth is very significant. Paired with consumption pricing, it's brought this problem to the forefront very quickly for lots of companies.
It is true that this problem can be mostly managed by the techniques we mention here. Those are actually pretty difficult to set up at scale, so many companies (including us) we only really did this in earnest once we started to see those large cost oscillations.
The main reason we shared this here is to maybe help other companies get infrastructure in place before massive cost swings rather than after.
This is how AWS made its fortune.
Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i have credits anyway". Two years later, someone looks at this and says "hey, this is expensive and I bet we can do everything we need on a machine half the size". But then they think "if i downsize it and that works, i'll get a thumbs up emoji on a slack thread. If i downsize it and it causes problems, i'll draw the ire of the whole team. I better leave it alone." And the truth is... by the time your company hits the end of those credits, you're probably at the point where that savings isn't gonna do much. Or maybe you are out of business.
And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
They’re not saying they regret doing it, or that it was a mistake.
They’re just saying they’ve gained experience and have leveraged the tools to an extent their usage can be optimized.
Pretty standard business or life iteration.
I would bet my entire Polymarket balance ($0) that some military contractors have already asked AIs on the public Internet to design software for them.
1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes
2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model
Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness directly or instead use APIs. Unless Databricks has some agreement in place they are violating the TOS and openly publishing an article about it. Would be interesting if openAi or Anthropic come back and claim for the API usage prices and all the savings go away.
[0]: https://support.claude.com/en/articles/9797531-what-is-the-e...
[1]: https://support.claude.com/en/articles/9266767-what-is-the-t...
Anyone using the enterprise plan are charged the API pricing, however the article is not clear if Databricks is using enterprise plan or not which is why added the following disclaimer
> Unless Databricks has some agreement in place
Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity
I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)
In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.
If my company told me yeah we’ve decided you don’t get Fable or Opus 5 because it’s too pricey, you gotta use GLM whatever, I’d be displeased.
Kudos to databricks, I also find it interesting that such different companies (Stripe, Ramp, Databricks) are all building the exact same internal tools.
I think building companies is going to look more generic in the future because intelligence is an API now.
Have you all attempted finetuning smaller OSS models on your repos for coding?
I think the more interesting lever is the fourth they mention: token efficiency.
> By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include.
I think there’s still lots of low hanging fruit in regards to monitoring and improving agent work. Look at your sessions. Look at how much time and context is being spent on, say, a web search returning dozens of results when one good single-pager doc would’ve been better.
My workflow is very simple:
1. develop requirements for code change
2. take manual notes for implementation, maybe use LLM for some discovery/investigation
3. present notes to frontier LLM
4. develop implementation plan (bulk of work)
5. let LLM rip
6. review diff, manually fixing/refactoring code as necessary, sometimes prompting for revisions
7. get automated LLM review
8. get human review
this reliably produces the work of 2-3 pre-AI senior engineers with a lower bug rate, equivalent performance, robust edge-case consideration, etc.
Does the LLM produce over-engineered solutions? All the time. I stop it from doing that, or manually fix it myself.
Does the LLM always adhere to the best system design? No, not at all. I often have to guide its design into a better, north-star aligned one.
I don't just sit in front of my terminal and say, "Ok Claude, build the app." It is a very iterative process, and not without its potential pitfalls.
But it is very, very productive.
I haven't gotten a chance to test the multi-agent capabilities, but the DeepSeek Flash prices are so low that I probably will soon.
Maintaining the cache is extremely, extremely important, so we're iterating fast but that's a major factor we track in the router's development. Couple things I'd look at:
1. The cache is generally reset after a compaction - this is the best time to make a switch if you want.
2. In many cases, the max duration of a cache is 1h, so if a session is being resumed after a long time, that's also a good time to re-assess the complexity.
We're iterating fast here and learning a lot! Definitely a lot to think about it in this area.
An AI-edited post by the way.
https://www.databricks.com/blog/introducing-omnigent-meta-ha...
https://github.com/omnigent-ai/omnigent
We're calling the layer coordinating harnesses "meta-harness'