HI version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
74% Positive
Analyzed from 2560 words in the discussion.
Trending Topics
#review#code#agent#cost#run#same#reviews#cloudflare#more#agents
Discussion (55 Comments)Read Original on HackerNews
"Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post.
> Trivial reviews (typo fixes, small doc changes) cost 20 cents on average
They did around 25,000 of these runs (about 20% of total). So CF spent $5k in the period making language models run through PRs which were <10 lines long. I get that CF engineers are paid well, but the labour cost of having an intern/entry level engineer spend ~30-60s looking through these is likely close to $0.20, and that engineer builds some human-capital while they're at it.
Did you do the math? Your estimate feels way off. First, I doubt an intern would process one PR in 30s. Maybe 2-3 minutes, to read 10 lines carefully looking for typos and indentation mistakes. We pay interns close to $100K these days (in a company like CloudFlare), so that's ~80c/minute. My estimate is therefore closer to $1.6 per PR. About 10X.
You are correct that there is a residual value with the intern, over time they would start learning (a little bit) about the code base.
Are you sure it's going to make sense to pay someone to do that moving forward?
I don't think it's worth it now, by the way.
It's definitely not going to be worth it in the near future.
Can we even blink for $0.002? What happens when the next 90% increase in efficiency happens??
As far as I can see, token costs have been steadily increasing over the past few months, so I’m not sure that buying the hype that another 90% cost reduction is just around the corner is warranted.
Opus cut its token pricing by 66% 6 months ago and it had previously been that higher price consistently for a year and a half (since that model launch).
GPT’s latest model is harder to track since it’s not named, but it’s historically inline with its history.
Not to mention what’s happening with other models like DeepSeek, GLM, and Kimi.
It seems to me the bigger change in costs is based on token appetite. People are discovering agentic capabilities are stronger than they used to be and use cases have broadened because of that. They’ll eventually discover too that these alternative models offer 95% of the intelligence at 20% of the price.
See Chart 13 here: https://www.rdworldonline.com/ais-great-compression-20-chart...
See here: https://epoch.ai/data-insights/llm-inference-price-trends
LLMs are so comically inefficient compared to the human brain that it is pretty easy to imagine this trend continuing for several more 90% drops.
If LeCun's JEPA or GRAM turn out to be a thing, we could see a 3-4 order of magnitude drop in a single release cycle / generation.
Keep in mind that performance per watt on the hardware side - at the same time - is still doubling every ~24 months - and this doesn't factor that in.
> We also extract a shared context file (shared-mr-context.txt) from the coordinator's prompt and write it to disk. Sub-reviewers read this file instead of having the full MR context duplicated in each of their prompts. This was a deliberate decision, as duplicating even a moderately-sized MR context across seven concurrent reviewers would multiply our token costs by 7x.
No, it would not, because neither is the prompt of the subagent 100% of its token usage, nor will the "shared-mr-context.txt" which is then being read have a size of zero compared to the creation of this shared context.
> You don't need seven concurrent AI agents burning Opus-tier tokens to review a one-line typo fix in a README.
Yeah, well you wouldn't have anyways. Earlier in the post it says that Opus is "exclusively for the Review Coordinator".
> Our cache hit rate sits at 85.7%, which saves us an estimated five figures compared to what we would pay at full input token pricing. This is partially thanks to the shared context file optimisation — sub-reviewers reading from a cached context file rather than each getting their own copy of the MR metadata, but also by using the exact same base prompts across all runs, across all merge requests.
I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.
It’s exclusively been the other way around where having a smaller number of larger squished commits (post merge) that’s made the project be more maintainable.
e.g. I don't squash, I prefer to see full history, not redacted one.
There is no universal standard which IDEs support for code review results (there is SARIF, but it is not supported that widely). A review result on a web page with comments from humans, is valuable.
Pre-commit hooks should be fast, as it's something you'd do (normally) a few dozen times a year. I don't believe sending a review job to a remote agent is fast, nor will waiting for a review to finish a commit be good for anyone.
CI on the other hand can be slower and runs async, it's fire and forget so you can switch tasks.
If noise is an issue, one possible solution is to create a merge request, have the tools review it, make the fixes, rewrite history like you did it perfectly the first time ("fix" commits are noise), then create a new one for human review.
we are finding lots of value in self review. its the “imagine you are doing a synchronous paired review with someone - anything that is difficult to explain, has a code smell, doesnt fit the architecture of the system around you, write a comment.” then at the end, agents do a good job of looping over PR comments.
the second thing would be a guided, educational code review tool - there are a few attempts at this, but nothing that has a good enough interface to actually stick. organize hunks by semantic importance, spend some tokens exploring the surrounding systems, showing how new code, public apis and data model flow with the existing design, and allow a human to traverse larger PRs more quickly.
thank you to cloudflare for publishing this.
> agents do a good job of looping over PR comments
This is the easy part. Most harnesses enable some sort of integration now, so you can actually create a smooth local experience around this as well - better code before it ships to more costly review or bloats PR threads.
> guided, educational code review tool
This is a bit tougher, and I find the main harness chat tends to work best. I learn better when I'm more engaged and aware of what I'm asking. It's easy to stick a code tour type of thing on a screen. It's hard to really nail the right attention and learning mechanism around it.
I think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.
The ROI here is so high that I don't mind using the strongest model available for the actual code review. I don't trust Sonnet and such. Just let Opus or GPT 5.5 do the whole thing and pay a bit more for less complexity.
I have about 15 or so subagents doing reviews from different perspectives (or providing some additional value, like finding agents.md files, doing confidence ranking, describing images attached to the PR, that get validated later on with Jira issue description).
I used it since about November, with large scale popularity in my company reaching in April - all that on a 300 premium requests (because they allowed starting subagents, and there was no limit how long a single request can last) - so it would cost something like $5000 and $8000 for April and May if it was API pricing. I had similar cost per review (about $0.90) with Opus 4.6 and help from Sonnet and Haiku for simpler tasks. It did about 4000 reviews during the last 2 months.
And starting in June, it will be dead because it will be API pricing and for $30 (or $19 since September) it will do just few reviews.
A fun project.
would love to look into it if any part of it is open source
Most people I know had the experience that signal to noise was way off, regardless of scale. So it was a burden rather than a help. Code review by AI ended up being a skill before creating the PR so the dev owning the PR addressed everything before the team got bogged down with it in review
I had the same problem in my recursive agent harness. It would always come back, but it could sometimes take up to 10 minutes depending. I fixed this by adding a required "purpose" argument to every tool and call/return event. As the recursive evaluation proceeds, every single thing that happens streams incremental purpose text to the user's browser (also using the magic of JSONL for this). The incremental progress events contain the purpose and a detail section (tool arg JSON) that the user can expand/collapse.
They apparently think they need to cash in on AI by serving models and at the same time blocking scrapers. So they need to fuel the hype by pretending to use it.
This shows how the US economy is fundamentally broken: companies that provide a useful service (in theory, if you discount SSL MITM and turnstile gatekeeping) struggle, quasi-religious scams like OpenAI and Anthropic get funded by mentally ill Boomers and Gen-Xers.