Google fixed more Chrome bugs in June than over the past two years, thanks to AI

The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.

Which model/agent/harness tool did you use? I've found what you describe was my exact experience some ~6-8 months ago, but since about a month or so the game has completely changed. Using 5.6 Sol with highest reasoning setting in Codex or Fable in Code, the models come up with a list of possible improvements from static analysis (ranked by complexity/benefit), write and run their own custom profilers and deliver significant performance improvements with barely any input needed from my side. So this is no longer a model issue, it's a user toolchain issue.

I've had a completely different experience. I usually know the problematic part and then ask the AI to optimize that exact part with benchmarks, telling it to try various different approaches, and it very often gains massive performance increases on various tasks, with me of course steering it and giving it pointers every now and then.

I was able to optimize a physical water simulation that was almost unrunnable on browsers to a buttery smooth 60fps version.

Ask it to create a small sample to reproduce the issue in isolation. If it can't reproduce the performance changes then it drops it.

That might work better. I haven't tried it with SQL but it worked with a few complex UI issues I had. It identified the actual issue after a few false starts.

I did a lot of SQL optimisation with claude code over the last year, what made it work for me was making sure I verify everything with explain analyze. The whole loop was not fully “agentic” because of this but it was still faster and better than when done entirely manually. Claude also suggested plenty of small optimisations I would otherwise not do.

The mistake there is to point it at code to figure out performance optimizations.

The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.

The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.

Yep, as a performance specialist I've had the same outcomes - it'll find "smoking guns" galore which will sound plausible and be completely wrong.

But that's ok - I still get a lot of value of the tool elsewhere (e.g. writing data analysis scripts, exploring code, breaking down bug artifacts...), but it doesn't stop the tiring BS from other people going "why not just AI it?" (especially from managers).

I usually let them try to "just AI it" and most of them learn quickly that it's not that simple. (The others stay deranged and are making my job miserable.)

I've noticed the same thing.

The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.

Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.

I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.

As others have said, these were uninformed guesses which LLMs seem still to be willing to hand out deliberately. If you provide more information (your schemas, the distribution of values in your database tables, the EXPLAIN ANALYZE outputs, DB configuration etc.) it will do a much better job.

Then you're not grounding it to reality properly. LLMs get bad pretty quickly when you just tell them to pull the answer out of thin air. Pin them to reality with actual performance tests to run to test theories and the results will be much better.

Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivial improvements.

> I've recently been using AI

That's uninteresting as long as you don't specify the model you used. For example, Mythos was far better at finding security bugs than previous models.

Google's entire modus operandi has been "automate everything" for decades. They've been doing this with fuzzers, with project zero and so on. Adding LLMs on top is a very obvious next step. And LLMs improving and finding more bugs also follows. Then improve the harness and the dev tools, to better use the LLMs. And then everything together end-to-end to find-triage-fix-confirm. Your LLMs are as good as the loop they run in, and the loop is as good as the verifier. Seems a reasonable enough dynamic without (or despite, depends how cynical you wanna be) the need for managers to show number goes up on some chart.

I would guess they have been confronted with an initial flood of newly discovered bugs whenever they added a new analysis tool or approach (e.g. automated static analysis, fuzzing, ...). Once working through that initial flurry of newly discovered bugs one would assume that the frequency goes down again.

More likely just getting ahead of the AI attacks before they hit. The threat risk increase caused by AI has gone off the chart.

1. Our backlog of bugs gets processed quicker because instead of staring at the code for 10 minutes fiuring out what's happening, there's a tool that can reason about it quicker.

2. Code reviews and security reviews happen quicker and produce more findings.

I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.

Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.

Exactly, also it lacks a lot of context as to why they did that.

Possible (probable?) scenario:

- Marketing: "we found and fixed lots of bugs thanks to AI"

- Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually

You're not supposed to ask sane questions here

> The post has counts for everything that went right and nothing for what could go wrong.

That's AI for you.

At Amazon we have many forums to share our AI wins, but none to share AI failures or disappoinments.

No wonder execs make bad decisions regarding AI, they only hear completely one-sided stories.

And how many were introduced by AI?

AI has skills no individual can have. That part is true.

But AI is not smarter than you, AI is as smart as the user who uses it. It often makes wrong decisions unless user corrects it.

Yes. AI is a tool, it’s supposed to be used a certain way, anything else is a misunderstanding og what AI is. You have to aim it in the direction you want it to go, not expect it to solve all your problems magically

> A lot of people here seem to be living in a different universe than me

I feel this way on this topic too.

> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.

The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.

My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.

Did you think for a second that maybe other people work with other languages, domains, tools and work approaches than you?

It's so bizarre that noone realizes that there's a bit of a difference between banging together JavaScript into websites, writing firmware for widgets, herding servers or fixing medical devices?

Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing; they did not slacken their code review and testing practices with the advent of LLM code generators. And I doubt they do much development on the affected areas at the moment, they mention a 13 year old issue for example.

But it's an open source project, you can go and figure out whether your assertion is correct.

That is a stat you invented out of thin air, based upon nothing but pejorative hope, against all logic or rational consideration of the timeline or the scale of the project, and you declare it the "elephant in the room"?

HN is so weird on the topic of AI, and so many desperately are trying to contrive a reality. Odd stuff.

won't forgive them either for making it harder to filter out ads

I'm not sure why you're asking here instead of opening the ticket and having a look at the current status.

> We really need an alternative to this greedy and evil corporations de-facto controlling a huge portion of the modern www stack.

We do / have? Multiple even. Of course, the vast majority of consumers doesn't care so they don't get as much traction.

If only there was alternative browsers...

I don't know about Chromium specifically, but in general Google requires 2 humans to have looked at every change (normally one is the author, but not necessarily for bot-authored changes).

Did you go and look at their open source repositories and open code review tools instead of fearmongering on HN? The answer can be found.

I’ve heard from people inside Google that they’re prohibited from using anything but their own models. AlbertaTech on YT, ex-Google working on YT said in a video that right before she quit someone had threatened to quit if they couldn’t get access to Claude Code

The Gemini (and Claude / Codex) that we have, is much different to their internal models / harnesses.....