Finding Miscompiles for Fun, Not Profit

mNovak•about 20 hours ago

Given the $10k price tag for tokens and high rate of bugs (several per minute) they mention, it'd be very interesting to see this experiment run with cheaper models too.

I wonder if we get to a world where a full repo sweep like this is a default Github action after commit.

1over137•about 18 hours ago

Most C/C++ projects I know don't even run tests with ASan/TSan/UBSan before each commit/merge.

Quarrel•about 17 hours ago

and in the meantime, just a sweep of the committed code (or the to-be-committed code for lots of us) and the code it interacts with, is increasingly catching lots of problems.

rurban•about 12 hours ago

Boy, I told DeepSeek V4 Flash Free to find compiler bugs, esp. with the gcc torture test suite, and it did find plenty. For free. To fix them in my compiler it did cost about $40 or so. Cooperate guys just have too much budget in their hands to throw at the rich.

jlebar•about 19 hours ago

Author here; I'm happy to answer questions, take criticism, etc etc.

ebiederm•about 17 hours ago

Thank you for posting this.

I had heard LLMs were finding a lot of bugs very quickly and now I can see what that looks like from a user perspective.

eqvinox•about 17 hours ago

> Codex and I collaboratively wrote a fuzzer.

Why are you using phrasing that equates AI and humans? You used Codex to write a fuzzer. It didn't decide to join you.

derdi•about 13 hours ago

Why are you using phrasing that equates AI and humans? Codex isn't in a position to decide whether to do work.

Traubenfuchs•about 9 hours ago

I wonder how much damage all those countless bugs caused in real life.

Does MRI low level code produce wrong images? Do some kind of unexpected http connection quirks happen? Does (LL)M inference produce randomly wrong and non reproduceable output? Graphical artifacts in video games? Application crashes that happen once every billionth request? Security vulnerabilities? Race conditions?

Finding Miscompiles for Fun, Not Profit

⚡ Community Insights

Discussion (9 Comments)Read Original on HackerNews