Towards a Theory of Bugs: The Ruliology of the Unexpected

In this framing, clean coding abstractions gain a purpose beyond helping humans understand the code and actually help enforce that the end result is reasonably bug free. Building code out of understandable pieces can ensure that the whole thing is reasonably understandable.

While the article is focused on “low level” bugs, the key insight is that human understanding has limits, and bugs show up when the human is overconfident in their understanding, and unwilling to verify the code by running it on a sample of test cases. This does seem to help explain why “high level” bugs occur, where the reality somehow doesn’t match the assumption. Maybe a dependency is on the wrong version, a global variable isn’t initialized in certain cases, etc.

In a world of LLMs producing code that isn’t always understandable to humans, it does raise the question of how to enforce that AI code is bug free, and maybe forcing coding agents to stick to human abstractions is the missing piece.

Am I wrong to conclude that the statement by Google's VP of Security Engineering that "we simply must eliminate every software vulnerability on Earth" (before AI Agents find them) is not only stunningly ambitious, but doomed from the start?

https://youtu.be/B_7RpP90rUk (at 3:00)