The Normalization of Inexplicable Failures

But it’s pretty clear that most people are not. For whatever reasons (mgmt pressure, trying to get ahead, skill issues, etc) they half ass it, accept the 10% (silent) fail rate and blame the bad outcomes on the AI as if that absolves them. Or, adopt the attitude that 10% fail is fine, and people who say otherwise are being picky, or are anti-ai luddites or whatever. You should accept that things will suck.

I have definitely seen more bafflingly-poor OSS software that just plain doesn't work frequently now than before.

But it's mostly software that wouldn't have existed before because it's trying to do super-niche things. So on the "hobby" side of things, whatever.

But from a "trying to develop software as a business that you want to be a going concern," quality from people who should know better is less tenable than it used to be.

Yes but that's the big thing, now isn't it? These are nice tools, used wisely. But their unwise use, oh boy...

The problem is one needs to be in a situation where the incentive is towards quality rather than speed. But that situation rather rare now - thirty years ago, Microsoft won the office wars with crap that had features. And nothing has fundamentally changed in web development since the LPad crisis.

The problem is those companies whose incentive is to allow bugs where it's the involuntary users who suffer will bite you no matter what quality you make your own software.

The main argument for LLM-driven development is much simpler: "It will get better".

The current state of LLM coding is about a year old. Imagine if we dismissed human coding efforts after a year. Rust, Python2 -> Python3 transition, Python type checking, Windows, C++, … nothing of that was done in a year and emerged in perfection in the first year. Everything takes ages to mature into a usable product. LLM coding is still in the "throw mud at the wall and see what sticks" stage, give it some more years and see how it will develop and what approaches actually work at. For the time being, LLMs are just the most useful development tool in the history of development tools, that's a pretty solid start in such a short time.

There's definitely still AI/LLM integration happening, but is kept out of specific areas of the business.

Not unlike legacy code, where the original developers are long gone and no one knows when that system breaking bug is about to wake up.

Even in the best of cases tho, fully human written and reviewed code will still fail eventually.

So I would say we are not normalizing failures (yet) but rather normalizing legacy.

The point is that it boils down to writing the tests correctly, regardless of who is implementing the actual code. Hand-written code without test coverage has the same problems as AI generated code.

Perhaps designing with assumptions that the entire stack is imperfect is the route to higher quality, along the lines of chaos monkey, which randomly kills processes on production boxes… which ensures resilience is real.

Even without AI our tech stack is so deep it’s hard to imagine every feature at every layer being provably correct.

One can be simultaneously a developer and a user. Distributed systems [0] weren't invented five years ago, after all. ;)

"Github owns this part that we rely on for correct operation and we can do fuckall about it when it fails." is a well-defined ownership model.

[0] ...implying the existence of distinct parts that can be independently developed and independently fail...

Without the evals, how do you know it works magically? I think that’s the entire point of the authors article, people are using AI tools, and trusting the AI own evaluation of its performance, without making any attempt to validate if the AIs confidence actually correlates to anything meaningful.

From what I’ve seen, the result tends to be very exciting demos, followed by incredibly mediocre results when deployed into a real environment. The lack of critical evaluation then provides a double whammy, because it takes longer than it should to realise the system isn’t actually working correctly, and then the team lacks the skills and understanding needed to improve the system they’ve built. You just kinda have to hope that the next model release is going to magically result in better performance for your particular scenario, which won’t be able to measure due to the lack of evals.

These are some of the most annoying messages a piece of software can possibly display, and I hate them with a passion!

It's like this with a lot of commercial software, where designers don't want to scare users with complicated (useful) error messages, so you just get a "whoopsie! Something went wrong".

I'd much rather have an error message that I _don't_ understand over an error message I _can't_ understand.

In the case of cars, it used to be that a generic "Check engine" light showed up on your dash, and you needed an external tool to ask the computer what caused it to light up, but why would that be nessecary in a modern vehicle? You've got a large, high resolution display (or more!) on the CAN bus, why can't it show the full problem so you could take a picture right as it happens and send it to your dealer for instant diagnosis?

But my point - this has been increasing for quite a while and it's something like the evil twin of the "it just works" school of design. Because when the default expectation is "just working", the failure mode is "it just doesn't work". I mean, the default assumption that users are morons has de facto made us all morons since we can only deal with the "interface made for morons" and makes failure into opaque failure.

It's world the developers know "all abstractions are leaky abstraction" but where developers still won't give users (or even downstream developers) tools to deal with the leaks 'cause a simplistic interface matters more.

"When there's no emergency exit, there can't be an emergency, can there?"

So they’re not dunking on that, they’re dunking on the idea that you can just trust Jev confidence scores without actually doing any validation.

Of course you aren't going to use Jev where you can easily make a decision deterministically with old-school programming. It's going to be used for stuff that might have previously been done by a person, e.g. reviewing app updates in app stores, reviewing CVs, labelling bugs, etc. Obviously that can already have inexplicable failures.

If so, it's definitely newish for those of us who started programming twenty+ years ago. Back then waterfall was the way to develop applications. Once it was burned to CDs or DVDs, that was it. We could not afford shipping broken things.

Ever since internet became ubiquitous we moved to on-the-fly patching for installed software and fast deployment cycles on cloud.

That said, the lack of forethought in design and backward compatibility is nowadays definitely on the rise and occurs likely on an order of magnitude more often.