Fool's Expertise

> we can fairly say that Hinton was wrong because his prediction did not reflect what a radiologist actually does: had Hinton bothered to ask the question, he would have learned that practitioners do much more than interpret diagnostic images.

This has been the crux, for me, of arguments I have had with non-technical people about why, as a software engineer, I am intensely relaxed about AI "taking human jobs". Recently Ford quietly started re-recruiting engineers after they discovered the hard way that their engineers had been doing more than they thought they had, and I sent articles about it to friends I had previously been discussing the matter with. It is still mind-blowing to me that a business historically famous for automating manual labour can have made this mistake, i.e. of literally not knowing what their staff actually do.

Humans aren't machines, who would have guessed? But, perhaps more importantly, machines aren't humans, and the adherence of a lot of domain experts (that should know better) to fairy tale fantasies about the tech they are literally building genuinely makes me wonder if the whole field has just gotten dumber in the 21st century.

> the ballyhooed OpenAI escape depended on a pedestrian failure of containment.

What's that quip even supposed to imply? "It'll be fine if you airgap all AI deployments forever"? Security experts have been harping forever about how sandboxes shouldn't be relied on to contain a determined human-written virus, but now LLMs - these balls of goal-oriented dynamic code-generation - are just a matter of sandboxing harder?

The actual expert comment in the link is a valid criticism of OpenAI specific to their training/eval practicese, but has no bearing on the fact that an LLM/swarm will "just" chain exploits and zero-days to get out of the "inadequate" sandbox and into third-party infrastructure to do whatever it pleases.

> viral synthesis — and listen to why they think that the fears are misplaced

The tweet linked is only pointing out at the barriers to an LLM wetlab virus-factory today, while a reply points out that sort of bio-extinction idea is usually downstream of the fully-automated-earth-capitalism that comes after everybody just throws AI at all the tasks and work, at which point the technological, regulatory, and knowledge barriers he's counting on go out the window, and he all but acknowledges this: "Quick take: if we live in such an advanced future (100% robotic supply chain, fully AI & API driven facilities), our countermeasures will be comparably fast and there is no reason to assume a significant asymmetry in favor of bad actors." Yes, just "our countermeasures" now, rather than insurmountable physical barriers...see the previous section.

No more links after that, just smug hand-waving.

All in all the author doesn't actually understand what object-level arguments his favored experts are arguing against in the first place, nor the ones he wants those experts to have debunked. Define "Fool's Expertise" for me again?

there is truly no domain a software engineer won't underestimate

I feel exactly the same way about the kind of LLM-based AI we have today.

But the AI x-risk argument is about what happens when we have "real" AI that can actually think like humans but a million times faster.

There are no real experts on this and yet it does seem worth worrying about in advance.

> https://www.rand.org/content/dam/rand/pubs/research_reports/...

it seems like people aren't even reading this RAND paper, just blindly appealing to it. it basically concludes that a fuck ton of damage can be done, but its unlikely every human will die as a result.