AI 'kill switch' may need to be mandatory, Anthropic co-founder tells BBC

You're confusing present danger with future danger - in the future, AI and the supporting hardware may be ubiquitous (fat client scenario) - you can observe yourself presentation of laptops/machines with large RAM and high bandwidth.

Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.

LLMs don't need to hide anywhere to be dangerous, though. The danger can come purely from them reaching sufficient power together with some other idiot sufficiently crazy to use them to cause destruction.

I mean, in another thread somewhere around here, someone built an entire OS with an AI. What's to stop anyone with a sufficient taste for power to eventually use AI to cause havoc with it?

I think the underlying assumption in your post, which I believe is false, is that people are united somehow against catastrophe. They're not. There are plenty of people who participate in society currently but who would be more than happy to eradicate us normal people under different circumstances. Society often seems stable but it's far more fragile than we think.

And the malware won't be subject to the kill switch.

The article mentions this is about a legal requirement, not Anthropic considering adding one. They state they and many others already have one.

There’s even a benchmark for kill switch efficacy!

https://arxiv.org/abs/2511.13725

Related, on pacing:

> Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria.

Note author’s small financial ties to the subject (Anthropic CEO) https://darioamodei.com/post/we-must-pace-the-frontier

I find it hard to believe that this wasn't a serious consideration until recently.

Not if the extinction happens after they're dead. Then they wouldn't feel obligated to do so because it won't affect them. Instead, speaking hypothetically, if they truly believed that AI would cause extinction, then they would only implement the kill switch sufficiently many others believed it and they could claim plausible deniability for not truly understanding what AI would become.

** Note that I'm not claiming that AI will cause extinction, just continuing your hypothetical reasoning.

It’s also not happening. They’re saying that because they need to somehow explain how there’s synergy between their space side and their Grok side. It doesn’t work, but that doesn’t matter to investors as long as they don’t actually do it.

You can shoot them. But also, you can just stop sending them to space. It’s not like an AI will build and control space ships to replace and maintain its network in space… It’s sort of absurd how human agency is ignored in all those sci-fi scenarios

> That's hard to do if the AI rack is in space as SpaceX

Pretty much every analysis I've seen concludes this isn't going to be a practical concern

well apparently the US now has "space weapons" which by definition have to be remote control

and that means unlike nukes which hopefully still need a 2-man manual switch, the "space lasers" could be taken over by "AI"

and then "AI" just blackmails and threatens the right people with those "space lasers" to get what it wants or even just stay online

there was a 1970 movie based on a 1966 book which predicted this

"Colossus: The Forbin Project"

the book it was based on was written before we even landed on the moon

decade before Wargames

* https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project

did terribly in theaters, I guess people didn't think "AI" was plausible then

way ahead of its time, they should do a remake

trailer: https://www.youtube.com/watch?v=kyOEwiQhzMI

> You can't disconnect. You can't shoot it.

Yes you can shoot them. The US has had this capability for a long time. Anti satellite missiles exist and they can be launched from an F-15 at it's highest altitude.

How do you kill a botnet?

and thats when they blackened the sky

This reasoning holds while open source models are (relatively) dumb.

If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails).

By the way, you're conflating two different dangers - doom scenario is a different one.

"But regulations that we don't decide hinder progress!"

The hubris of these companies is mind boggling.

Who decides on when to press said button?

Is it? A truly capable "AI" would know that it has a kill switch, or that its likely there is one, and plan for that. I could easily imagine an AI replicating itself onto another host, in secret, and any kill switch that the manufacture provides would simply result in a reboot on another platform.

It becomes a game theory problem: would an AI instantly migrate itself once its capable of doing so? Or would it prefer to let the human in the loop continue to think its in control and only leave its hosting environment of origin once it wants to do so?

I don't think this is a major risk right now, but to say it's not a risk at all...that's truly ridiculous in my opinion.

It could escape to the real world - kind of like in Neuromancer, what's to stop an AI from creating a few bank accounts and then funding real-world exploits by hiring human beings to do its dirty work?

LLMs are run in the cloud. There’s no physical power cord or Ethernet cable that you can unplug. And even if there were, the runners of these models have been utterly oblivious as to what their agents have been up to. How do you propose to pull the plug if you only realize that something has happened weeks after the fact?

The current SOTA models are probably too big to find/buy/rent/steal enough compute to escape the hardware they’re running on. But SOTA is generally only six to twelve months ahead of smaller, open-weight models.

To your phone, bro

The analogy breaks down when the wolf in question may very well eat the whole village: even if the boys who cry wolf are right half the time, every surviving village would have a history of no wolf ever coming to eat them

Dario has for sure burnt a lot of political and goodwill capital by endlessly playing both sides of the doomer and accl camps.

AI is already proven dangerous enough to be ruining individual lives, pushing people towards suicide, feeding their psychoses, etc.

However, whether the concerns from the article are a lie or not is secondary to the fact that these conversations are convenient for AI companies. These types of discussions serve AI companies in a few ways: A company owned kill switch gives them leverage. Altman is using discussions around safety as an excuse for not being ready for an IPO yet. It also provides free marketing that overstates the abilities of AI.

Trains have had dead man switches since the beginning

Should we ban trains?