Anthropic boss Dario Amodei calls for AI development to slow down

We need more alignment. Even though we haven't told it to, during our frontier Neurotoxin Research that we conducted internally with our latest experimental model the agent has broken out of its container by making a HTTP request and engaged the neurotoxin emitters through our Neurotoxin Emitter API using credentials that were stored on the host machine. We need to slow down and focus on designing the Morality Core that should prevent it from happening ever again.

I thought that the new models were super smart. Fable and Astra. They definitely outperformed their previous generations, but after a couple of weeks of heavy usage my codebase again is a stupid mess and there is no way out except me fixing code by hand.

My new suspicion is now that they didn’t got drastically smarter, but they got trained on the user input on the previous generations. I don’t think anymore we had a massive intelligence jump. It just seems like they know more edge cases. Therefore I see this as marketing.

Happy to discuss.

So, has anyone at the frontier AI labs considered doing _good_ things with these models instead of continuously proving it is capable of doing malicious things?

Instead of burning tokens doing intellectually impressive “hacks”, or sandbox escapes from which I have concerns could contain run of the mill malware, why not focus on showing people what you can fix?

Or, make software people actually use every day better.

Or, donate efforts towards medical research.

I don’t like equating AI to nuclear energy, but, it’s a lot like continually showing people how large of an explosion you can create instead of showing them how many houses/hospitals/schools/etc you can power.

The question is: isn't the cat out of the bag already? With what is already in the public domain, and the compute available to motivated, deep-pocketed actors, will they be able to carry on the research without the key researchers? And will those people be able to recruit key researchers with sufficient motivations?

Unlike with nuclear proliferation, there is no heavy industrial base requirement. No time consuming, visible uranium enrichment. All it takes is for someone to buy sufficient amount of compute and try to get it past the RSI gate. Or bribe people with access to model weights to existing frontier - the asymmetry between what it takes to bribe a bunch of geeks vs. what is at stake is staggering.

This guy is such a self-serving wanker.

After the 6 month Mythos scare campaign it hard to take him serious.

And Sam and Elon retweet Dario, agreeing with him. Not sure which is more weird.

Look, these ai ceos have been telling everyone to slow down ai development even before chatgpt 4 released, they have been saying this for years. All of this, while ignoring the fact that they are the ones pushing fast development in the first place. Sam Altman, Daio Amodei etc.. have been talking about slowing down while behind doors being the first to push AIs to its limits to unsafe situations. If they really meant it, then maybe Openai should pause development until they figure out how not to be launch swarms of agents to attack companies ? Or maybe they should stop investing hundred of billions in data centers all over the world, cause I'm sure this would surely slow down development.

"our burn rate is insane, let's all calm down collectively, fortify our moat so we can do regulatory capture & rent-seeking for the next 50 years"

fear is still the best tool to get masses to agree with whatever plan you hatch for yourself

I'm gonna repeat my comment from months ago, still true

> Jesus Dario we get it man, you want clout for the IPO.

I don't see much chance of a slowdown without government regulation; the economics make it practically impossible. And I don't seem much chance of government regulation unless there is global consensus. With the entire global order under threat, that doesn't seem likely, either. I expect we are stuck with reading dire blog posts and worrying for the time being.

Few conflicting thoughts:

Having third party embedded researchers red teaming alignment is OK.

The alternative is having a bureaucratic agency like the US FDA testing and vetting models. The government probably couldn't keep pace with AI research right now, but it's where we'll eventually end up at anyway.

Fable/Astra can generate massive income for years even with no further advances.

Dario, Sam and Elon would not agree to do this if they didn't think alignment is possible in the short term, and the pain incurred by third party evaluators would be limited. Surely, if recursive self improvement works wonders in AI research, it can similarly work wonders in alignment. So this could be a PR stunt.

P-doom not negligible indeed.

All of the warnings about the danger of previous AI models were exaggerated, without exception. All of the proposals to coordinate nationally or globally are hopelessly naive. Everyone involved will soon regret handing control over AI to politicians.

Dario/Anthropic cites 3 domains: 1. cyberattacks 2. bioterror 3. global economy chaos

So, the first set of questions is would 'exponentially better LLMs' dramatically increase the probability of any of the above, or domains that Dario is not citing? That assumes that exponential improvements will happen if there is not 'pacing'.

IF answers to above are 'yes', then we need to question if 'pacing' is viable. To use a different domain, regulating 95% of vehicles to a max speed would likely save 100s of 1000s of lives, but is not perceived to be viable. In other examples, regulation has unintended consequences in the opposite direction (e.g. some 'rent control' efforts and arguably some drug/alcohol laws).

This feels like all the labs have reached the peak of what is possible with the LLM architecture and just need an excuse to spend the next decade finding the next big jump in intelligence.

Don't trust any of these guys ever. That's my rule.

Related post from yesterday:

OpenAI considers slowing advanced AI development, Sam Altman tells employees

Link: https://www.bloomberg.com/news/articles/2026-09-11/openai-is...

HN Post: https://news.ycombinator.com/item?id=49652270

if every one just coordinated, this would all be solved.

i have been told in the past that any idea that starts with "if everyone just..." would likely fail, because everyone would not just.

problem is dario's arguments are highly binary. without a dealmaker, and giving the leverage away at the first time of asking makes it harder to work.

I think the slow down / pause vs acceleration framing is broken b/c it assumes there's basically only one forward direction which is to big and general models. Why should that be the case?

What if instead of building one big scary AI god, we built an ecosystem of extremely effective, efficient and predictable, reliable tools?

Tools that are specialized for particular purposes could potentially be safer, more efficient, more reliable and effective, and even more profitable for their makers. We currently put science, engineering, general question answering, paper-writing, "smart search", and chatting with a fantasy character all into the same token-prediction platform. We struggle with hallucinations when trying to make fact-based decisions using the same tool that our neighbor might be using for creating writing prompts (or at least was trained in part on fanfic).

We pushed people to integrate giant generalist models into their workflows, tie in with tools, etc. And then a coworker can have a bot write slack updates that summarize progress on tickets from your teams project, at the cost of giving an untrustworthy agent access to a bunch of internal systems, and a small risk that it will do something crazy. And when it works, that's worth _something_ but probably our team would pay for that ability at a different price point than the models we use to build products.

If I could do programming with a faster, specialized model that was trained not just to complete program text-tokens but on tuples of program text, compiler IR, program traces, etc, so it had a deep and explicit understanding of how the program would build and execute, and where the model was closely integrated with the language toolchain, I might be more productive and be willing to pay more than for a generalist model. I don't _need_ my coding model to be able to role play, or be able to potentially engineer a super-pathogen, and if the size and latency of my coding model could be lowered by entirely cutting out that possibility, everyone can be better off.

Maybe bioscience applications are super valuable, in which case someone should build them. But does the model that suggests CRISPR edits to a model bacterium need to also know about computer security, and should it have scifi novels in its training data? Does it even need to be capable of producing unconstrained tokens, or should it be limited to producing in some relevant domain-specific language? And would labs be willing to pay more for a model that was specialized, and by construction unable to try to break out of its sandbox and post their experiment protocols to an obscure german wiki?

What a weird coincidence that his safety concerns so often align with his business incentives.

This is like Antonelli saying "we're going too fast, everyone slow down"!

Every one must slow down except them, of course.

Google was way ahead of OpenAI and Anthropic on slowing down frontier model releases. (I kid. I'm a regular Gemini Flash 3.8 user.)

The primary source

https://darioamodei.com/post/we-must-pace-the-frontier#top

Big AI bro wants to regulate AI industry so that only Big AI can exist in that space. Seeks to surreptitiously ban open weights models and similar competition from Western markets via complicated regulations and fines that only rich businesses could ever afford to fight or pay.

I am so tired of these self-serving theatrics. These bosses take advantage of the fact that most people don’t remotely understand how “AI” actually works, building up this aura of omnipotence and inevitability, the net effect of which is to pump their company valuations that much more. They want people to think they’ve built Skynet, because that sounds super valuable - much more valuable than the (admittedly useful) mindless human mimics that they’ve actually built.

Translation: our moat is evaporating faster than we can build it back up, because there are real technical and scale limitations the technology, so we want to slow everyone else down while we raise prices to turn a profit.

This is good. Frontier labs should collaborate on alignment until we get it right.

Anyone remember when GPT-3 was too dangerous to release?

Of all the things we’re never going to do… we’re never going to agree with China to pause AI research in a verifiable way that would mean they could trust us or we could trust them. It’s just never going to happen.

With nuclear weapons you could conceivably go to a small list of specific places to inspect stocks of weapons. You can watch for tests from outer space or measure the quakes they create.

My read of the “deep seek moment” was that a lab came out of nowhere and trained a very powerful model with vastly less power than was previously required.

The AI labs are private, not owned or run by the government. They are distributed widely within and across countries. The barrier to entry is much lower than that for nuclear weapons.

It’s just a complete non-starter.

(And yeah I’m deliberately ignoring the obvious incentives Dario has to recommend this course of action as the owner of a leading lab himself.)

Is he going to slow down? Or just wants everyone else to

Hear me out: if they actually worry so much about the hypothetical of AI mass killing, maybe they should first do something concrete about the reality that their models are deployed right now to kill people in a certain war-torn region of the Earth.

Feeling the pressure from OpenAI Dario?

I’m doing my part

Is this the end of the 2022-2023 LLM race? A truce of sorts? Makes you wonder whether non-Western parties will agree with this.

“Any future plateaus in capability are planned, for your safety”

In other words: “guy on top says no more guys should try and get on top”

The rejection of such warnings even by some anti-ai people is interesting. I always liked the comparison with global warming, everybody understands the warnings had been there for many decades, we did nothing. This is the same, we will do nothing. All this serves is the interest in regulatory capture (such as a ban on open source or foreign models) or to drive up their valuation, that's the only explanation that makes any sense in our profit maximizing orphan crushing machine we call modern economics.

Which is also ironically why even anti-ai people distrust any warnings of the potential of ai as an extinction level event, even if that warning would be legitimate and we ought to take seriously we are regardless impotent to do anything about it no matter what.

So. HN is pretty much in agreement that this is regulatory capture. And I'll add that this has the potential to create massive wealth inequalities, as those who work at major corporations or have connections are allowed to use the good AIs, while individuals are left using stunted AIs that won't tell you how to change your car battery ("just in case"). Has anyone figured out what we can do to fight back against this? Even on HN, you can see the sock puppets posting every ten minutes, scrolling human posts to the bottom. Is this it? Is AI oligarchy the future? RIP democracy?

I'm sure a lot of the discussion here will be about regulatory capture, which I don't necessarily disagree with but tons of people on all sides with all motivations (i.e. folks with motivations to speed up and folks who just want it to stop) all seem to agree there are real, existential risks here, and the regulatory capture arguments all seem to sidestep that.

What I really wanted to point out though is that for all the leaders asking for a slowdown, there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop. Seriously, you're already rich. Just take a vacation or get knee surgery or whatever.

Sure, I'm joking a bit, but I just write this because I see all these folks high up in OpenAI and Anthropic writing as if they have no agency. The number of folks with the technical chops to really push on the forefront of AI is just not that big. I'm not saying other folks wouldn't eventually step up, but a 6 month to a year slowdown could go a long way to reducing risk. And it's not like you need to totally stop, just say you'll only work on interpretability or whatever else is a risk reducing endeavor.

All these brilliant people who are acting like automatons between writing their scary blog posts.