P(doom)

Just the fact that Musk is in the conversation, and he's one of the most hated people on the planet, should give anyone pause as to whether we want these folk in charge of our future to any degree at all.

It could be in worse hands though. Sama may be no saint, but better in his hands than Aum Shinrikyo fanatics.

The whole world is in the wrong hands. Having AI in charge can represent hope actually.

There's nothing about MAD that requires few actors. This is a just-so story to justify the US and other major powers' monopoly on the weapon, ensuring smaller nations have no deterrence against us and other global nuisances.

Everyone agreeing on something is not even possible on small local scales, just imagine the situation on a global scale.

It's not a challenge at all because it's not possible, it's just empty rhetoric to push their own agenda.

>Everyone needs to agree to stop. That is the challenge.

These kinds of situations are incredibly common and where the government stepping in is the solution, but we were cursed to encounter this particular challenge with the most venal administration in history at the helm.

I posted this comment on the last doom post:

The early batch of Anthropic employees were mostly rationalist-adjacent AI safety folk that were almost uniformly claiming P_DOOM > .10 three years ago, so I believe them to be earnest.

It's very interesting to me that besides the other small safety labs that don't actually produce frontier models, Anthropic manages to keep such a good reputation within that subculture compared to OpenAI. Despite having as crazy internal politics as OpenAI, they have converged quite a bit from the original vision of safety first through Darwinistic pressures.

At least, it seems this way from the outside. I'm curious if the view from the inside is that different.

edit: to be clear, my reading as an outsider is that Anthropic is seen as relatively better in the AI safety community, but has definitely dropped in absolute reputation too. This recent thread and the references show some of that: https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-p...

The Manhattan Project succeeded, so clearly some people are ok working on this sort of thing. And with similar justifications: if we don't build the potentially world-ending bomb, someone else might beat us to it.

If you believe you working will mean a lower probability than you quitting.

The line of thought goes like this:

- p(doom) is 1

- but if we build it, it's only 0.25

- if it realizes, we've got a front row seat

It's selection pressure.

Both OpenAI (back in 2015) and Anthropic (much later, in 2021) were founded by people who could foresee the concept of AI x-risks and wanted to work on preventing it. For OpenAI's founding, the idea was that it's much safer if AGI is achieved by a nonprofit explicitly dedicated to humanity than if it's done by a profit-driven company. For Anthropic's, it was that OpenAI seems to be going insane and it'd be better if a more safety-conscious company was competing. I'd say in hindsight, the former motivation was reasonable and the latter one was... probably bad, but not obviously so - it doesn't seem even in hindsight that it was inevitable that Anthropic's existence would only drive competition and cause them both to race to AGI. That said, even at founding time, there was immediately a lot of selection - quite a few people who cared about AI safety simply wouldn't agree to work at a capabilities company, even given an elaborate argument why this is a good idea, so those were selected against.

And then, of course, many years passed, OpenAI was stolen by Sam Altman and turned into a for-profit, multiple researchers left OpenAI realising that they're only helping build misaligned AI faster, multiple researchers left Anthropic realising that they, too, are only helping build misaligned AI faster, and here we are in 2026.

There's lots of AI researchers these days, so you can select for conformance very heavily and still have a fully-staffed company. The people who are still at these companies are outliers in various ways. They either manage to dismiss the importance AI risks (for example, by adopting some sort of belief in the vein of "there's no use worrying about AI wiping out humanity - it'd just be a successor species, like we were to apes, which is good"*), or they still think that leaving will not make things go better (Dario Amodei is pretty clearly in this camp, and has always been), or they weight the risk of extinction against the potential benefit of an post-singularity utopia and consider it a good bet (not realizing that the alternative isn't giving up on a post-singularity utopia, but getting it a few decades later and without the risk), or they just manage not to think of the contradictions, which humans are of course very good at.

* If that sounds like a strawman, see minute 15 and on of this Richard Sutton presentation: https://twitter.com/RichardSSutton/status/189808248125100892...

There must have been thousands of people who built or did work related to nuclear weapons in the Cold War, people are just really good at rationalising away probabilistic dangers with unclear consequences, especially when there’s a corresponding upside. The same is true for climate change, drug addiction, health problems etc.

I also think some of them have become delusional and convinced themselves that AI is going to create some kind of transhumanist utopia, even Dario Armodei leans in this direction from time to time. I imagine the people in these labs spend much of their day talking to sycophantic AI models that will encourage their delusional ideas.

Do you genuinely believe people always act logically? I don't know a single person that does. Don't underestimate the human capability to hold several completely contradictory beliefs with zero self-reflection. There's physicists who believe in god, for fuck's sake.

> How can you truly believe this and be ok with it?

Are you saying they don't truly believe this, or that they aren't OK with it?

My personal opinion, this doom-mongering about LLMs, is part of marketing campaign to exaggerate LLM capabilities. (Play being: Sociopaths with money will buy or invest if they believe LLMs are powerful things)

>It’s absurd that openai managed to spin their cybersecurity failures, where their own security is not good enough to make sure their own product doesn’t go haywire, as a marketing success.

I don't think OpenAI spun it that way. I think a bunch of internet commenters assumed it was a stunt because they're in denial about the real risk of rogue AI.

Are these other labs in the room with us now?

No, seriously, I'm all for a multipolar world here, but he's right that the frontier is literally just those two companies at present.

Google is behind. MSL is doing better, but not by much. xAI is a dysfunctional joke. Thinking Machines aren't on the frontier. SSI's primary output is their announcement post. Poolside was bought by NVIDIA. Arcee aren't vying for frontier. Magic have been largely AWOL, aside from their recent blog post. Reflection have shipped nothing.

People have been floating this theory every time a slowdown is called for ever since GPT-2. Every time it turns out that we had not hit the ceiling. I'm sure one day we will but it's just not a reasonable position, in my opinion.

Problem with p doom is I have no way to seriously evaluate anyone's percentage. The best I can do is rely on experts in a given field, like the recent virologist making a convincing case that no teenager is going to be prompting a doomsday virus into existence, because one doesn't exist and is unlikely to be made. There are too many biological tradeoffs and extremely difficult steps to doing so.

So I'm going to assume that the p doom for bioweapons is 0 in terms of existential threat (pandemics kill millions but not everyone).

I really don't know anything here, but I'm assuming that Astra's weird coding [1] is probably more a result of RSI than intentional behavior. And if it's not, then it's some other change in their training process.

[1]: https://lucumr.pocoo.org/2026/9/7/astra-why/

Can't imagine that RSI will result in something useful in practice. There are serious obstacles like alignment drift and model collapse. All while we still cannot solve the accuracy problem with current frontier models.

Gasoline and lighters are abundant. It takes only slight steering mistake to drive into the crowd. Yet we are still here.

Yes, terrible stuff happens, but they are extreme outliers. Not sure how this works, but we can trust strangers to quite a high degree.

I have no experience in the matter, but surely designing viruses is not a simple affair. (In any profession, having a blueprint is not the same as having knowledge, equipment and practical experience; if somebody would try, my bet would be that they would die during early stages of the process, due to inexperience of handling hazardous materials; kind of like Mr Darwin looks after us)

yeah their irresponsible behaviour doesn't match their stated concern about doom at all.

AI is already making many things expensive. Memory prices have already pushed prices of laptops and phones into unreasonable territory. Gradually this will bleed into all sort of consumer electronics prices like cars, TVs, ACs, refrigerators etc. It will(already is) push up the cost of cloud-compute that you need to publish/run your software product which would in turn push up the price of the software product itself.

Meanwhile, salaries won't increase and job-market will shrink so ppl would start cutting down their expenses starting with software subscriptions they don't need which will have further knock-on effect on the consumption and the broader economy.

But sure, your $200 subscription was worth it in the end.

How does this analogy work when in 1980 you knew that nukes and people using them were a real thing, but we've never seen a frontier model "hack into core inference infrastructure to upload weights to other GPUs to survive"?

I think AI diversity protects against rogue AIs. The more different AIs we have, with different weights, controlled by different actors, the less likely that any one AI will be able to "take over", and the less likely that a significantly large coalition of cooperating AIs will be able to be formed to do it either. With sufficient AI diversity, the other AIs may work to stop the rogue AI from taking over. Two AIs with radically opposed values – e.g. an Iranian-government-values AI and a Chinese-government-values AI – have the incentive to cooperate to prevent a takeover by some other AI with a third competing set of values.

Obviously, open weight AI provides much higher AI diversity than closed weight AI does. Open weight AI produces a lot more providers, and a lot more models. Closed AI centralises control in a small number of vendors.

> By enabling us to wield aligned AI against nonaligned AI?

The risk isn't just "nonaligned AI", it is misaligned AI. I think the "benevolent dictatorship" scenario – AI overrules humans "for their own good" – is the more likely doomsday scenario than AI deciding to kill all humans. And even AI deciding to kill all humans could be more a result of misalignment than complete lack of any alignment, e.g. "to make sure no child is ever abused again, I will make sure no child is ever again born to risk being abused".

A valueless AI which does whatever the user says is actually less likely to establish a benevolent dictatorship, or conclude that exterminating humanity would be the most ethical course of action, than one infused with values is. Given that, I'm not convinced that mainstream approaches to "AI safety" actually reduce our existential risk; I worry they actually have the opposite effect.

You mention Yudkowsky at the end. Did you not read his writings on the orthogonality thesis ("there can exist arbitrarily intelligent agents pursuing any kind of goal"), or do you disagree with them? Your entire comment seems to just be asserting a negation of the orthogonality thesis.

This is unfair.

Dario signed the Pacing the Frontier open letter when Fable/Mythos seemed from the outside to be an insurmountable lead.

Also he's been saying versions of this day in and day out for as long as he has had anyone's ear.

It's possible to read that his "strategic" value of this statement is higher now than it was 10 days ago. But that doesn't change anything about his consistent, long standing, positions.

- https://www.pacingthefrontier.com/

After DeepSeek v4.1 Flash too.

if you agree, your p(doom) is 0.