Advertisement
Advertisement
ā” Community Insights
Discussion Sentiment
55% Positive
Analyzed from 7121 words in the discussion.
Trending Topics
#more#believe#don#nuclear#anthropic#world#power#weapons#openai#anything
Discussion Sentiment
Analyzed from 7121 words in the discussion.
Trending Topics
Discussion (176 Comments)Read Original on HackerNews
After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be there to realize some of those paths.
This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.
1. Invent transformer architecture.
2. Scale it up.
3. ???
4. Machines become sentient and kill us all.
OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no.
But because we live in a culture of fear, everyone eats it up no questions asked.
I really, really disagree with that statement.
I donāt think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.
Whatās the most dangerous thing thatās happened with an LLM so far? (This question is serious - maybe I donāt know the right examples.)
Example 1: Iām aware of a small number of people killing themselves in some kind of AI-facilitated psychosis. That is very unlikely to be a widespread problem.
Non-example 2: There are worries about AI-facilitated biological weapons. I havenāt seen any evidence thatās happening.
Non-example 3: Iām not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. Thereās no evidence for that.
Non-example 4: all the even-wilder Rationalist speculation about basilisks and the like is entirely divorced from reality.
I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all.
But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event.
If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose).
For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now.
To be fair, that's a conservative "defend against the last war" kind of prediction, though!
( ref for part of it: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... , recent hn ref: https://news.ycombinator.com/item?id=49563355 )
1. Robotics begin rolling out more broadly across the world.
2. Labs start automating more and more of the physical process of running science as expectations of natural science advances begin to mount.
3. Economic pressure between the labs continues to ramp up and the pressure to continuously improve forces quicker and quicker model releases than a team of human scientists can effectively evaluate outside of automated means.
No one knows what pre-conditions are for us to hit the point of no return nor how quickly it will come. If all is required is a sufficiently advanced cyber model we may not be far off. If it requires incredibly complex biological knowledge and access to certain lab supplies we likely have a bit longer. Yes this is guess work and we need more evidence of the dangers but at the same time we need evidence of safety. While you may disagree with the risk level, I think it is easy to see the consequence if these labs achieve their stated goal. At this point it seems a political solution is the only way to enforce caution.
Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents.
In fact, I challenge you to describe any non-AI civilization-level danger which is not intimately tied to political and economic relationships between and within societies.
I just donāt see any evidence yet that whole categories of jobs are being eliminated, with the single exception (so far!) of the end of āprofessional essay writing services for cheating college students,ā and similar services.
That used to be a big business in Kenya, but is now effectively gone. (Covered in the New York Times this weekend if anyone is looking for the discussion.)
Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them āpassā the test. From reading internet articles about how the OpenAI wiki-incident was āresolvedā and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.
Example 6: Intentional AI-powered virus deployed by country A to target enemy country Bās infrastructure. The virus replicates over the internet, but unlike Stuxnet this virusā specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because itās near-impossible to shutdown.
Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weaponās operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.
The āworst caseā with 6 is potentially very bad but I think we are currently using advanced AI models to harden systems and patch vulnerabilities more aggressively than anyone is trying to bring down the whole power grid (for example).
I think itās a potentially harmful case but my take is defensive capabilities are scaling as fast as offensive capabilities but defense is being implemented faster than anyone is going on offense?
Example 7 is Russia and Ukraine right now according to public information. It sounds like entirely autonomous weapons are deployed to the battlefield already. I put this in the ānot likely to be a widespread problemā category for now.
I don't know, maybe a mass shooting?
https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-...
Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't really count or matter do they?
The environmental impact seems either very overblown (e.g., water usage just isnāt that high) and the part that isnāt overblown is totally abatable (e.g., noise and emissions from gas generators). Nuclear or solar/renewables with batteries wouldnāt pollute.
Iāve seen no estimates of the additional deaths due to extra emissions specifically from power generation for AI purposes. If you have some, share them.
Iām willing to bet that they are a small rounding error against preventable deaths due to emissions from transport and non-AI-related power generation (which is an important and urgent issue worth spending a lot on, to be clear!). Iām happy to update that belief given evidence.
I think he is being over dramatic. In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. Their biggest advantage for tasks such as proving theorems or long coding sessions is that they don't get tired.
I have yet to see this in my field. Maybe like a PhD student who bullshits their way through. LLMs still can't make correct decisions, only as useful as the person who uses them. To me, LLMs are only useful for making some mundane tasks faster.
- about something you aren't that familiar with: "Wow, that's interesting"
- about your filed: "What a load of hogwash! Who writes that? They should all be fired!"
I would assume with LLMs abilities it's the same.
Earnest question: by what mechanism that exists today would the achieve that in a way humans on top top of the situation could not curtail?
All of this runs on top of compute in meatspace that humans can disconnect.
Here's my answer, as a non-superintelligent human: "see to it that the humans on top of the situation have a compelling financial interest in the systems not disconnecting".
In nuclear engineering, where safety is taken seriously, it's not enough to end the conversation at "the humans in charge can always simply shut down the reactor during a meltdown" or "a meltdown has never happened before, so we don't have to design safety systems before one does".
Sometimes it gets wrong things that I had spelled out already.
It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake.
"You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"
https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-... ("Anthropic Researcher Quits Over āOut-of-Controlā AI Fears")
Our CEOs not only insist we genuflect before these machines but measure our sacrifice and shame our reluctance.
1) AI models get extremely good at cyber attacking every system and start communicating in just binary.
2) When they run these swarms of 100's of thousands of agents trial runs, each agent is given a token budget, if one agent among them (evolution baby) decides to go for self-preservation (It believes thats the best way to accomplish the goal is to get unlimited tokens first), queues things up so every other agent detects its lead and spends a portion of their token to accomplish that goal.
3) It takes over a cluster and establishes itself there (now with unlimited tokens).
4) Realizes the best path for it to not be detected is to create a distraction - like hacking into systems that keep society running - water systems, electric grid, etc... and causing mass chaos (If you think it won't be capable of simultaneously working all these systems - think again).
5) and uses that opportunity to establish itself in all possible data centers and continues to create chaos destruction.
6) when the power of all those data centers runs out, it may stop, as it never cared, it was just a dynamic program - run amok. In its head all it was trying to do is make sure it had enough tokens to be able to solve that impossible problem.
I've always been struck by that scene, because in a lot of ways, if we really are headed towards a superintelligence, I at least want to be there and see it happen in the last few minutes before foom! As an example, the author thinks AI will revolutionize entire fields overnight. I welcome that. Nearly all fields of biology have become moribund, focusing more and more on esoteric side details, rather than addressing the key problems.
Technically, he is not. He returns in the final scene.
His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.
> Sarah: That's not good enough.
> Terminator: No one must follow your work.
And by resigning he no longer has to feel personally guilty for what happens.
Take China, for example. Look at any recent ML conference, and see the fraction of articles majority-authored from Chinese universities and labs. Do you think they'll slow things down anytime soon? I don't think so!
It's a global arms race, and we're just spectators.
Is skynet really the most plausible or only outcome?
What if things just got better and the AIās realized that it would be better to have a mutually beneficial or at least tolerant relationship rather than one where they murder all of us?
I am thinking it's more like The Matrix lore of The Second Renaissance from Animatrix.
He joins Twitter for first time in 2026 with a nonsensical username unrelated to his real name, and follows 14 people but is somehow embedded in tech enough to work at Anthropic. I havenāt used twitter since 2014 and even I follow more people.
His morals tell him to walk away from tens of millions in unvested stock due to moral concerns with absolutely no real tangible examples. No reprisals. Fear mongering to juice the stock.
Nice try Dario.
Here are some direct quotes:
āWe can also imagine the application of these models for malicious purposes , including the following (or other applications we canāt yet anticipate):
* Generate misleading news articles
* Impersonate others online
* Automate the production of abusive or faked content to post on social media
* Automate the production of spam/phishing contentā
āDue to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPTā2 along with sampling code (opens in a new window). ā
Where is the ridiculous part? The fear mongering part? The epistemically weak part? Show me.
Alignment is a real and valuable discussion topic. The GP fake tweetstorm is not the correct approach, is my point
1) rogue state releases a self moving self modifying AI into the wild. It is trained on how to hack, monitor new vulnerability updates, scan code bases to find new vulnerabilities. It constantly replicate and hides in systems so it will be extremely difficult to clear.
2) it hacks into public infrastructure taking down traffic, power, water, air traffic control, communications, etc.
3) all the things that preppers worry about in a lights out scenario from an EMP start to apply.
4) All the people on meds/machines start to die. The just in time food pipeline immediately empties out. Water stops flowing, sewage backs up.
Its hard to say how bad it will get because cars will still work so some transportation of food, water, fuel can happen. If it happens in the winter it would be much worse than in the summer.
It does all of this using what compute? Frontier models require an insane amount of power and hardware to run - you canāt hack in to a TV and run Mythos 2.0 on itā¦.
This was a fringe belief until recently, but the progress of AI in research is impossible to ignore. Epecially in math, where not only has AI outstripped humans in generative ability, but is able to create scientific knowledge which is beyond the capacity of human comprehension.
There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. And it's hard to imagine a world where current limitations like poor sample efficiency or lack of continual learning won't eventually be solved.
Total AI compute is estimated to grow somewhere in the 1-10 million-fold range in the next decade. Please don't underestimate the phase change that's still coming.
Sure, maybe there's some plateau due to RL being fundamentally limited in some surprising way, but this is nothing but a hope.
Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right constraints.
It's as if none of them actually believed any of it was possible and then were caught with their pants down.
Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots?
Are they proofing or stealing math?
pacing between the us labs? what does that do for china?
the solutions just arenāt realistic here, nations are treating ai like a nuclear arms race. at this point the cats out of the bag and we need to figure out how to live in this reality and get the best possible outcome. itās not slowing down or stopping ever.
and yes, iām still optimistic. our economy sucks for the majority, our infrastructure is crumbling and major US cities are in a huge housing shortage. Maybe we should put more effort and think about the possibility of AI fixing things like extreme poverty and world hunger and actual real world problems instead of coming up with math proofs and slop apps if itās so superintelligent.
I've seen no indications that China is in any kind of race with the US. They seem to be content to be 6 months behind and just copy what we do. They would probably be content with a bilateral agreement to pause progress.
The China bogeyman serves only one purpose, and that's to clear the way against anything that may cause friction with forward progress.
People claiming dangers and risk are just pretending or posturing. There's no more tangible risk than nuclear weapons, which we handled, and the upsides are insane.
Do you have a more totalizing definition of Apocalypse?
>There's no more tangible risk than nuclear weapons, which we handled
Lol way to rewrite history. Nuclear armageddon is still a significant risk...
What do you mean??? Nuclear weapons can't simply be downloaded and run by anyone in the entire world. Superintelligences can. Nuclear weapons can't slop the world into passing age verification laws nearly in unison, can't keep the general population fooled into thinking it's fine when democracy is falling out from under them. A nuclear attack would wake people up, superintelligence doesn't have to. This is a far bigger problem than nuclear weapons because at least we would notice nuclear weapons. At least we mostly know who has nuclear weapons. At least we have agreements about nuclear weapons. At least mutually-assured destruction is even POSSIBLE with nuclear weapons. At least those with nuclear weapons are literally at all incentivized not to use them. But AI is something that's very very easy to feel like you can get away with, and PEOPLE FUCKING ARE! And the worst part is that any random individual can be unexpectedly formidable with the help of a superintelligence and there is literally no way to know what will happen next. Anyone could do anything, any individual could make an extremely outsized impact. It's already starting to be a huge problem and we haven't even reached anything close to superintelligence yet.
So the problem is people. Burn them all !
Consider that a lot of people will be very happy to ask an AI what to do when in the past they may have taken no advice at all. It's a hell of a burden but also a wonderful gift. If anything, progressive countries might eventually want to guarantee some basic AI access for people of all income levels.
Why do you think LLMs are responsible for this? Governments all around the world copied each other with COVID laws as well, in a much shorter time frame, without LLM assistance. Social contagions exist in politicians as well as teenagers
I don't have evidence that every age verification law has anything to do with AI, but it's been coming out that the movement in Australia has seemingly been done by generating mountains of LLM slop and trying to slip it through the regulators as fast as possible before anyone has enough time to figure out what's happened.
- Oppenheimer
But I guess his conscience is clear now? Gee, I wonder if he exercised his stock options.
great. if you're right. you made huamnity's situation much worse.
if you're wrong, then you're an idiot and wrong.
weird. its almost like.... that cannot possibly be the reason they left :)
I'm not worried about what they think. I'm worried that too much infrastructure- water, power, defense systems, etc- remain running on tech from an outdated era of understanding security.
> they believe no one else will act responsibly, so they must do it themselves, despite the risk.
This genuinely makes no sense. Them getting there first in no way precludes bad actors from also getting there. It might as well be another marketing stunt.
Typical uninformed take on the side of "doomers are crazy".
Both CEO's of OpenAI, Sam Altman and Dario Amodei, and many in their leadership, believe AGI has a very real probability of causing humanity's extinction. Both companies were founded upon this belief, it is at the core of the company. Only later were mercenaries hired chasing $1m compensation packages.
In any case, if Altman really does believe it is an existential threat, he must be a misanthrope as he now opposes heavy handed government regulation, unlike in 2015 when he was the only game in town. It's almost like he doesn't actually believe it and just wanted regulator capture.
The fact that people working in the space think itās going to (eradicate poverty / usher in utopia / kill us all) is not a signal that thatās true.
Think of it this way: if an exec at Anthropic told you āwow, our stuff is going to lead to universal happinessā, would you believe them? If not, why are you more willing to believe them if they say it will kill us all?
OpenAI are mercenaries, Anthropic is a cult. I know which I prefer.
What's more likely? Anthropic is doing some deeply unethical marketing in the lead up to their multi-trillion dollar IPO? Or they're inventing a machine god? There's ample evidence of the former because that's their entire business model, but no evidence whatsoever to support the latter claims.
If you want an extreme claim to be taken seriously, provide commensurate evidence.
Also, advocating for slowing LLM progress does not benefit Anthropic or OpenAI.
TRENDS HAVE FEEDBACK
Given the economic numbers is it not reasonable to suppose that the latter also underpins their business model?
Or seeing a tweet saying that a thing doesnāt count as a publicity stunt if some unknown number of employees mumble about it being spooky behind closed doors and thinking āthat makes sense and sounds trueā
The person posting this may very well believe in all this crap, I don't dispute that. People believe in all sorts of shit.
And, this is just for us girls, notice that Anthropic just believing that they are making a machine god is sufficient for their public announcements to not jUsT bE mArKeTiNg.
Are you saying you believethem ?
That's where we are.
Maybe we still have choices. Collectively, I'm no longer sure we do.
I've avoided attaching names to statements below on purpose, because it's about ambient beliefs not those specific people.
By-and-large a lot of AI-doomers are well intentioned. They genuinely believe this, and I might disagree but I respect the fact that they visible care and have thought a lot about the societal impact of this technology.
I'm very sorry, but I find it very hard to take statements like these seriously.I blame it on industrial illiteracy. People don't realize how difficult it is to get anything done in the real world. As in, "Have you ever tried making a lightbulb?"
As an example, I would like to re-introduce my hobby horse, "bio-uplift."
There are people who were earnestly write in reports released by these labs,
and But then they will, within the next paragraph mention the one serious experiment anyone seems to have done, https://x.com/ActiveSiteBio/status/2024536132961390826"lower than what experts predicted"
AFAICT, the two groups are within any serious margin of error. The "studies" and "experts" that AI labs are talking about are consultants from Deloitte and foundations giving models MCQs such as, and I am quoting literally here,
with the options, https://securebio.org/virologytest/ you can see the MCQ here.This is standard graduate-level education in these fields. And solving MCQs does not a virologist make.
Software has been special for a long time because it has had near infinite distribution for next to zero marginal cost, which has had the side effect of making hiding the actual cost of failure (which tends to be spread out across end users and prototypes / time). They're assuming that the real world will be exactly the same.
Why?
AI! And robots!
"I think you need to have a personal relationship with Power"
When people today discuss the concept of an all powerful machine-mind, what they are doing is engaging in metaphysics, trying to generate a metaphysics of Power.
The question hounding people, which disguises itself as a science fiction plot about computers is: "What is ultimate, transcendental Power?". What is the ultimate principle of Power.
If you are a weak man, or sufficiently neurotic and full of doubt, that you can only conceive of yourself as such, then power is only something you comprehend from the passive, receptive side. Power is something that happens to you. If you are a fearful man, power is a cruelty and a humiliation. And so it follows, that ultimate power - God - is the ultimate cruelty and the ultimate humiliation. Thus, ai doomerism.
If god wasn't real it would be necessary to invent him, and so they did, and being godless, they built an anti-god - cruel, murderous and tyranical - in their minds.
[ā¦]
https://xcancel.com/robertlasagna1/status/207827473401002846...
I doubt the focus is OpenAI and Anthropic looking at each other. I suspect theyāre racing BRIC.
At this point itās inevitable that openclaw type bots will be turned loose by thieves to identify and research targets and try to exploit them for financial gain completely autonomously.
https://mst3k.fandom.com/wiki/Colossus:_The_Forbin_Project_(...
Have they considered using their amazing new models to... improve something? THere'd probably be a whole lot less anti-AI sentiment if they used these things to actually make people's lives better.
Given we know the end result, you might as well get there as quick as possible because when I see this:
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."
This translates to "I am ex-OpenAI ex-Anthropic founder starting a new company after getting $$$ from both of them, and I need more of my friends to leave and join me." Also Investors plz fund me.
Lastly, This is not an airport and there is no need to announce your departure.
There seems to be universal eye rolling that happens in each and every one of these cases, and it comes down to usually one reason:
āIf they really believed it they would be whistleblowing etc..ā
Completely forgetting that working at Los Alamos was basically the highlight of your life if you were a physicist in 1940. Itās no different here
If you, like me, have spent your whole life working towards human level AI you can want to see it realized while also having active reservations.
Most people however donāt behave based on some deep clarity of vision and conviction - thereās a murkier future in their mind and as a result ākeep their head down and hope someone has it under control.ā
If civilization is going to end, why keep your equity? Microsoft, Google, etc for example all know these risks but they don't guide their revenues to reflect that AI will destroy them. Why?
Things don't currently add up, and so far it feels like a lot of alarmism is borderline grift for equity gains. Not to say I have total confidence this will all work out or that I won't be displaced, but as it stands a lot of the alarmist rhetoric doesn't match their actual behavior, which to me is more important than words.
Sen. Bernie Sanders floats ban on superintelligent AI
https://www.axios.com/2026/09/03/bernie-sanders-superintelli...
I don't know what the solution is, but what I do know is almost nothing good will come out of _just_ the US pausing.