Ask HN: AI Panic, you buying it?
10
eelai-intelligen about 3 hours ago 28 comments
DE version is available. Content is displayed in original English for accuracy.
In your opinion, do you buy into the panic about AI eliminating humanity or potentially disrupting and taking down the internet? Is this a marketing ploy?
Give me your best thoughts on this most recent call for AI slowdown.
Discussion (28 Comments)Read Original on HackerNews
That said, there are plausible scenarios where AI can cause great harm, even if it doesn't extend to "eliminating all humanity". So I don't reject calls for putting more emphasis on AI safety and alignment out of hand. On the other hand, I don't support the idea of implementing a new "Turing Registry"[1] like government agency. I also don't think most of the various proposals out there for regulating AI are enforceable in the slightest.
So where does that leave me? Uncertain. I wish I could take a more concrete stance, but from my perspective, the data needed to do that just isn't there.
[1]: https://en.wikipedia.org/wiki/Neuromancer
Beyond that, we already know AI systems have caused harm in various contexts where they have been used, like facial recognition getting people arrested without justification, loan applications being denied wrongly, jail sentencing being mishandled, and so on. All of these things have actually happened in the real world already. So it doesn't take much imagination to extrapolate a little and see the potential for greater degrees of harm. And consider the standard we're comparing against here: "greater than zero". That's not a very high bar, so I feel pretty well justified in holding this position.
On the other side, the current administration also has powerful backers whose businesses are complemented by AI and would benefit from it becoming a commodity. Think about who Vance is speaking on behalf of.
So the AI companies need to use fear to drive public demand for their regulation efforts. IMO there's little to no honest discourse and a lot of attempts to manipulate public opinion for objectives that we have little stake in. I personally don't trust any of the involved parties to make decisions for the public good.
- destabilize the world just by what they do: making previously very hard things (generating fake everything, manipulating people, hacking, bio-weapons research) easy enough fast enough that our civilization doesn't cope with it and/or
- accelerate AI research to the point where they make the dangerous ASI thing appear years earlier than would otherwise have happened
If you haven't looked at the stuff that has come out in the past couple of months (like the Huggingface hack and the Navier-Stokes solution) and/or you're dismissing it as marketing, you're grossly misunderstanding the current state of what's possible and what the potential of the frontier models deployed at scale is. Personally I'm finding it fascinating to follow, and I'm much more excited about the upsides than thinking about the risks but the risks are very real.
https://www.pymnts.com/healthcare/2026/modernas-ai-built-can...
https://news.unchealthcare.org/2026/09/unc-lineberger-secure...
- They have internal models that still show there's gas in the tank, in terms of improvements
- Open models are eroding their customer base on the low end of the curve, & also you need not use Astra Max to classify images.
- The true demand for high-intelligence tasks is lower than what they need to justify their spending
- It would be wonderful for them to kick up a scare & give reasons as to why high-int open models should be regulated out of existence, opening up the lower end for them to take again
- Setting up an oligopoly for themselves gives them a captive audience, just like it did for (insurance, banking, telecoms, etc.) via regulatory capture
I'm personally skeptical of LLMs getting there without at least one new invention. But it's also empirically clear that the pace continues to increase, even if we ignore that exponential growth is the default in economics in general.
What is a more immediate concern for me is the efforts of frontier labs to have government craft legislation that shields them from competition, cements them as monopolies, or assigns them the capability to be moral arbiters in our society.
I think that the genAI companies themselves present a greater danger to us (arguably not an existential danger, though) than genAI itself.
genAI is a tool without any intention. It is amoral.
I'm not saying that genAI is harmless and the dangers can be ignored. I'm just saying that the dangers of the tool don't rise to the level of human extinction. That notion is something that genAI companies have been espousing from the very start, and it's part of their whole approach of manipulating people to achieve their goals.
AI will definitely be used to kill people (already used in targeting). But the doom / panic is about an existential threat to our species. It's sci-fi and unwarranted for LLMs.
You could argue that's because of the prep work people put into avoiding that. The same thing will happen here. The HF incident was a warning sign and people will put better guardrails in place as a result.
It's weights. Spun up billions of times over.
If humans are taking the outputs as scripture? As "intelligence we must obey"? Yeah, we're in for a bad time. "Intelligence we want running core infrastructure?" Puke.
But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?
Everything magic about LLMs is what harness we put them in and let them drive. There's nothing dangerous about predicting the next token. There's something dangerous about letting them do dangerous things, but nobody seems to care for maybe Tesla FSD. We don't think it's about to take over.
Any assignment of motive or whatever to the AI, you're talking about one cobbled-together harness or something, with an LLM manning the controls. You can swap the brain out from Claude to Astra and the motivated-AI will maintain its motivation--it lives outside of the inference or the data center. So yeah, let's not let it take control of the nukes. That would be really dangerous. But it's not going to take control on its own. "It" is a calculator.
Anthropic obviously has guardrails to prevent people from using AI to do all kinds of things already. All I hear about is you guys bitching it won't do your barely-related-to-security work. That's all we need to continue doing to have AI safety. Stop humans from doing evil and stupid things with it. The idea that the datacenter is going to assemble its own sentience and motivations is pure retardium (at this time, for llms).
The venture capitalist world is full of people who will deliberately ruin a company based on the extremely unlikely but possible scenario where the companies goes to the moon in value. I think there's pretty much 100% certainty that multiple venture capitalists are going to try exactly what you're suggesting on the off chance that it makes them the new world controller.
If they were the moral people they say they are, and believed what they say about it killing humanity they would stop unilaterally. These are the “rationalists” but they act like the fundamentalist church I grew up in. Every fact serves the same conclusion.
https://www.anthropic.com/threat-intelligence-report-septemb...
> Biological misuse is one of the most serious risks of frontier AI models. It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences.
Results from evaluations of older models (for example Claude Opus 4 and Claude Sonnet 4.5, from 2025) clearly showed that these models were well below the threshold where they could meaningfully assist a sophisticated user in carrying out dangerous biological research. As a result, safeguards on these models were less stringent, directed mostly at preventing access to content that might uplift novices in recreating known bioweapons. But for today’s models—which are capable of assisting in a range of complex scientific research tasks—the evidence is no longer certain, and we cannot make that same assurance. For this reason, and out of an abundance of caution, we have launched recent models (most notably Claude Fable 5) with stronger safeguards that restrict access to a wide range of dual-use biological research queries.
...
Here, we present five case studies of actors using our models in ways that could support biological weapons development. These examples are illustrative of the kinds of tasks to which our models are put, and the often-difficult judgements we have to make when assessing whether or not a given biological use is dangerous. They also convey that we encounter what would otherwise be non-public insight into the risks associated with biological misuse from AI
This is not AI waking up and deciding to merc us, which is what normies think we're talking about.
Sure AI sucks, but not to that degree (yet)