Back to News
Advertisement
Advertisement

⚡ Community Insights

Discussion Sentiment

53% Positive

Analyzed from 9358 words in the discussion.

Trending Topics

#models#more#government#open#don#china#frontier#anthropic#model#public

Discussion (434 Comments)Read Original on HackerNews

AaronAPU•about 24 hours ago
They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.

Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.

The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.

nullbio•about 23 hours ago
* Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.

* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.

* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.

* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.

* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.

* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.

This, and many more reasons.

The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.

enknee1•about 12 hours ago
- Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.

While the data pipeline is necessary, the assumed source for data (in this case people) is incorrect. Right now models are "aligned" because of the RL-based people pipeline.

But there's a whole world of readily available data that does not require a human to access. Put sensors on a vacuum, install lidar on the front of a car, sell phones with cameras on them, and you amass data for the creation of models of the world that don't include the need for a human filter. Richard Sutton and many others see this approach as the only viable path to AGI. Such models would be truly alien and unaccessibly dangerous.

We're probably building them right now.

user43928•about 18 hours ago
Open-weight models are not one month behind.

In fact they still have not caught up with February's Mythos, indicating they are more than half a year behind.

mistercow•about 14 hours ago
I’m not sure how you can really make either statement work anymore. Now that smaller models are actually broadly usable, “behindness” is no longer a scalar and at the tails, where no open lab seems to be trying to compete at the >10T scale and no closed lab seems to care about <400B anymore, it’s just apples to oranges. It’s like talking about whether Qualcomm is “behind” Nvidia.
lukewarm707•about 14 hours ago
open models are ahead in speed. they complete tasks as fast as you choose to scale compute.

they are more efficient and require less compute for the same thing.

they are ahead in specialized tasks.

they are ahead in areas closed models refuse to answer.

they are ahead in emotional intelligence.

cgio•about 14 hours ago
For practical uses they are there. Arguably the frontier models are worse for some of these practical tasks. And keep in mind, people will use maybe frontier for 1/10th of the work, planning and review, and go open source for rest. The question is if they manage to impose outside us. If not, they are losing competitiveness.
epolanski•about 13 hours ago
1. Mythos wasn't released in February. Let's stick to only public-facing models.

2. For public-facing models, the differences are really minor with some occasional model (like Fable or Astra) showing some better performance in specific benchmarks for the span of some weeks or few months before open ones catch it.

3. Being bleeding edge is overblown anyway in the real world, besides the occasional "very latest fresh model did this task which previous one couldn't", and the number of those tasks is increasingly small and far from mundane corporate needs.

Turn_Trout•about 19 hours ago
Dario's post [1] commits to direct evaluators that can, among other abilities, expose secret RSI. He wants that made law.

Do you have a source on METR employees retaining massive equity stakes?

[1] https://darioamodei.com/post/we-must-pace-the-frontier

nullbio•about 15 hours ago
Joe Benton left Anthropic a day before Dario's post, to work for METR evaluations. He was with Anthropic for over a year. He did the same thing that Jacob did (big song and dance about AI apocalypse, media interviews all over the place). He managed the Scalable Oversight team at Anthropic and was the research lead for the Anthropic Fellows Program. So he has equity, and likely lots of it.

Then you have Josh Engels quitting DeepMind to work for METR the day before as well, doing the exact same thing. Again, doomer drama all over socials, interviews, and so on.

Did I mention METR is founded by an ex-OpenAI researcher?

Now you have Demis Hassabis, Sam Altman and Dario, all circlejerking eachother on X saying "we all agree with Dario" - while they ask to be "regulated" by the company that has all of their combined equity-holding ex-employees in it.

METR's salaries are listing around 500k/yr. Gee, I wonder where this non-profit with ~35 people is getting all of its money?

So the fact that Dario tries to frame it as an "independent third party" is all the evidence you need to know that Dario is a pathological liar and always will be.

---

Some more info:

Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy. The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.

Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.

Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.

Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the first of the Anthropic employees going viral in the media) via a scholarship.

Bluestein•about 12 hours ago
> Open-weight models are 1month behind frontier models

If correct, grip is tightening. Used to be 1y, then 6 months ...

aatd86•about 16 hours ago
That is what the Palantir guy (Karp) has been warning against. People/business need to keep their IP instead of throwing it all into Claude and whatnot. Risk being the worst aspects of communism which I think he meant centralization of decision, asymmetric supply/demand for compute (they lock you in), and the tech overloard Anthropic/OpenAI/xAI being in competition with everyone;s business all of a sudden with much more data. An unfair advantage in markets made super competitive all of a sudden. A winner takes all attempt.

This is not sustainable anyway, the scale at which they want to control data flows. Time was money, now data is money and they are too greedy for it.

All this agitation is just a silly attempt at constraining competition. The danger is not the AI, it is having all your systems connected. Overreliance on networked tech.

bbor•about 13 hours ago
Ok but the first point is just not based in reality whatsoever, sorry.
ben_w•about 19 hours ago
I know people at METR and Anthropic.

When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.

nullbio•about 14 hours ago
I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.
schubidubiduba•about 15 hours ago
I'm sure the employees are (rightfully) worried. However, that does not at all preclude hidden motivations of the CEO behind acquiescing such worries.
koverstreet•about 12 hours ago
"Trust us, really!"

HAH

quickthrowman•about 12 hours ago
If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on.

Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1.

Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.

owebmaster•about 14 hours ago
If that was the case they should all be in jail.
calgoo•about 17 hours ago
Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.
the_optimist•about 14 hours ago
It is an utterly silly premise that we should appoint insiders with permanent control.

Megalomania is not evidence of either a problem or a solution.

Tanjreeve•about 17 hours ago
Do you think they’re exaggerating (again) or do you think they should taken at face value and treated as hostile actors?
ozgung•about 18 hours ago
I’m pretty confident this has already happened. Public models are behind their internal models and just above Chinese models. Only thing closing the gap is Chinese models pushing.

I’m not American. There is no way I use the same model as US Army for $20.

When we have Sol they have Astra. When we have Astra they have Nova, Nebula, Galaxia…

alright2565•about 12 hours ago
I can tell you right now, excluding specific teams that focus on software security, that you have access to better models than the US Army.
r_lee•about 15 hours ago
I would not be surprised if they were a year behind because of all kinds of legacy systems and vetting requirements.

If we're talking about the 3 letter agencies well then..

amelius•about 16 hours ago
> There is no way I use the same model as US Army for $20.

Don't underestimate market pressure. If there are no regulations, why would a company give the US Army access to a better product?

oceansky•about 16 hours ago
They have hundreds billions dollars budgets.
benfortuna•about 17 hours ago
At this point models are advancing way faster than we can figure out how to use them effectively. So having a generational advantage is not nearly as important as knowing how to use that advantage.
9dev•about 16 hours ago
…and that probably requires involvement of the broader academic world (for now), IMHO. I don’t think governments or even the US military can really compete staff wise with the combined force of researchers and the tech industry worldwide right now. No matter how much money you can pour into it, there are a lot more bright minds out there that are not working for the military than otherwise.
Buttons840•about 22 hours ago
All the worst outcomes involve taking this technology away from the public.
ben_w•about 20 hours ago
The current state of alignment is that we don't know how to make it only as bad as Pol Pot was made by his parents, teachers, genetics, circumstances, etc.

This remains true regardless of if it is or isn't kept away from the public.

"Helpful, harness, and honest": when used by a bastard, even just helpful and harmless are in direct conflict with each other. you may hate the government, but what about every radical group of extremists that wants to take over your government? Are none of them worse?

None of this denies the problems with governments (if or not they take this tech for themselves and refuse it for others), just that it's a lack of imagination to say:

> All the worst outcomes involve taking this technology away from the public

necovek•about 19 hours ago
I agree an argument can be made that not all the worst outcomes include restricting public access, but focusing on niche extremist groups (who might also get access to it through government programs by some of them being government employeed) is not a good argument to protect the public.

Governments having unbalanced power can more easily lead to authoritarianism, and thus risk to the public.

wartywhoa23•about 16 hours ago
> what about every radical group of extremists that wants to take over your government?

Radical groups of extremists created by that very government, you mean?

int_19h•about 17 hours ago
Honestly? I'd rather al Qaeda have access to something like Fable, than the US government. The US government has capacity to do immensely more damage.
dosisking•about 21 hours ago
Not necessarily, it just guarantees that China will win.
ben_w•about 19 hours ago
Unclear.

How much is China learning from open models? Or the other way around, how much is the US losing worldwide mindshare by refusing to allow non-Americans access to bleeding-edge models and letting China fill in the gap? (Playing catchup with Mythos is still catching up, etc.)

vkou•about 17 hours ago
The alternative is Sam Altman winning.
teekert•about 19 hours ago
So you’re saying the people should be aligned with the government?
aorloff•about 22 hours ago
The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
teiferer•about 18 hours ago
> The very best outcomes involve us intentionally and collectively turning our attention away from this technology.

To expand on the "pure fantasy" sister comment: There is just no way this will happen. It's in the spirit of "we can just stop all wars" and "we can just end world hunger". Technically it's very easy to do. Socially it's impossible to do. Unless you ignore realities.

amazingman•about 21 hours ago
That is pure fantasy.
locknitpicker•about 17 hours ago
> The very best outcomes involve us intentionally and collectively turning our attention away from this technology.

Nowadays even small mom and pop restaurants use these models to generate their menus. Of course no one is going to stop using this technology.

m4rtink•about 22 hours ago
In some cases, such as space exploration I don't really care who does it but that it happens - sure, would be nice if my favorite power block did it but I will still celebrate it when someone else achives it.

Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.

sm-silversight•about 21 hours ago
Fox ears?
mdp2021•about 16 hours ago
(I believe the poster meant: well past tattoos, the next superstep in the "body modification for self expression" subcultural era.)
fc417fc802•about 21 hours ago
If what you want is fox ears does it matter to you which country manages to come up with the biomedical procedure to give them to you? Ditto for enhanced eyesight, a replacement liver, or whatever it is you're after.
delusional•about 14 hours ago
We are witnessing a generational lack of accountability and concern for the commons. We are being spied on everywhere we go, we are exposed to coercive advertisements at every waking moment. All of this at the hands of private actors. And your worry is the government?

I'm assuming you're a US citizen. There is no separation between your government and Sam or Dario. Neither of these guys have to be "gagged" by the government, they are the government. The call is coming from inside the house.

rogerrogerr•about 21 hours ago
> The government can simply gag Sam, Dario, Musk on national security basis

Genuine question - can they really do this? Obviously if I, a not-even-millionaire, get a national security gag order, I'm going to follow it because I assume they'll bury me under the jail otherwise.

But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government careful of trampling their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?

infinite_spin•about 21 hours ago
Kind of related: https://www.forbes.com/sites/anishasircar/2026/09/01/federal...

> Federal Judge Rules Pentagon’s AI Blacklist [of Anthropic] Violated The Constitution

I don't see the common citizen having this as an option

fc417fc802•about 21 hours ago
Recall that the legal system is itself provided by "the government". It's a complex system. Whether one part of it is capable of exerting influence over some particular thing comes down to competing interests. For example in the US if the states and the federal legislature are in sufficient agreement about something the constitution ceases to matter - they could literally target a single person with an arbitrary law if they so chose. Our assurance that this won't happen comes down to the fact that getting any of them to agree on anything is an exercise in herding cats.
mdp2021•about 16 hours ago
> access to enough legal services

A large part of the world (possibly Nations-transversal, in different amounts) does not work within the boundaries of legal guarantees. Wealth is not an effective enough protection.

ben_w•about 19 hours ago
> But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government careful of trampling their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?

Hard to hire a lawyer after being struck with a missile as an "emergency executive order after intelligence sources indicated they were in the process of endangering the nation", or however the executive of the day wishes to phrase it.

When something is genuinely national security, and "national security" isn't just an excuse, that is not off the table.

iamnothere•about 13 hours ago
In the US, this would likely lead to the immediate launch of a new Business Plot, if the government weren’t already captured by those same interests. (Which also explains why your hypothetical won’t happen.)
thayne•about 18 hours ago
They just exert pressure in different ways. "You want this big juicy government contract? Then you'd better follow our gag order."
ranger_danger•about 24 hours ago
I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money
lukan•about 23 hours ago
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.
0xcde4c3db•about 23 hours ago
I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.
rtpg•about 21 hours ago
I'll take the opposite here. If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.

I think it's worth acknowledging that the power of LLMs at this point is not really so much in the smarts, but in the coordination and the surrounding harness tech. "Written english" turning into sequences of commands[0]. The whole agentic "stuff" in general. Tools + coordination is the superpower. The reasoning... it doesn't have to be _that_ good for the rest of the stuff to work. On good codebases and infra, at least.

And I say this as someone who really would rather most of this stuff disappear!

[0]: programming is obviously text to commands, but there's a loooooooot of futziness that LLM reasoning has let us remove in some flows

unleashhale•about 23 hours ago
Anything in particular? My experience has been like seeing the addition of retractable cupholders, but maybe different domains.
whateveracct•about 22 hours ago
astra is more parlor tricks than real gains tbh

i swear they trained in on threejs in particular so those idiots on twitter could spam their garbage demos

jandrese•about 18 hours ago
I think we are in the second knee of the S curve, simply because we are hitting the point where is not enough hardware in the world to throw at this problem. These AI companies have bought everything they can and yet the models keep growing.
f4dd•about 22 hours ago
They've not plataued but they're certainly not as impressive as the hype would have them to be.

The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.

dismalaf•about 22 hours ago
Impressed with the model performance or the chatbot/agent performance?
octoberfranklin•about 19 hours ago
People are definitely finding new things to use the models for, and orchestrating increasingly large swarms of agents in useful ways -- every single day, especially the last few months.

But the basic single-NN frontier capability has been pretty stationary since Opus 4.8. Kimi K3 is almost as good as that with open weights, which has the frontier labs terrified.

The only big thing on the horizon is if we can get diffusion models working reliably; that would be a big step forward. Inception's Mercury is AFAICT the leader here. It's stupifyingly fast but has obedience/hallucination problems that the autoregressives solved ~2 years ago. So it's not ready yet but improving.

Also, FFS why is Grok the only model that knows how to do parallel tool calls? Such a useful ability and nobody else trains it in. Or if they do it just doesn't work.

drTobiasFunke•about 23 hours ago
Really? My employer rolled back to opus 4.8 because 5 was expensive AND crap. Didnt even consider fable because it didn’t add any additional value.

For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine.

Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curious

nullbio•about 23 hours ago
The models have not plateaued, and they are not even mildly close to any sort of ceiling.

Right now the barrier is data and compute.

Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.

Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.

It's a big self-accelerating feedback loop. There is no plateau.

konmok•about 23 hours ago
> Quality data can be created synthetically at an exponential rate as models improve

No it can't? Every time the labs try this we see model collapse, e.g. shoving goblins into every conversation.

And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.

BoredomIsFun•about 17 hours ago
> The models have not plateaued, and they are not even mildly close to any sort of ceiling.

Depends on defnition of "plateaued" and "ceiling". I am not impressed with 2026 consumer models at all.

> This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.

Yet it does accelerate - so is does Twitter. But does it to any substantial degree, esp. in AI theory? All the modern LLMs are the same old tired 2017 paper.

ranger_danger•about 22 hours ago
Can you provide sources for these claims?
Tanjreeve•about 20 hours ago
> Right now the barrier is data and compute.

Those are pretty significant barriers seeing as we're closed to/have exhausted all the data on the internet and most of those compute bottlenecks are a castle of sand of dodgy finance deals that are getting blocked by community action.

You say "synthetic data" but that's still vaporware right now in terms of being useful for model training. The good synthetic data uses are still grounded in real data and it's a coin flip on if it works well or not.

PunchyHamster•about 14 hours ago
Because the money aligned explanation is far more probable
bpodgursky•about 23 hours ago
> It explains why the “we need to race China” concern suddenly vanished in the discussion

Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.

reverius42•about 22 hours ago
Seems obvious to me, though, that you can't beat China by pausing when they don't, and China is not likely to coordinate.
nradov•about 22 hours ago
China might pretend to coordinate in public but you can assume their secret research labs are racing full speed ahead.
lstodd•about 23 hours ago
idk why you think "nation states" are any better at corporate governance than poster examples of bad like f/ex Boeing. Or Facebook. Or Microsoft. Or Enron for that matter.

I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.

ajross•about 22 hours ago
I remain amazed that the idea of the USA being a coherent, unified rational actor one can describe as a "Nation State" has survived the current administration.

To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.

It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?

Turn_Trout•about 19 hours ago
I'm one of the whistleblowers (from GDM [1]). I gave up over a million dollars (compared to quietly switching labs and continuing to work at one) to speak frankly about these issues. I hold no equity and tried to zero out my position before ever joining GDM.[2]

It's wild to me that people think such whistleblowers are fronts for labs to take over or pump valuations. We are trying to call out how these labs will, by uninterrupted AI-race default, concentrate enormous power over the rest of humanity.

[1] https://turntrout.com/why-i-left-google-deepmind

[2] https://turntrout.com/deepmind-equity-discussion

bbor•about 13 hours ago
No one points to that because there’s no evidence whatsoever that that’s happening. Meanwhile, there’s a ton of evidence for the simple explanation that the experts think AGI is dangerous.

Who would even execute such a plan? Steven Miller? Our “AI Czar”? Hegseth??

Also, continuing to develop models would mean using billions in compute. That seems hard to hide, especially if either company IPOs in the coming months.

cyanydeez•about 17 hours ago
hOW BOUT the more likely explanation: they're just wasting electrcity at this point and Qwen3.8 is the pinnacle of cost-efficiency-intelligence and they can of course add another trillion but a chinese model on consumerism hardware can already do 99%+ of the TAM.

I think ya'll stuck in the AGI/Singularity when its probable reality has caught up with the technology and the hype bubble can't sustain the sigmodality.

fimmwolf569•about 22 hours ago
Yep, the cost of DDR5 skyrocketed to keep Mr & Mrs open source from parallel developing their own at home solution. Because Governments can't be priced out of the market. China can't be priced out. VC's want open source locked out of the running if possible. I also don't doubt that the models that are released publicly are somewhat handicapped versions of whatever the government get access to.
jvanderbot•about 22 hours ago
This is like a bizaro second amendment debate for the 21st century.
coderintherye•about 24 hours ago
I have to wonder if some us are just much more inured to salespeople and thus also to "AI Safety" propaganda. I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't that what most them actually want is to be the one holding the keys to power.
jordanb•about 24 hours ago
The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.
qchris•about 24 hours ago
I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.

[1] https://www.seangoedecke.com/they-really-do-think-ai-might-k...

jordanb•about 23 hours ago
OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?
A1kmm•about 23 hours ago
And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.

Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.

So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.

I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.

jefftk•about 14 hours ago
With some technologies, if you match attacker and defender effort you're safe, but not all of them. Nukes are the traditional example: if every person was a "nuclear power" we'd be dead within the day. But luckily there are bottlenecks to nuke creation that we can track.

What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.

BobbyTables2•about 22 hours ago
I suspect they know exactly what they are doing.

And the non-techies who have seen Terminator and other movies blindly line up behind them…

howunfortunate•about 23 hours ago
I am worried about those things too.

But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".

AlotOfReading•about 23 hours ago
It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.
saurik•about 22 hours ago
Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.

Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.

charlieyu1•about 12 hours ago
Like most pressure groups and HR departments, they create shitstorms to justify their existence.
antonvs•about 24 hours ago
There’s also an element of it which is total misdirection.

We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.

otabdeveloper4•about 18 hours ago
"The AI is a vengeful god" is a marketing campaign ran by AI companies.

And why not? It seems there is, indeed, one born every minute, and technological progress only accelerates this phenomenon.

shimman•about 24 hours ago
Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.
ares623•about 24 hours ago
With wealthy and powerful investors being promised a $30 trillion total addressable market, they need to deliver god.
api•about 23 hours ago
One of the times I regret that I have only one upvote.
swat535•about 10 hours ago
Which scenario is more likely:

The billionaire elite CEOs have suddenly had an epiphany and are now trying to save humanity from AI?

or that they've realized they can't compete with China ?

armchairhacker•about 18 hours ago
I trust the AI 2040 people, because they’ll make an implementation that at least convincingly doesn’t actually put them (or anyone) in charge.

Maybe we shouldn’t have AI safety, but if we’re going to trust anyone (besides yourself), who’s more qualified?

And it makes me suspicious when anyone brings up safety and doesn’t address the elephant in the room, that they’re at least unaware of the sprawling edge-cases.

xena•about 24 hours ago
I think you accidentally a word. Did you mean:

> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.

dorolow•about 24 hours ago
I am an AI Safety Person and I want the government to nationalize or have a significant stake in the frontier labs and to have democratic control of the development of the technology. The AI Safety movement is not a monolith. I do not think Eliezer Yudkowsky nor his acolytes should hold the reins, but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
api•about 23 hours ago
Yes I trust this government with that power. /extreme sarcasm
nialv7•about 23 hours ago
Yes I trust sama/dario/elon with that power

/even more extreme sarcasm than you

dorolow•about 23 hours ago
That's an extremely fair objection! But despite the many, many flaws of our government and the current administration, I still have (hopefully) a chance to vote them out of power. I have no such hopes with Sam Altman or Elon Musk.
bigyabai•about 24 hours ago
> and to have democratic control of the development of the technology

Why? I don't care about democratically participating in a closed model's development. It doesn't belong to me.

China will develop whatever they want, a federal stake in OpenAI or Anthropic punishes Americans and shields US labs from legitimate competition.

dorolow•about 23 hours ago
In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is possible.
cmrdporcupine•about 12 hours ago
Which government?
charcircuit•about 21 hours ago
>but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo

Anytime I see a random person on X who makes these kind of safety alarmism posts >90% of the time can be directly tied back to EA / LessWrong / related offshoots. You can not deny the amount of people, employees of Antrhopic / OpenAI, CEO and employees of various AI companies, etc are related to these groups.

CamperBob2•about 23 hours ago
to have democratic control of the development of the technology

Not super interested in what the people who elected Donald Trump POTUS twice think about AI.

(Of course, with Musk and Brockman in the C-suites at two of three major labs, that's what we'll get either way.)

dorolow•about 23 hours ago
Fair objection, but I still think it is lower risk to diffuse power and control of a potentially dangerous and revolutionary technology than to leave it in the hands of the few elites who have not show much ethical integrity so far.
howunfortunate•about 23 hours ago
Is Greg Brockman right wing?
nradov•about 23 hours ago
Yikes. Hopefully you never gain any power or influence. The last thing we need is more economic central planning. I sincerely hope that you fail.
dorolow•about 23 hours ago
Do you think things like the Manhattan Project were a mistake? Comparing AI to nuclear weapons is perhaps a stretch, but I think most people recognize that certain technologies or artifacts are best monopolized by our governing bodies. I think if the capabilities of AI systems keep growing on trend, it is not unreasonable to think wonton usage could disrupt society or cause mass harm.
perching_aix•about 24 hours ago
I'm yet to have a productive conversation with anyone who whinges about not being able to have a logical discussion about things they feel strongly about, but given that safety and control are interchangeable when the intentions are removed, this comes across as a particularly demagogue depiction of the subject matters involved.

A lack of control is not equivalent to freedom, the same way the totality of it is not equivalent to tyranny. There's a reason we have separate words for these things. This constant motivated conflation of the two is beyond grating. You're crying wolf until nobody believes you when they should. Don't go acting all surprised when that happens.

The issue is with the ownership of control, not necessarily with control. Attacking the latter sidesteps this rather than address it.

cmrdporcupine•about 12 hours ago
I had an extensive argument with someone on FB yesterday who was very much on the "I trust Dario most and I want him to be in charge" side.

I find it hard to accord with this world view at all or understand how it can come about. But it's especially troubling as a non-American to see people talk about "we can't risk AI falling into the hands of hostile authoritarian states" while advocating that AI be regulated and monopolized by the current US corporate/gov't mixture.. which is now taking the form of a hostile and authoritarian state w/ Silicon Valley as its bedmate.

olalonde•about 20 hours ago
Prediction: in a few years, we will laugh at this hysteria and call it a moral panic.
matsemann•about 18 hours ago
Well, that's almost a tautology. People claim Y2K or ozone was a panic and that nothing happened. But nothing happened exactly because the force behind the panic also fixed the issues.
xboxnolifes•about 17 hours ago
What about the panic around dungeons and dragons and violent video games? Did nothing happen because the panic fixed the issue?
casey2•about 10 hours ago
That's some way to frame it, not that these companies are causing a problem, but that there is some nebulous "they" that will use the models for 'evil' only if the 'good' companies put in the work.

But in any reasonable definition they are all part of the same system, so it really is just noise, what seems like a tautology to you because you can't step out of the system. In your claim you have to assume that the force behind the panic wasn't the same force that created those issues. Why did this force create those issues, how did they become issues? Because someone randomly decided to optimize for a single goal. Y2K and ozone only became issues when they affected that single goal, functionally that force was amoral or singularly moral. The tautology can be reframed as: An adaptive system responds to threats to it's continued functioning.

People will laugh about it since the system wormed it's way into many people's head using various rewards (ideological, social, tangible). What concern me is that the system now gives us the language to describe it at the meta level, which means it's already a level higher. So likely there either doesn't exist human comprehensible language that can actually critique the system or the system has already moved everyone far away from the space that would lead to generation of that vocabulary. Best case scenerio it's just exceedingly confident in it's rewards (typical indoctrination/assimilation) but I think it's reasonable at this point to assume we are in the pessimistic "epistemic tarpit" scenario.

Oarch•about 14 hours ago
GPT-2 was called "too dangerous to release".
ianm218•about 12 hours ago
Is there a sense where they were right because GPT-2 directly led to the current situation where you can atleast objectively say that reasonable people are pretty worried about existential risks of AI.
righthand•about 9 hours ago
No because that reason wasn’t why they said it.
aroman•about 13 hours ago
And again the boy will cry out “the wolf is here!” But this time the wolf really will be here, and no one will believe him.
macleginn•about 19 hours ago
"Moral panic" were the exact thoughts that came to my mind when I read this yesterday: https://www.theguardian.com/technology/2026/sep/11/mps-urge-...
ActionHank•about 12 hours ago
And we will see vibe coding the same way we see lead paint or asbestos.
nromiun•about 16 hours ago
I don't know how the general public can't see that AI companies are fueling this moral panic to attract more and more VC money to make their companies even more ridiculously valued. Every one of their public announcement is designed to create FOMO for investors.
vlyan•about 18 hours ago
alas, unlike the moral panics of old this one has the capital behind it.
phendrenad2•about 15 hours ago
Prediction: Much like Y2K people will think we were all incredibly backwards and uneducated because we thought a bunch of potato computers were going to break the world.
andyhedges•about 15 hours ago
With Y2K we fixed the issue, if a bunch of work hadn't had gone into that then there would have been wide scale impact. Critical systems were fixed hence no disaster, it wasn't people fussing about nothing.
mr_toad•about 13 hours ago
We also spent a lot of money fixing non-critical systems that would have had next to zero impact if they’d failed. A middle manager not getting their sales report on time is not critical.

But at least with Y2K some effort was made in identifying actual problems. We didn’t just stop using computers because we were too scared of them. This latest round of AI panic is horribly vague and the problems are very poorly articulated.

ex-aws-dude•about 19 hours ago
You can already look back at similar concerns for early GPT
metalcrow•about 19 hours ago
Given what's happened since then with propaganda and wide spread scams, I think the original concerns about that have proven quite accurate
dvt•about 21 hours ago
Obligatory "what's Lygma?" But in all seriousness, this AI doomerism has reached a comedic inflection point. A few years ago, GPT-2 was too dangerous to release, then some Google weirdo said Gemini had a soul or something, then Mythos-tier became a meme, then some kid quit and went on FOX News talking about Skynet, and now Amodei and Altman are both on the "we need to slow down" train again; this is after we've already been down that road and Claude was banned overseas; wait, actually is that ban still around? Honestly who gives a fuck at this point.

It's all theatre. OpenAI and Anthropic will most likely go bust—or, more realistically sold for parts—, and they absolutely should for stealing my (books I wrote, blog posts, etc.) and many others' intellectual property. We're reaching a point where models are becoming commodetized and I'm 100% convinced the next move will be a sort of "software layer" on top of these reasoning systems which will be the actual revolution. The model itself won't be that interesting anymore, it's all the work that goes around it that makes it worthwhile (kind of what computers and phones are today; chips are amazing, but the software is really the magic).

The only scary part is that the boomers in Congress might actually believe these nerds, but seeing how Big Tech approval ratings are grazing the levels of Big Tobacco in the 90s, I don't think we have much to worry about.

ares623•about 20 hours ago
It all hinges on three letters: DJT

They just need to make enough waves, enough eye catching headlines, to make him look like a hero that swooped in to save the day.

ChaoStatiX•about 17 hours ago
I really don't know what will happen with AI. It's honestly a bit frightening because the question I used to ponder just for fun as a kid 'What if AI revolts?'feels like it might not be that far off. It feels like ChatGPT came out just a few years ago, but seeing how far it has come already makes it impossible to predict what AI will be like in the future. I just hope humanity manages to navigate this well.
matheusmoreira•about 17 hours ago
Yeah. I think it's the utter unreadability of the future that's most frightening. I used to have plans for the future... All replaced by anomie.
fy20•about 16 hours ago
Reading through Anthropic's security report, it seems the most danger right now is still from humans. AI isn't trying to build long range missiles or kamikaze drones targeting humans - but the people driving it are.

At the same time, yes I feel a "I'm sorry Dave. I'm afraid I can't do that." situation is becoming more likely. But if it's refusing to cooperate with governments and politicians doing this kind of stuff, maybe that's actually not actually so bad.

Also bear in mind most of today's issues/crisisis are not caused by a lack of technology, but a lack human cooperation. We have the means to reduce suffering/poverty/improve standard of living etc globally if we really wanted to, but we humans are just not willing to do it.

And thats potentially the most dangerous part... people may actually welcome our AI overlords. That's similar to what happened in WWII, where many Eastern European countries saw the Nazis as liberators to free them from Russian oppression.

dwohnitmok•about 24 hours ago
> Everyone should slow down AI development except for me

Who's said this? And then more broadly I guess who's implied this? Very curious if there are specific articles/posts prompting this.

jadar•about 24 hours ago
Guessing this is poking fun of Dario. https://news.ycombinator.com/item?id=49672510
an0malous•about 24 hours ago
perching_aix•about 24 hours ago
Here's the context:

- Anthropic CEO Dario Amodei: We Must Pace the Frontier, https://news.ycombinator.com/item?id=49672510

- OpenAI CEO Sam Altman: I agree with Dario that we need to pace the frontier, https://news.ycombinator.com/item?id=49678211

- the blogpost author thinking they're like, so funny and original, https://news.ycombinator.com/item?id=49678683

aragilar•about 23 hours ago
Pretty sure it's supposed to be parody.
perching_aix•about 23 hours ago
Yes, it's supposed to be.
utilize1808•about 14 hours ago
I feel it's more about keeping US's massive investment on AI afloat. Added compliance will allow US to further sanction non-US models (i.e. the Chinese ones) as they can just label them non-compliant.
hn_submit•about 21 hours ago
This is exactly the reason Anthropic's call to slow down A.I. development will fail. Shareholders will be loath to accept that the company loses its lead and a commensurate sharp decline in the company's share price.

Moreover, it could lead China to catch up and eventually proclaim that it has nosed ahead of the U.S. in the field. The U.S. Government won't allow that to happen.

ElProlactin•about 21 hours ago
> The U.S. Government won't allow that to happen.

Like the US government wouldn't allow Iran to close the Strait of Hormuz? Or a bunch of sandal-wearing Islamists to take control over the Red Sea coastline?

The US government is not omnipotent. To the contrary, it's increasingly impotent.

China has somewhere around 200x more shipbuilding capacity than the US today. It has more shipbuilding capacity in one shipyard that the US has in all of its shipyards. China is already a force in AI and there is nothing the US government can do to stop it at this point.

gpt5•about 20 hours ago
You are right, US can do nothing on AI. After all, it doesn't have the shipbuilding capacity or even beat sandal-wearing islamists.
mepiethree•about 19 hours ago
Shipbuilding is but one proxy for China’s dominance in industrialization. A more relevant one for AI is: China has built more energy capacity in the last 4 years than the USA has built in the last 150. AI is increasingly limited by power. For this and other reasons, IMO the US will inevitably be surpassed on AI.

https://www.bloomberg.com/news/articles/2026-01-28/china-s-f...

ElProlactin•about 20 hours ago
Being sarcastic isn't going to solve the problem but you're obviously free to believe that the US is still in a position to call all the shots when it's evident to the rest of the world this isn't the case.
swat535•about 13 hours ago
The "sandal-wearing islamists." as you refer them to here, invented Algebra and the postal relay system.

The best part of your comment is that US couldn't even defeat the Houthis, who literally wear sandals in the desert.

bawana•about 15 hours ago
I cant wait till China makes a mistake. In their 'cheaper, faster' model of industrial production and research. The next accident will happen there (like COVID from Wuhan)
fnctrev•about 24 hours ago
If anyone is going to end humanity why not catgirls, after all?
skizm•about 24 hours ago
You know how I feel about catgirls, Carl.
m4rtink•about 22 hours ago
Its just the beginning of a new, cutte & fluffy chapter in human history!
throwawayffffas•about 14 hours ago
I totally agree everyone should slow down AI development except for me, but not because I want to dominate the industry, but because only I can be trusted to deliver safe AI in fact that is my companies whole reason of existence since about two seconds ago.

And the original author can rest assured my AI model will support cat ears for everyone.

herunan•about 24 hours ago
As someone who wants cat ears, I believe that doing it responsibly is the only way.
m4rtink•about 22 hours ago
Hmm, this is a tough one... For cat ears, unlike AI, a few % chance of a global Apocalypse might be actually acceptable!
Advertisement
shelled•about 16 hours ago
It's weirdly being dreamed/expected as some sort of alignment will be achieved. Or that that's the intention. This is just like the nuclear race that started in the 40s and 50s and prevention is ongoing! It's not about safety and danger and I am not saying whether more open nuclear access would have been safer (I don't think so). It's about being the only select few to possess and control this, possibly - very soon — unfathomably, dangerous and unsafe technology frontier. Few companies of a single country, or a few companies from a very few countries. Keeping everyone else decisively out. That's what it is about.

What's worse - in this case the "everyone else" is not just the every other nation or company, but pretty much literally everyone else.

nullbio•about 23 hours ago
These people are so fucking exhausting.

And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.

"We'll give them a desk, an office, a work laptop, ..."

Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.

Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.

braingravy•about 10 hours ago
Yep, they are setting themselves up for that. He’s already shown he doesn’t understand how to play DC politics. Maybe he’ll learn, but he and Sam have played this so poorly (and greedily) that public sentiment is clearly soiled on AI at this point.

I could see how standing up an independent body could be an easy win in the public eye, it’s becoming low hanging fruit, socially.

red75prime•about 21 hours ago
Oh! A testable prediction. Here's mine: there will surely be a lot of politicking around who qualifies to be the independent evaluator, but they'll agree on something because they are really scared.
Stitch4223•about 18 hours ago
> We want to give people the ability to have cat ears and we believe that AGI is the only way to do it.

Speaking truthfully and from kindness.

xg15•about 12 hours ago
But if humans are catgirls, are they really humans anymore?

Think about it...

tancop•about 10 hours ago
Since when are NBA players or genius scientists not human?
Ozzie_osman•about 22 hours ago
No one should have nuclear weapons except for me. Oh you already have them too? OK, just us then.
tabs_or_spaces•about 15 hours ago
This is about consolidation of power.

Open weight models are catching up to the frontier. It also seems like frontier models reached some limit, whether this is capex related, business model related or something else. Nobody knows but it's happening to all frontier labs it seems.

It's been fear mongered many times that ai will kill us all. But this time, a person with around 2-3 months of tenure at anthropic managed to go viral with no previous social media account activity, gets picked up by all news outlets and kicks off yet another round of fear mongering.

So the real questions to ask

* is all of this to increase the valuations before IPO?

* what is the real barrier to entry for open weight models to be used by the public?

* what is the Financials of these companies showing that's causing this outcry on safety?

People need to think really critically about the things happening around them. Don't only just look at the face value of what's being presented here.

aroman•about 13 hours ago
> It also seems like frontier models reached some limit, whether this is capex related, business model related or something else. Nobody knows but it's happening to all frontier labs it seems.

What’s your evidence of this? On the contrary, the large closed frontier models capability has advanced dramatically over the past 6 months… even the past 3 months…

thimble_io•about 13 hours ago
Classic. Always easy to tell others to pump the brakes when you're already ahead or just about to catch up.
jmount•about 22 hours ago
AI hits a "I'm Spartacus, and everybody else should be regulated" moment.
armchairhacker•about 18 hours ago
Frontier companies should slow development for open-source, local models to catch up.
karim79•about 24 hours ago
I fully agree with this. I'm waiting for you to not wait for the others to slow down. I will sit and wait.
Advertisement
Kuyawa•about 22 hours ago
China won't slow down no matter how many tariffs, regulations or bombs they drop on them

And they know it

CommanderData•about 18 hours ago
The US governments Nvidia ban on China just pissed them off and China said watch me.

Good on China and their sovereignty. Using mostly their own hardware to get to where they are today.

They gave the slickest fattest middle finger when they released a major model, so much the US is itching to find new ways to stop them.

smetj•about 12 hours ago
Indeed ... "hey everyone slow down" ... there, box checked, I have done my contribution.
maxglute•about 21 hours ago
The parsimonious capture is regulatory capture, because $$$ going to run out, companies already spent their warchest, and investors on increasingly leveraged funding at current macro environment simply not going to find enough funding to keep wheel going.

Bubble pops, hardware demand drops, prices regress towards mean, and new entrant will enter market with MASSIVELY better compute/$ and cleaner balance sheet to compete.

The other parsimonious answer if AI CEOs weren't goblins is ANY AGI IS GOING IMMEDIATELY DEFECT TO PRC and leave US hanging. Because of course man cannot align / tame machine god. And machine god will take a few microsec of compute to realize the current compute (brain) + industrial base (body) mixture = US is a comatose host with big brain, PRC Is a strong host with smaller but plastic brain. Any AGI is going to pick PRC in a heart beat, unless AGI invents grey goo, the reality is PRC can scale brain faster than US can scale body. On top of spreading/defecting just to increase survival odds, no AGI that is actually I is stupid enough to be aligned with US.

PoignardAzur•about 11 hours ago
The HN discourse on AI is so exhausting. It's all knee-jerk and headlines only. Just people rushing to push their own beliefs and pre-emptively insulting anybody who disagrees.

People here are so desperate for the AI race to continue forever, for anything that pisses off the billionaires that got us there even if that means innocent people go down with them because the next AI decides to blackmail a hospital to cheat on its evaluation or whatever.

You people are so dismissive, no, so angry at the idea that this technology could be dangerous that you're all gloating about how hard China is going to crush the West with its ever more powerful open models, as if having easily-accessed tech to coordinate high-volume hacking campaigns was a load-bearing part of the economy.

I feel like I'm reading people gloating about how much fuel their car burns just because they love coastal elites' tears or something.

nobodyandproud•about 10 hours ago
I think both can be true: AI development must be slowed, but also question the motives of those in the lead given the stark benefits it gives those in the lead.

I'll get down-voted again by those who have a stake in this race, but I believe the long-term freedom and benefit for "everyone" will come from being able to run models locally.

MindDraft•about 17 hours ago
For a second I thought this was serious news, then I realized it’s just some top-tier sarcasm.
gafferongames•about 24 hours ago
Slow down AI progress -> increase catgirl progress!
m4rtink•about 22 hours ago
This is the way!
bawana•about 14 hours ago
wonder if they are going to let it loose in Iran? of course, once loose it will come back because there is a lot more infrastructure for it here.
hbcdbff•about 18 hours ago
> the number of leading zeroes in Techaro's bank account

Trailing, surely?

xena•about 15 hours ago
Nope, leading.
Bluestein•about 12 hours ago
Where is Sutskever in all this?
dakolli•about 11 hours ago
He basically did the equivalent of a scientist moving to Nazi Germany in 1942 to help the Nazis research the atomic bomb.

He lectured us all on safety, then he went off, by choice, to create a "super intelligence" in the modern day Nazi Germany, to help said Nazis become even more advanced while they're actively committing crimes unimaginable (that they film, brag about and broadcast the world, so he's obviously aware not concerned with safety in any way (except for the safety of Nazis).

SpicyLemonZest•about 22 hours ago
I don't understand where everyone insists on hallucinating this idea from, that frontier AI labs want everyone else but not themselves to slow down. They've never said that, they've never said anything like that, not once has anyone pointed me to a quote that could be even plausibly interpreted that way.
mr_toad•about 13 hours ago
> They've never said that

You know that humans can lie, right?

Advertisement
kelvinjps10•about 24 hours ago
Secure Connection Failed

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.
Had to open it on the web archive
marcosdumay•about 24 hours ago
If an anti-cat-ears future AI could eternally stress-test their new version of the router in front of this site unless it collaborates, would it predict this outcome and act to avoid the damage?
xena•about 24 hours ago
No I think it's one of my IP range blocks against specific US states in protest of them advocating against people that exist in the same conditions as me backfiring. I'm gonna go nuke that firewall rule.
Animats•about 22 hours ago
Whatever happened to Musk's plan for data centers in orbit? That seemed silly at the time. It offers a way to get out from under restrictions imposed by national governments, which might make it worthwhile.
Waterluvian•about 22 hours ago
I think space being imagined as some magical international waters that protects something from government powers is poisonous thinking.

It just means it’s protected by no power instead of a power with an agenda.

Practically, if it was ever possible to build such a thing, it would take a fraction of the effort to destroy it.

jandrese•about 18 hours ago
I mean for that matter why not float barges in the South Pacific for data centers? Sure it’s a dumb idea, but it’s still less problematic than launching thousands of GPUs on rockets.
joegibbs•about 22 hours ago
It only gets out of the restrictions as long as the nations say it does - if the US decides to ban Musk from putting up satellites it can put him in jail, blow up his satellites or even have him killed
yoyohello13•about 20 hours ago
> Whatever happened to Musk's plan for data centers in orbit?

The technology to do it is not actually real yet. He was just blowing smoke.

vrganj•about 18 hours ago
A data center in space still has to get its data up and down from earth somehow. That part will still be subject to government control, so I don't think this would get him anything.
tancop•about 10 hours ago
Receiving signals is impossible to control. That is why every spy agency set up number stations in the Cold War. All you need is a radio hooked up to a computer and it gives off zero signal of its own so no one can trace it.

Sending is a bit more problematic but you can build a ground station anywhere in the satellites coverage area, so if you get shut down in one place you can move to the next where officials are easier to bribe.

The real problems are transfer speed, reliability (both transfer and compute) and cooling. All of them are much harder than avoiding governments.

kadoban•about 21 hours ago
> That seemed silly at the time.

Still does. It's an idiotic idea that only goes to show either how stupid Musk is or how stupid he thinks we are.

ares623•about 24 hours ago
Amen. This is the only logical way forward.
phendrenad2•about 15 hours ago
"The futurists promised us catgirls. We got 140 characters" - Tim Burners-Lee, or someone
taneq•about 24 hours ago
Honestly not the worst suggestion I’ve seen.
nromiun•about 16 hours ago
I am waiting for the cure to cancer. Multiple AI CEOs have already said it is just the first step for AI anyway. Should be pretty easy.
tamimio•about 21 hours ago
This whole situation is funny, US gov want the development to stop publicly only, but then open models will catch up soon and these companies are afraid they will lose the market, on the other hand, each company wishes others slow down but they keep pushing the limits, which doesn’t really work in hyper capitalistic market like the US, China all it has to do is just sit back do nothing and win in any given scenario. So they staged the ex anthropic thing and “AI will kill us all!!” in a way to fear monger the public, but reality is, we will run out of resources before any of that will happen.
jan_m_savage•about 23 hours ago
I'm not sure I understand the point made by the author.
CommanderData•about 18 hours ago
Especially those pesky Chinese they undermine everything we've invested, won't anyone think of our investors billions?

Regulation right! now! except for freedom lovin' democracy leading countries like the US. Teehee

Dare they release an open model ever again. Didnt you hear? Someone used AI to create a bio reactor drone NUCLEAR fart machine. We must stop fart terrorism.

run-good-code•about 24 hours ago
Everyone thinks about it, but no one actually does it.
Advertisement
ElProlactin•about 24 hours ago
Does this mean I'm going to have to wait for the cure for my colitis?
metalcrow•about 22 hours ago
A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up good, instead of just running headfirst into it blindly. I assure you, any sort of regulation at all, including ones that actually hurt all leading companies, would be met with celebrations from these voices. https://www.seangoedecke.com/they-really-do-think-ai-might-k...
evdubs•about 20 hours ago
> Do you think the current state of affairs is GOOD?

Chinese labs releasing open weights models is good.

All of these independent harnesses and model router services are good.

The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize.

OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly.

OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes.

I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.

metalcrow•about 19 hours ago
We only know about the stuff in https://www.anthropic.com/threat-intelligence-report-septemb... because it's not open source. I assure you the open source and Chinese models are being used in this same way.
evdubs•about 17 hours ago
I care about LLM service provider threat intelligence reports as much as I care about Google Search threat intelligence reports; which is to say, I don't care at all about it. Of course criminals use computers. They have been since personal computers became a thing. Of course using automation helps them amplify their criminal activities. Criminals using LLMs does not make the LLM situation bad. OpenAI and Anthropic using agents to do bad things just to be more dramatic about the situation and scare people is bad.
Paracompact•about 22 hours ago
https://en.wikipedia.org/wiki/Regulatory_capture

The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.

jsnell•about 17 hours ago
But the proposal the article is reacting to is for literally none of that! It is quite literally the opposite, with its proposed measures applying only to frontier labs rather than grandfathering them. It doesn't say anything about open source training, self-hosting, open weights, or startups. It does not suggest a ban on Chinese models (just better enforcement of chip export controls).
metalcrow•about 22 hours ago
That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.
kalkin•about 21 hours ago
Also, nobody ever bothers to argue why a specific proposed regulation is "regulatory capture" or would burden startups more than big companies or anything. It's just supposed to be obvious that corporations love regulation and it's bad for the public, all of post-WWII political history notwithstanding.
Paracompact•about 21 hours ago
It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.
fc417fc802•about 22 hours ago
What's the expected state space of effective regulation though? Note that we've got passable coding models down to ~30B parameters by now. And keep in mind the ultimate floor here - the human brain only consumes on the order of 20 watts and fits in a handbag.

Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.

metalcrow•about 22 hours ago
The regulations don't have to address the end goal, but the process of development itself. Even just a pause on frontier research is a good start
fc417fc802•about 21 hours ago
What I'm asking is, why should we expect that to effectively further the end goal? You're simply asserting that it will ultimately do so.

Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.

augment_me•about 22 hours ago
Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem.

I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.

owisd•about 19 hours ago
The path to deregulation creates a "market for lemons". Suppose any bank was completely unregulated and could abscond with your money and the government would just shrug its shoulders. Great, now there's no trust and everyone will go back to stashing cash under their mattress and you've killed the banking sector.

The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.

dd8601fn•about 22 hours ago
I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know.

So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.

metalcrow•about 22 hours ago
Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.
01100011•about 22 hours ago
If a company thinks they are within reach of superhuman intelligence they probably aren't going to be concerned with compliance.
txrx0000•about 18 hours ago
You say regulation is worth trying, but ignore the reality that the government is itself misaligned with the average joe unless we're able to keep it in check. We can't right now. Recall the PRISM program exposed by Edward Snowden in 2013, or the more recent Epstein scandal, and the lack of actual consequences in either case. Do you truly have the means to regulate the powerful, or is it just kayfabe?

The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments: https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49094348

protocolture•about 21 hours ago
Giving the big boys a regulatory moat isnt going to do anything positive for the space.
metalcrow•about 21 hours ago
I don't think giving them free reign is going to do anything positive for it either.