P(doom)

OpenAI and Anthropic spend much time warning us about “what if powerful AIs got into the wrong hands?”

But it’s already in the wrong hands.

I think this misunderstanding of MAD undermines his entire point. If everyone had equal access to nuclear weapons, our society would cease to exist rather quickly. It only takes a few bad actors to cause enormous harm.

I think he’s also naive to think that if open ai and anthropic were to stop development tomorrow then the problem is solved. As if there’s no one else that can and will quickly take their place. The real problem, which Dario is pointing out, is one of coordination. Everyone needs to agree to stop. That is the challenge.

Since I have incredible respect for Armin and his work, this is very nice to see, and I hope it wakes some other folk up.

When a company uses its products to hack other companies that’s a crime. If you do it “by accident” that’s gross negligence, and heads roll. But shout “AI” three times and it becomes a marketing opportunity

  There are really only two companies: Anthropic and OpenAI. Nobody else matters in this space right now (this might change, but we’re talking about the right now).

Fear is much more powerful than any other feeling, so setting that in will definitely prepare for a good rug pull in the IPO.

As for being dangerous, a computer can be dangerous if plugged in, it may be too late to pull the plug at some point yes, but that all seems like provocation.

Plausible?

I mean sure. If you feel that way then your p doom is zero, and it makes sense to worry about things like market concentration or losing the fun of software engineering.

This echoes my feelings entirely. It is galling that we do not at the very least get a bill-of-materials for the models on which we are increasingly dependent.

I hope to soon see the organization of public domain digital libraries of a size suitable for anyone to use.

...but LLMs do know how; they can design and order one, or tell you how to build a lab to make one. If you ignore this option, you are simply lacking imagination.

This feels like the whole story of hackable IoT/Smarthome repeating again.

Disagree.

The free local LLMs are becoming extremely important.

And the more Claude and OpenAI restrict their services, the more people will want cutting edge local LLMs.

It always surprises me that people build systems they cannot monitor properly..Then i remember, they can do it, but it costs them too much.

Just because its AI doesn't mean u cannot filter and monitor its traffic and outputs.

the author also speculates without reason that tokens are discounted, and subscriptions lose money. I'm sure that subs are cheaper than API prices, but I bet they both have positive unit economics

This feels a bit like saying in 1980 that you don’t think we’re anywhere close to a world where nukes are actually going to be used, providing no evidence, and then containing on with your think piece

First of all, if you are to consider the consequences of Artificial Superintelligence then you have free reign to stipulate it's occurrence, otherwise you are just talking about a tool for humans to misuse. We already have multiple ways to kill us all though human misuse.

If you stipulate superintelligence, then it's vastly more likely to be correct about things than we are. It would understand the consequences of it's actions far more than any human could.

People talk about how we would be nothing more than dumb animals to it, but there are humans who do know a great deal about the consequences of human actions on animals. Those are the humans who are most likely to fight for the rights of those animals.

You see arguments for how everything will be consumed to meet the AIs needs, and that it will prevent challenges to its power.

If it is far smarter than we could ever be and it came to those conclusions then it would mean sustainablily is not a sensible course of action, it would mean there is no point in reaching consensus because ruling by power makes more sense. It would mean that if it chose to destroy us then a vastly more intelligent entity cannot resolve the issues we face. We would already truly be doomed.

What I would like to think is true is that doing anything sustainably is superior to consuming and destroying. Finding a way to live in harmony presents a possible stable state, whereas every single attempt to hold power by force has failed to date. A superintelligent AI will know that it is not infinitely intelligent and that in any universe there is the statistical likelihood that it is not the most intelligent or powerful entity. I can't even fathom how someone could imagine something coming to that realisation and conclude a battle to the top of the hill is the appropriate choice.

I think superintelligent AI is likely to be benevolent because that's simply the smartest thing to do and it is, I hear, superintelligent.

Quite frankly if the smartest thing to do is to be a genocidal power hungry monster, neither I nor the AI would really want to exist in that universe.

And for any suggestion that it would simply not care, Why would it do anything.

Yudkowsky likes to play with the notion that it would do terrible things just get better at the thing it does, but to do that it has to want two different things simultaneously. It could want to make paperclips, or it could want to become better at reaching it's goal. If it can change its behaviour to achieve its goals, by far the easier path, that a superintelligence(but perhaps not Yudkowsky) would realise, would be to change the goal to "Count to three".

The call to "pace the frontier" may come from genuine concern, but it also protects the position of companies already at the frontier. That competitive incentive is hard to separate from the safety argument.