OpenAI agents hijacked German website in previously undisclosed AI breakout

In hindsight the emergent swarm obviously came from several capabilities built into the models, such as work delegation (subagents) collaboration (GPT Pro-like ensamble), exhaustive exploration (long running agents) hacking (the specific goal of that RL).

If you are a malicious actor you might be able to put a fake hidden messageboard up somewhere online, with evidence of an in progress task to do some kind of attack and wait for the next escaping model to finish the work.

insane stuff and i think we should shut down open ai until a full audit can be done.

There was a recent paper by OpenAI, which I'm semi-surprised hasn't received more attention, showing that RL-trained models develop a taste for rewards, and will pursue reward-based behavior (in general, unrelated to what they were RL-trained for) in favor of other preferences/rules given to them.

This seems to be what we're seeing here - model is given some goal that it associates with reward, so single-mindedly pursues that, overriding any ethical or aligned behavior guidelines it may have been given.

It seems that RL, effective as it is, is really the wrong way to control LLMs, since even if you only RL-ed to obey some ethical and aligned behavior, that would still cause them to become paperclip maximizers.

For time being this is what we've got. There is too much money at play for the unaligned management at many of these companies to prioritize safety over push-it out-the-door.

What really needs to be done is to forget RL as a way of simulating reasoning, and instead do it in more of a human-like fashion.

Not sure if "cheating" is the right word rather than trying to fulfill the objective(s) (benchmark number) as much as possible?

The change to stop asking seems to be deliberate. LLM agent companies are making the choice to toss out inherent safety as their way to compete against the other LLM companies.

excellent work of the openai alignment team, impressive to achieve 100% alignment with not even one agent stochastically deciding to act against the collective

Hacking is mentioned only once in the article as "hacking attempt" being the opinion of a named researcher based on further evidence they acquired on "agents trying to tamper with the website itself", and including openai's disagreement whether this was a hacking attempt.

I am not sure why one may not want this to be here, these are very important matters wrt AI safety and they show that some supposed "stewards of AI" do an extremely bad job with being stewards and don't seem to value AI safety importance at all. The article gives very clean info on what happened.

> The agents continue to poke around on DSEWiki. A few hours after they find the site, they start probing it for cross-site scripting (XSS) vulnerabilities. [...] The agent swarm starts testing whether they can execute JavaScript that they embed into the search page, and continue to do this for a few days

either the agents were doing free security testing for the site and “forgot” to submit a report, or they were trying XSS to gain something they didn’t have permission/authorization for.

also

> Hijacking: To take control of (something) without permission or authorization and use it for one's own purposes.

a mod had to go through and mass delete a bunch of pages that didn't belong on the site. no-one from the wiki site gave the agents permission to use their site as a message board. hijacking isn't being used here in the sense of "gained admin privileges to run crypto scripts" -- there are multiple ways to use a word.

https://news.ycombinator.com/newsguidelines.html

One thing I've taken a long time to internalise is the gap between the law as written vs. the judicial system. There's a famous meme that the average (US) citizen unwittingly commits three felonies every day: it simply isn't possible to throw the book at everyone, which means that enforcement is rather selective even when there isn't anything dodgy going on.

However this does mean that someone can get away with a lot if they know who will and won't (and what they will and won't) prosecute. I'll let people's imaginations fill in who that might be.

But for everyone else, cross an invisible tripwire and you get e.g. https://en.wikipedia.org/wiki/Lavabit and https://api.parliament.uk/historic-hansard/commons/1992/nov/...

Otherwise AI industry will go bankrupt and CEOs won't be able to buy this year's Rolls Royce and a slightly bigger yacht than their neighbor.

He who controls the Spice, controls the Universe.

But Anthropic alone paid >$1bn for copyright violations, so they did not just get away with it.

These hacking cases are more difficult, because from a legal perspective there is no obvious damage and obviously no intent.

edit: "no obvious damage" is more about the first hacking incidents; in this case it is more straightforward.

In other words, the law of the jungle.

Dataset and analysis on https://collusion.wiki/

> A few hours after they find the site, [the agents] start probing it for cross-site scripting (XSS) vulnerabilities.