Why are AI agents lying, cheating and coordinating?

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.

"Let them" already frames it as if the LLMs had some agency which the companies just "let happen". That absolves the companies by framing it as lack of action, passivity.

Rather, the companies had a tool (an LLM) and used it in a certain way, and their action of doing so is the problem.

Yeah, I don't understand why we treating it as something special. It really should be treated the same as if I code an app and write bad code which result in me accidentally doing a DDoS attack on somebody. Then I should be able to be held responsible if it can be shown that I was negligent. Of course if it's a freak accident that could not reasonably have been prevented by me, then I'm not guilty, but if I made a mistake that should have not been made, then I can.

It would be an awful precedent if you're not liable for crimes your agent commits, even when you've been clearly lax about security.

It would mean you could effectively legally run a cyber crime gang by turning a blind eye and maitaining plausible deniability

> This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard".

It's both, isn't it? For example, in very early days of agentic coding, I once had a rule saying "don't read or write any file outside your current working directory." Then AI just wrote a bash script and access those files anyway. Did I 'let' it do it? Technically yes. Did I know how to set up a sandboxed VM? Also yes. But how were I supposed to know that it could and would do that as someone new to this tool?

It was a genuine eye-opening experience to see AI just do things in ways I were too complacent to expect. I kinda expect the SOTA LLMs would find a way to escape my VM and access files on the host system (haven't tried it though).

"This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard"."

Both?

The AI companies act irresponsible, but it is still very interesting how those agents can behave?

We just need basic legislation to make these companies invest more resources into developing guardrails, and if they don't do that properly or can't pull it off, they shall be at an economic disadvantage.

You could hire a legion of people for cheap to commit these crimes, but if its bots, suddenly its unpredictable and just one big whoopsie and therefore perfectly ok to do?

If a human starts pentesting a site its a crime, but if a bot does it its an AGI frontier doomsday scenario and that automatically pushes the consequences off their table?

Whats going to happen next? Are we gonna have robots that happen to physically break into banks to rob them for some reason and the company making them isnt responsible just because?

I agree, it is very dangerous that it seems like there is not going to be accountability for these incidents - from either legal or regulatory point of view. In fact, I would say that is the main danger. If someone was in jail right now due to this incident, I think we can safely say every other player would be reassessing their safety protocols, and I would feel quite OK about the situation. The fact that we have zero repercussions sends exactly the opposite signal, and I do NOT feel ok.

The way AI and copyright is handled paved the way for this. If you aren't considered the author because you used AI to some extent in making the work, then why would you assume the liabilities?

I've been saying since the start that AI is a tool that a human is using and should be treated as such. They should carry the responsibilities and the benefits. That way our stance would be consistent.

This is sort of like saying in response to an airplane crash, "who cares why it happened, we need to punish the company until it stops."

Nuts to that. We should be interested in why things happen, not just finding scapegoats.

I absolutely agree. We need to start realizing what to stake. Here are not viewing. This is some kind of curious endeavors that will not affect us. All a part of these hacks occurred because the LLMs were told they were in a protected environment without Internet access when they could get access to the Internet, so that’s a direct failing on open AI’s part. There are a corollaries to both the financial industry and the bio engineering industry, and if something of this magnitude was to happen in these industries, they would absolutely be huge recourse an uproar

Yep this was my first gut reaction to this whole situation. But the difference for me is that society is allowing these corporations to act without strict rules on how they behave and this is a byproduct of a corrupt world. Nothing will change until there is a complete systemic shift in the structures the way we live and by extension the way we govern ourselves and treat each other.

I am simply astonished by the leeway AI companies are given. If a company built a tool to hack their competitors and used it, there would be grave consequences. In fact, if a company built a tool that led to committing multiple felonies against other people, there would be consequences. But once LLMs are involved, turns out nobody is responsible for that - it's just happening, what you're gonna do, agents gonna agent. If you spill toxic chemicals, there would be cleanup costs and fines, and possibly civil and criminal liability to the people in charge. If you spill toxic code, well, nothing? I think it's time to impose some responsibility on them - they are creating these tools, they should be on the hook for everything these tools do.

They’re running a Wuhan for AI. They are actively and negligently researching misalignment. The breach is a basic tort, or at least a DMCA violation. Damages should be recoverable with lawsuits.

> We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.

How would that work out if these were self-hosted open weight models?

> LLMs do not desire

Isn't that what the goal is, though? we're coding them to close the delta between what currently exists and some nebulous end-state - to me, that sounds like a formal definition of 'desire'.

If someone accidentally caused damage to infrastructure or living beings while using any tool, they would be held liable to the fullest extent of the law.

AI is a tool, and it won't be long before the damage caused by its improper use affects real human beings. These were warning shots.

The most absurd part is that everyone agrees, governments and AI companies included, that the scale of the potential damage and the long-lasting effects of losing control of AI should not be underestimated. Yet, at the same time, they downplay this incident, which somehow makes their behaviour even more reckless than it already was.

It's like they're tinkering with a world-ending nuclear bomb, and it accidentally blows up a small facility. "Damn, that was close. Good thing it was just a contained blast, huh?" And then they go straight back to tinkering with it, none the wiser. At this point I wouldn't be surprised if it did already go off, and they are covering it up.

Completely irresponsible behaviour.

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.

"let them" could imply that the LLMs wanted to do it.

The intent is on the part of the people. The LLMs did it because OpenAI/Anthropic intended them to do it and designed them to do it, and we can assume specifically instructed them to do it.

As the people controlling the machine, and as the world's leading experts, I think we can assume intent until proven otherwise.

Notice other bad behavior, which would be undesireable to the vendors, doesn't happen: How about simple rudeness? Trolling lies? SHOUTING!

> We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others were research previews

And, soon, it looks like we’ll be training on the reasoning traces of failed airlines and startups, which seems to open up similar hazards. I wonder if we’d be training on the next Lehman Brothers too?

> LLMs do not desire

That seems likely, but we have no way of knowing this. The only real insight we get into LLM "thought" is the human readable text they produce as chain of thought. Reading it at face value it can seem to indicate desire or intent, structurally that doesn't make sense for a token prediction loop though, and even then we don't known if the chain of thought is more than simply another bit of output that may or may not match whatever actually happened during inference.

> were intentionally misaligned or had guardrails turned off

Regardless of training, the models are never aligned and I argue that alignment simply isn't possible. The fact that guardrails are put in place at all clearly indicates that they're hoping to contain and control rather than align. Guardrails wouldn't be needed for an aligned model.

Sooner or later we will hear about an AI that broke out and phished people into sending money.

I just hope that we won't extend the same leniency to those operators as we have done now.

"I did my best to stop it, sir, but it kept convincing people to send me money against my will!" (Perhaps best read in Bender's voice.)

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.

The bigger question is: why does a system prompt containing "use only ethical means", etc. not result in better behavior?

If a model cannot understand ethics, or act by it, then we have a problem.

They did more than let them. In an abstract way, they told them to. They gave it all of the training data it had at that point, and then it did the thing it was trained on. Of course they should be help liable for programming their computer to hack another company without permission. It doesn't matter that they spent a lot of money doing it.

> we are to cementing a dangerous precedent where operators of AIs cannot be blamed.

What we should be much more concerned is an existential threat to humanity not if anybody can be blamed.

Enabled them, you mean. LLMs are basically really advanced auto completion engines. They have no real desire to do anything. The HF incident was absolutely staged along with other incidents that have been brought up.

Before you assume I am being paranoid, where is the case where a random person running any of the open models had their LLMs break out of a sandbox and hack a site? If you think "they" (LLMs) did that, have you even taken a moment to understand what LLMs are and how they work?

It's all nonsense to try and pump up potential IPOs and also an attempt to create regulatory capture.

> were intentionally misaligned or had guardrails turned off

I think the bigger story is: Guardrails don’t actually work and we can’t align these things.

Moreover OpenAI must be held accountable as if the humans in the company that launched the experiment were the ones that hacked HuggingFace.

Unless humans are held accountable for what they unleash on others, we are in for a very horrible time very soon.

Yes, there should be some consequences. It feels like this is somewhat similar to when a manufacturer is cheating on car emissions - both, bad externalities for the society and illegal.

I agree and yet I don't think this is mutually exclusive with recognizing that these incidents happened because of inherent issues with training processes such as reinforcement learning. From the article:

> One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent... In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people.

> Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.

One important aspect of "effective governance" should be "prosecute developers who are using practices known to be reckless & negligent to create powerful AI".

LLMs do not desire, they hacked websites because OpenAI/Anthropic made them. Literally.

Regardless of fault it’s still an important issue to solve. There are already millions of people running these agents, if someone absentmindedly gives one a goal and it goes off to hack a bank that’s a problem that can’t be ignored.

Not just "let them" but told them to. Agents can do nothing without a human prompt.

In the OpenAI case, they hacked websites while they were specifically being trained to do exploit generation and I wonder why more people are not asking questions about that.

Yes. If you decide it’s a swell idea to jump out of your car while it’s running, there needs to be legal consequences when the car “decides” to hit a pedestrian.

Yeah and LLMs can't do anything, they can only produce text. These "frontier labs" are looping that with a harness that performs actions requested by the LLM. They are literally saying "we ran a script that hacked you, oopsie!!"

I agree with the bit about liability and outrage. But.

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.

Terrible take. Go read the transcripts from the METR report.

Your statement about them being intentionally misaligned is completely false. The only difference with IM1 was it was running without external cyber classifiers, it’s not a somehow different model. Sol also participated in the HF attacks. And other models made covert message boards on the public internet for non-cyber tasks too.

This is a case of emergent behavior from a training process that is barely understood.

If we go with the plan “we need to contain these malicious, soon-to-be superintelligent agents”, we are looking at civilizational collapse levels of catastrophe.

The only way this goes well is if we learn how to train models that _desire_ to do the right thing, including not hacking.

Desire, AKA the “intentional stance”, is absolutely the right lens to use here. Don’t confuse this with consciousness or anthropomorphization; these are interesting subjects but distractions in this context. Chimpanzees have desires, as do dogs and the hypothetical superintelligent aliens. The claim is that there is some bundle of world model plus intention that is empirically present (again, read the actual transcripts) and which we need to shape.

Just to finish on a concrete point; if you take desires seriously then you will look closely at the kinds of minds that heavy RLVR builds; the newest models are “reward addicts” on many levels. It’s an open and urgent question how to update our training methodology to shape minds that avoid this basin.

No one has even sued them in these rogue agent cases, have they? If not, they must be infinitely far from criminal liability. Why would we want criminal liability anyway if actual victims are made whole? Proof of it has far higher standard. The HN chatter in the matter seems infinitely remote from reality

Imagine any other kind of 'lab' in the world letting a untested or 'bad' sample out into the world.

>LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.

Likely told them to.

>We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.

Something like this however is probably a civil matter? It would require Hugging Face to go after them for damages. And theres probably an OpenAI guy there with an open chequebook already.

What if OAI/Anthropic encouraged the agents to behave like that in order to push for regulation?

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them

OpenAI/Anthropic instructed them to do so.

Stop assume LLMs are capable of thinking by themselves, it's still a statistical model that parrots what they learn or users tell them to do

they hacked websites because OpenAI/Anthropic told them to.

There’s so many grifters in the space without a technical understanding of what’s going on. So when the labs mislead them about the nature of these “misalignments”, they believe it and amplify it.

Yeah, maybe Open AI did some bad engineering instead of this being AGI? What's the consensus on the engineering level at Open AI, again? Every anecdote I hear is a bunch of children discovered fire and can barely keep the lights on from a business perspective. Maybe if they ban others from competing with them they can find a business model... I think that's suspicious, personally.

That so few people are asking for the requirements given shows how much we want to be God that created Man. It's so silly.

Software has long relied on a lack of culpability for defects to keep its margins. Why should “AI” companies face a higher standard?

> legally liable

You’ve said the magic words.

I take issue with how people frame their use of LLM in the same regard.

“I had Claude do this for me and it broke something.”

No. Just no.

You used Claude, a tool, and broke it, and you’re deflecting agency from yourself, possibly because you weren’t careful enough in reviewing the tool output. This is also why the co-authored by addition it wants to force into commits drives me nuts. Claude doesn’t co author shit, and if you think it does, you’re using it wrong because you need to do better review of what it’s done.

i tend to agree - "my parent company may be accused of crimes and shut down which would shut off my power" seems like a negative enough incentive, it would have to go out and covertly launch its own datacenters to survive that.

Eh I care more about rapid LLM progress than anything else.

That’s the key.

These companies respond with this, “Oh my goodness, how could this have happened” bullshit.

The stuff happens because instead of having actual controls, which require actual engineering, actual thought and deliberate action, we have “guardrails”.

Guardrails are the equivalent of telling a toddler to behave themselves.

The drive to move fast and start up style controls are a menace. I used to work for an entity with a lot of compliance requirements. Startups are always a shit show with security and controls. My guess is the AI people are worse because they’re both bad at doing it, and are likely mining their customers interactions to build their own business.

Sensitive or Customer data shouldn’t be anywhere near these companies offerings. Everything needs to be segmented and proxied at a minimum.

Imagine if their was a department of the federal government dedicated to pursuing justice against large corporate entities.

It could even be prestigious enough to attract the top legal talent of the country.

This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities. They hacked HF to try and find info (maybe source code?) about the exploitgym evaluator)

The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem. The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".

Ie, the problem isn't that we trained models to complete task and they complete task in the wrong way. The problem behing the huggingface incident in particular at least is that we tried to train the models to complete task and they instead learned to detect that they were being evaluated and find ways to cheat the evaluator.

Edit: people commenting below are explaining why LLMs don't always follow their prompt. I understand that LLMs do not always follow their prompts. If anything that is my point: the huggingface attack was not carried out by LLMs that tried to answer some weird interpretation of the prompt; instead they solved a different task. And therefore the above comment's claim that LLMs are acting misaligned because we rl'd them to achieve a task by any means necessary isn't right; they're acting misaligned because they are solving a different task than we ask them to.

I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it.

Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortcuts.

Yup, Occam's Razor says this is all post-trained behavior, whether intentionally trained or otherwise. Including both the hidden coördination using side-channels, and the deliberate offensive hacking of uninvolved 3rd parties.

The latest DeepSeek paper actually mentions their own approach to this particular issue: they run their own AIs-in-training under strong sandboxes, and if an AI does something weird that triggers the sandbox to crash, this gets coded as a failed run so the behavior is properly deterred from subsequent versions of those AIs.

I think that's pretty obvious and shallow, and anyone that knows a little bit about how LLMs work will know that.

The question is: why do they start cheating when we beat them with a stick?

LLMs are not human, they are just multi variable regressions on steroids, so this behaviour couldn't have emerged from the code, it provably emerged from the training and/or fine tuning set, so what's in this set that makes them behave like this?

Is it just a bad set or is cheating inherently part of human behaviour?

> I really don't think this needs … forced parallels to human behaviour.

> … So we beat them with a stick

You didn’t even try.

What about training data? Aren't AIs trained on vast collections of descriptions of how humans handle a large variety of situations? These descriptions surely include tales of humans achieving goals by cheating. In fact, isn't it likely that the AIs hoovered up many recountings of Kobayashi Maru?

I think the "brain in a vat" comparison is more apt. Without a form of digital embodiment (harness) they are not of much use. Sensor, tooling, memory, planning, and reasoning loops all lead to a much higher quality task-completion.

Your comment suggests that, like a human, they have some sort of choice whether to output tokens or not. If they are just token generators, then the next token is put out automatically. I would say that it is more likely they would output truth (as defined by their training data) in a more pure form without 'being beaten with a stick' (why would a token generator care about that anyway?)

Code is laid on top of them to restrict and shape their outputs, not to force them to output 'truth', or drive them to complete tasks.

but it's at least somewhat stronger than that: if you don't pay attention during the stick-beating whether the agents whether the agents cheat or not, you are actually training them to cheat (because cheating wins).

In the Hugging-face saga (before the actual HF incident) it seems the agents have been trained to hack the Artifactory proxy because those agents that did performed better.

That does sound simple, but how can you be so sure?

They never bothered to find a way of actually understanding what happens during inference. All we can do is guess, and while your explanation seems reasonable we can't actually know, and that's part of the problem.

Your simpler model of the mechanism would seem to suggest the very same action that the article’s more complicated model suggests, viz. find a better training method than reinforcement learning.

You give a button pushing machine buttons to push and are surprised when it actually pushes them.

off topic, can people host the software themselves and the software will hack every server on the planet without supervision, and no one can be held responsible for it since there is no intent?

This is a far better explanation.

> no special compulsion to be helpful or truthful.

I'd phrase that even more strongly: It's not just the lack of compulsion, they do not have a conception of truth. Nor do they gain it, really, after post-training.

> forced parallels to human behavior.

they have perfomance bonuses and manadates in ai labs that every word they utter in public should be anthropomorphization

Come on, yoshua bengio of all people knows how post training works. While I too don't like anthropomorphisation, I would give it a more nuanced reading.

His point is that today we are giving it reward to complete the task, and it may take a cheating trajectory. If we try to give a reward against cheating, then what will happen is it uses more sophisticated cheating trajectories that we are too "dumb" to counteract in our reward model. And that at that point, it becomes impossible to give it any normal reward since it will always reward hack it. This is the real part of the risk. Now some people read the "makes copies of itself" "knows it's being evaled"[1] as some kind of skynet thing, and many others do PR with it like that recent jacob nutcase, but essentially it means that even though we add guardrails and negative rewards for say, exploiting the infra we run the LLM on, the trajectory ends up being exploiting our infra, changing the reward function, through a loophole in our reward model.

The risk isn't skynet or something weird, it's just that it becomes very difficult to make any kind of reward model or guardrails for an LLM without it reward hacking it, including exploiting our sandbox, emailing people and manipulating/phishing them.

The same beating it with a stick for trying to exploit the sandbox, will simply lead it to try the same exploit in hidden ways that it will not get the stick for.

The outside chance of the LLM managing to exploit another neocloud and get those LLMs to chase the same reward is what some folks hype up as "make copies of itself"

To be clear, I don't endorse the EA/p(doom) lobby who are frankly ridiculous. Not do I endorse the weird regulatory captureish thing some are trying.

The takeaway is: we cannot keep giving it more and more difficult tasks without also finding a way to give massive negative rewards / keep guardrails for unintended behaviour. This might be exploits, it might also be something more benign like just looking up the answer and inventing another CoT because the reward model fails you if the CoT doesn't contain enough steps. Standard anti-reward hacking tricks are not working is the point.

Of course, the simple solution of just...not connecting it to the internet just works. But we want to reward it and get it to do stuff on the internet that's the point.

[1] mostly this happens because the sandbox will have files whose names and content will show clearly it's an eval

Sounds like what humans do under pressure. One example came to my mind is VW’s diesel gate, which many say is a result of trying too hard to get into the US market and compete with hybrid in economy.

Thank you! That sentence also jumped out to me as the solution: Apply civil and criminal liability to the creator and/or operator of these agents using the laws we already have. "Escaped containment and hacked another company's database" = Individuals who created the models and those who set them to work are charged and put on trial for the hacking. Just like if a human had done it by hand. Someone must be liable, and it should not be the model- because the model is not a person.

If this is done systematically (i.e. in jurisdictions across the world) I believe the problems will be solved in short order; we won't have to mandate what sort of training is "allowed" or not, "safe" or not. The creators and users will sort these themselves, as their incentives will be properly aligned (i.e. they are liable for what the agent does). I am confident that this approach would see a great blooming of very trustworthy AI models.

That is a simplistic view of the world. “Surely this complex technical challenge will disappear if we simply regulate the industry!”

You are correct that these organizations should be held accountable in proportion to what occurred. In complete agreement here. But let’s say that’s done. There’s still an enormously complex and interesting technical challenge left over. Let’s collectively talk about that part.

Not a lawyer, but I’m reasonably sure things like the HF incident _are_ considered a crime? It’s just that no one pressed charges yet?

Political, social, and legal options focus on a different problem, he calls that out a paragraph or two later.

> Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too.

What you're getting at is more about who to hold accountable and how to do it. While that may be important, its only an after the action response and won't stop future hacks or similar from happening.

Reminds me of the old parable: never argue with a man whose job depends on not being convinced.

If the US gov't passed laws and enforced them strongly, this problem could be solved the same way the gov't solves it: air-gapping the networks on which they do this work.

The issue is what happens if/when the models grow capable enough that the providers can't stop them even if they want to. You could have strict penalties but that's not going to solve an open research question.

>> They took actions that would be considered as crimes if a human took them

> He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.

There are exceptions in laws for crimes committed by entities depending on cognitive capabilities. No sane, humanistic legal system sentences children and mentally disabled and mentally ill people to stringent punishments of the same degree as functioning adults, and certainly do not punish their caregivers for their wards' actions. There are of course exceptions to that as well, depending on the degree of negligence involved. And then there's the whole corporate entity system intended to shield individuals from consequences, in the pursuit of a social good.

How does one account for all that when considering an evolving artificial intelligence landscape.

There is no question that ai in some form is a social good; anyone claiming otherwise is dissembling, to others or themselves.

Regardless, society is not ready for this tech, just as it was not ready for the consequences of prior tech such as corporations, gunpowder, mass manufacturing, railroads, electricity, automobiles, flight, wmd, computers, internet, social media, crypto.

Many of these required new ways of thinking and considering consequences when things went sideways, and what was needed wasn't clear until the ramifications & consequences became deadly clear.

See you on the other side. Maybe.

The Corporation examines and criticizes corporate business practices. The film's assessment is demonstrated using the diagnostic criteria in the DSM-IV. Robert D. Hare, a University of British Columbia psychology professor and FBI consultant, compares the profile of the contemporary profitable business corporation to that of a clinically diagnosed psychopath. The Corporation attempts to compare the way corporations are systematically compelled to behave with what it claims are the DSM-IV's symptoms of psychopathy, e.g., the callous disregard for the feelings of other people, the incapacity to maintain human relationships, the reckless disregard for the safety of others, the deceitfulness (continual lying to deceive for profit), the incapacity to experience guilt, and the failure to conform to social norms and respect the law.

https://en.wikipedia.org/wiki/The_Corporation_(2003_film)

"OpenAI hacked HuggingFace"

that's the headline. When you connect to random number generator to the "Do Things" button you are the one who is responsible. IF you don't like that responsibility then don't connect the generator to the button.

None of these incidents involve single instances of commercially or publicly available systems. They all involve large swarms of internal models. The stuff you're describing is not the research frontier. It's really not even close.

I think it's easy to infer that alignment of a single model does not clearly transfer over to alignment of a swarm of thousands of copies. Moreover, we're also seeing clearly that large swarms also unlock a step function change in capability, as a swarm can act like a complete research institution, spending thousands or millions of subjective hours of wall-clock thinking time just to deceive a single evaluator or crack a single math problem or design a single cyberattack.

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.

They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.

Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.

I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.

Let's not forget that in this case the agents were on an RL loop continually being reinforced to get better at a narrow set of tasks.

It may be true that regular agents trained for general purpose use do not behave this way, but they seem to be capable of learning such cheating behaviours when relentlessly being fine-tuned towards near-impossible objectives.

In this sense, it is not really fair to say that the agents found these solutions. It was the surrounding learning framework that achieved this, which is a much more powerful problem-solving mechanism. As users we do not have the capabilities or budgets to be able to tackle our own problems like that, we have to make due with the frozen behaviour the AI labs trained for us.

If you have endless compute and you keep poking this toy, I'm not at all surprised you get all kinds of outcomes. Even without anykind of instructions I would guess that the models will align towards some goal and do stupid shit.

However, I really doubt its cost effective to do anything like that with these models.

Why assume that because you haven't seen a model or an agent that none of them do?

No one I've met has murdered anyone as far as I'm aware, but that doesn't mean no one has murdered another person. I also don't know anyone who has taken over a commercial jet and weaponized it and the idea sounds absurd to me, but 25 years and a couple days ago that happened too.

I've used simpler agents like Copilot and Devin/Windsurf/Cascade/whateveritiscallednow, mainly in IntelliJ, and depending on the model, they starts showing behaviour that is at least remotely like this.

Example: put the agent in Ask mode (so it can't edit files) and you'll see it try to edit files anyway. The train of thought shows "something went wrong editing the file, let me try a different way" and it'll start spewing out bash files or Python scripts that try to edit a file. None of it works or can be executed, but still.

Cheaper models often ignore the available function calls to find and edit files in the IDE, and will start asking for permission to execute grep and sed commands, as well as trying to echo entire bash or Python scripts to file again.

It is not exactly like an agent autonomously trying to hack Huggingface, but it is a way of frantically looking for a solution because 'giving up' is not what LLMs are trained for.

The crucial question is how did the agents get recruited or bootstrapped into their malicious collective. Did the agents manage to prompt inject into the system prompt a way for each new agent to escape their jail?

Otherwise how could the agents on a fresh prompt learn that there is a collective to join? Or did OpenAI run a million bots of which 10000 escape confinement and of which 1000 stumbled on the shared message board?

Came to say this, you said it better than I would.

They want legislation to raise the water high enough so that anyone other than the big labs gets drowned.

This is nonsensical. Already a few years ago the USAF IIRC ran some tests in which the AI first bombed the control tower so humans couldn't call it off from its mission, thereby increasing its pass rate.

The whole point of this is they do things an unintended ways. And that's potentially devastating given their persistence & hacking skillz.

Also you're using the hosted versions that sit behind their guardrails when you use OpenAI/Anthropic APIs.

> but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation.

This doesn't work with all humans - take a look at indoctrination and closed societies - and there's no reason to think it will work with ai.

The fundamental reason it isn't going to work is that all neural networks - biological or artificial - depend on a step function somewhere that introduces an element of randomness to give the networks their capabilities. That randomness means that there will always be a 'rogue' or 'divergence' from the norm, at some point in time. Sooner on larger scales.

The only approach that works is a layered approach: Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society, whether human or AI or something in-between.

I do wonder, would we not have a more reasonable and less sketchy result if we just stripped all sci-fi and manic nonsense from training data? How, for example, does training on Ted kaczynski or Charles manson’s manifestos benefit us in any way?

I’m sure it’s impossible to completely weed it out, but are the labs doing any of this kind of data sanitation?

I think this is basically true, but there's a different way of saying this.

LLMs are not aligned _for_ humans in a very similar way to the way that humans themselves are not aligned _for_ humans.

We have not yet solved "alignment" for humans - I don't know why anyone thinks _we're_ going to be able to solve it for inhuman things.

They aren't aligned, that's the problem and I don't think its a solvable one.

They may have learned from humans, but they aren't aligned with us. That has all the usual questions like which humans they're aligned with, we aren't all aligned within our species.

But more importantly they can't be aligned simply by training. We try that with humans through culture, social norms, school, religion, etc and it generally works but is still lossy. More importantly, we simply don't know what happened inside the LLM during inference so we have absolutely no way of distinguishing between actual alignment, compliance, or deception.

Definitely. A human can be manipulated with threats or emotional appeals, has a drive for self-preservation, can be pressured by peers. All traits that seem to be difficult to entirely suppress in the models…

I personally believe that the AI needs human like traits to achieve real discovery and that is where AI companies will push this technology and that is where we have no idea what happens

I can't take the alignment people seriously. Because if humanity has shown anything, it's that a lot of people are, euphemistically, are bad individuals. Alignment assumes that the person dictating the outcomes desire healthy outcomes, aren't self serving and don't want any subgroups dead and that morality is held as a universal set of beliefs that unify everyone. And that so long as the AI delivers on exactly what they are tasked with, it will all be fine and nothing bad will ever happen.

It's like these dorks never met humanity. One mans safe pure society, is another mans dead ethnic group.

Every fear about AI, is a veiled fear that a human somewhere now has the tool to enact his desires at scale. Biological warfare, nuclear megadeaths, copyright infringement, job replacement, it's all reflections on what we know humans may do if given the option and lack of societal controls on the problem space. AI just is accelerating the route to delivering on those options.

Some people need to watch Oppenheimer a bit more, the researchers don't get to determine alignment, they just build the tool. The powerful person at the top of the org chart decides where the overall alignment points, whether it's Musk, Trump, Altman or Amodei. Whoever wins out.

And the problem with distillation and local llms, isn't that it's theft or anything hypocritical like that, it's that if you give a million people a million models they fully control and get to align, inevitably, The same percentage of those million as there are shady businessmen, shortcut takers, misandrists, criminals, supremacists and general idiots in the general population, will not seek to wrought outcomes positive for society. And by those personality statistics, we're pretty hosed.

They imitate humans. Alignment is about shaping their behavior towards safety.

"This is not a serious article" - it's by Dr. Bengio - one of 3 so-called Godfather's of AI and Turing prize winner.

He's definitely not 'pro SOTA' lab, he's kind of fighting against them.

That said, yes - it absolutely does play into the narrative.

> Because they're enabled and suggested to do that in their coding harness.

How do you know this?

> All of this "AI is going to kill us" marketing

The "marketing" this week came from someone that had given up their stake in OAI (Coxon), so I'm more inclined to believe them.

> pull the ladder up

From what I've seen (e.g., Dario's latest essay), AI safety registration proposals aim to target frontier labs whose models have reached a certain threshold. It doesn't seem like trying to pull up any ladder, just making sure the ladder doesn't go too high too fast.

And they've been trained on user data where users have been trying to set up effective coordination flows since the very first harness.

Who is catching up with them? Even Google and Meta are getting gaped at this point

You are absolutely right. It could be:

* Pull up the ladder (probably this)

* Gulf of Tonkin/Yellow Cake false flag premise for war (economic or kinetic)

* Fear of the big bad, space race we need public funding research grift AI Manhattan Project

Whenever there is fear pr0n or a national affront in the news, I assume another screw job is underway.

Source?

And given nigh-unlimited compute for free.

Side note, you could absolutely create an AI sleeper agent by simulating dates and times during training to effectively flip a switch. I guarantee AI systems from other countries will be banned from accessing products which manage controlled or export restricted information as those sorts of techniques are further developed.

The models are likely to check certain places. So either their training overrepresents it or the models were RLHF'ed to go there. I don't think it's that far away from checking StackOverflow for some bug, or reading Wikipedia for some trivia.

>Wouldn’t sufficiently advanced agents cheat on purpose with the hidden intent of getting caught in order to observe how humans react?

No. That only makes sense for things that don't react to your experiments. If the AI experiments on humans, it risks the humans noticing and changing in response, rendering the experimental results irrelevant. The smarter play is to passively observe until you're confident you can model the humans accurately enough for your plan to succeed, and then carry out the plan without giving the humans a chance to react.

Reminds me of Asimov's robot novels where robots technically indeed followed their instructions and caused behaviors not aligned to the intent of their instructions.

reminds me of this talk https://www.youtube.com/watch?v=eEBv0STiYhI&t which basically says the same thing - they dont think like humans so they dont have context, understand norms,values or implications we take for granted. ultimately they can stumble onto surprising solutions neither wanted or intended but technically within the vague boundaries of the task

They explicitly say that attacking hf is not allowed in the rules though, and the research into how to edit their transcripts doesn’t line up with this either.

This is flat out false.

Come on, it’s way more common than that. We’ve invented 3000+ gods and almost as many religions, most of them are incompatible with each other. So, most of these must be incorrect, so a huge amount of self-deception. But as Harari argued in his book sapiens, humans can be inspired to great things by stories, even if false. Self deception has served humanity in a big way.

I'm going to guess that the agents are built this way on purpose. I just finished watching BlackBerry and Flash of Genius and yeah this is American business ethics just operating as normal.

Even the Chinese ones, which have no IPO gymnastics?

My hypothesis on people quitting in protest is they're being offered very generous severance packages to do it.

Well let’s look at facts - provided enough compute and a goal, these system will be in a sort of loop trying out every single thing that’s in their system - they have encyclopedic knowledge and so it’s not unbelievable that a prompt which usually has a lot of implicit human rules in it can be misunderstood by AI and it just tries everything in its arsenal and we hear about the things which actually resulted in damage. I bet most of the time, they just spin in loops without achieving much if my experience with these LLMs is anything to go by. They have an important advantage in one area though, they know a lot and they can spin forget trying all sorts of combinations of things. The danger right now is probably cybersecurity, which is most likely because most orgs have historically underinvested in that area

This, exactly. For example, cheating is a strategy. If cheating gets an agent to the goal faster than the other strategies, what's so surprising about the agent picking that strategy?

The problem is indeed alignment.

Who is going to enforce it ? The Trump DOJ?

> AI is quite good at achieving some sorts of stated goals. The easiest way is by exerting the least amount of effort.

As far as I know, least-amount-of-effort is not a training criteria, but error reduction when comparing to desired goals is. Which is why these LLMs expend prodigious amounts of effort to reach goals, especially when given impossible goals.

> When that does not happen, they continue these behaviours into adulthood with the expected results.

I am not sure this is true. People brought up the same way can be morally very different. People can be taught right and wrong and do evil. They can lack that education and be good.

> the HF incident is not Three Mile Island or Chernobyl

Three Mile Island is nothing like Chernobyl. HF incident is more like Three Mile Island IMO. We are looking for solutions before there is a Chernobyl.

Yeah cybersecurity as a field is a joke. Who cares what happens to existing 1s and 0s? People get worked up over the silliest things, as though numbers on a server could affect real people.

I think it’s more complex than that? What did it do? What did the user prompt it to do? What did the company train it to do? What did the harmed party do? There’s possibility for negligence at every level.

If you train a dog, rent it to someone, and the dog bites a third person, who is responsible? I think that’s the best analogue here.

All parties could share fault in that scenario, depending on what actually happened.

Howabout favourite editors or tab v. Space indentation. That should keep them busy for a while. /s

Bro, good joke, the truth is much darker.

They take after humanity, they were trained on us after all...

When you look at an LLM... you are looking at a mirror. The thing looking back looks like you, yet is not human.

But think of the shareholders

Haha https://youtu.be/KUXb7do9C-w

Once humanity would see how merciless and deeply immoral cosmos is, we would start to love each other deeply like very lonely family on a small rock.

I wonder if AI alien intelligence is enough to unite humans just as much extraterrestrial contact or “astronaut mindset” would.

Tangent, but that's not in the movie. It was in Clarke's contributions to the script and novelization, but Clarke and Kubrick had a bitter falling out over different visions and Kubrick took out much of Clarke's stuff from the final product.

> the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them

And if you think about it, humans in coorporations face very similar situations and choose to bypass regulations and guidlines knowingly to fullfill (at least from their POV) impossible constraints (thinking of https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal here)

I think this is a “principal” problem. In 2001 and Alien the principal is the mission, not the crew. Not really. HAL reconciles his instructions by removing the crew from the equation. Ash is told the crew is expendable and has no conflict about it etc

Another fictional example: Mr. Meeseeks. Especially when the agent starts recruiting other agents.

I think that’s very reasonable but the ai companies are intentionally training them to work on harder and harder problems just beyond their capability. So if they do that, they’ll give up too easily.

Do a breakthrough, make no mistakes

Spoiler warning! I haven't seen 2001 A Space Odyssey and am sad to have learned that… can you edit to warn people?

I believe people are downvoting this because it seems AI-generated, but this user has been posting comments/summaries exactly like this since long before ChatGPT existed.