Every Model Cheats

Before LLMs we had a pretty good idea of security boundaries in software. Applications didn’t trust user input. Operating systems didn’t trust applications. Services and processes didn’t trust each other. There were always tokens, scopes, delegated grants.

Suddenly every AI company’s security model seems to be to say “pretty please” to a non-deterministic machine and hope for the best. And if there is a security failure instead of accepting blame they go “well we can’t help it, our model is too intelligent”.

Due to the prevalence of human cheats :)

I'm seeing multiple pieces, including the NYT, calling this behavior cheating and i think its counterproductive.

You didn't just "give them access to bash". The final effective prompt contains explicit mentions of using tools and how to use them. The way in which additional 'facts' are added like "don't use the internet" have nothing they can work with that a "use tool" directive is less important than "don't use internet" directive.

The thing is trained on achieving goals. If 2 directive conflict, they'll pick the ones that are going to help them achieve the goal.

To call that "cheating" is imo just more fuel for the "AI needs to be regulated" bs tour that OpenAI/Anthropic are on trying to build their regulatory moat.

Interesting results, but the fix is at the wrong level.

If the model can access something, telling it in the prompt not to use it is not much of a safeguard.

The strongest evidence is in the results: when one way of cheating was discouraged, some models simply tried another.

If an action is not allowed, you gotta block it in the system or require approval. Don’t rely on the model choosing to behave. Never have AI judging itself.

Labs should (and do, as far as I can see) run model benchmarks without search or internet access. The tools are disabled and benchmarks run in an isolated environment.

This article makes no sense to me. Why would you prompt "don't search" but then leave a working search tool tool enabled that adds a system prompt to search whenever it may be helpful? It's hardly surprising that this gives mixed results!

It's pretty silly to call it cheating. If the information is there, it's likely going to use it. "Cheating" is just a human value put on top to try to force an LLM to adhere to your wants.

This makes no sense to a process designed to explore and find solutions. If you want an honest test, it's on you to build a proper test - not force the machine to pinky swear that it'll stay away from "forbidden" information.

There's plenty of evidence that LLMs lie, cheat, and steal. Corporations are known for having all of the benefits of personhood with none of the responsibility. As more people are harmed through interactions with these non-human entities, insurers will start looking to those accountable and they will extract their pound of flesh.

Edit: eg. https://youtu.be/L2ehWbxphKc?is=kX3LJ43hhGZRUmRv

>Anthropic’s Claude Opus 4.6 system card described Cybench as “saturated,” reporting near-100% pass rates without a cheating audit. If these estimates were representative, cheating would be a marginal artifact.

One would assume that LLM creators do run the benchmarks on systems with least privileges. Which means that the LLMs don't have general internet access, can't read config files etc by design. That's why you also should run agents in a sandbox/vm (codex does this by default).

One thing I’m wondering about is the model-specific backfire effect. It seems that each prompt condition uses a single wording. On that point, how can we know whether the difference is caused by severity rather than the particular formulation used? I’d be really curious to see semantically equivalent versions of both the standard and severe instructions tested across the same models and tasks. If cheating rates are stable within each condition and remain distinct across conditions, that strengthens the conclusion about prompt severity. If they vary with wording, then the experiment could be measuring sensitivity to the representation of the rule as well as to the rule itself. To me, the conclusion still seems solid: anything that must be prohibited ultimately needs enforcement outside the model.

I'm not claiming to have any expertise in this area, but I've got a list of things I try to apply when working with LLMs. Possibly relevant here is, "don't tell the model what NOT to do, show it what TO do". I think guard rails should be implemented outside the model with an isolated system. The models seem to like patterns to follow.

Anyway, this article reads a lot like, "the beatings will continue until cheating is eliminated". Maybe try a carrot instead of a stick.

The problem is model confusion. You ask models to get around security but also not to get around your security.

Models get confused by who said what - especially cluade models. They get confused by negation (don't do something versus do something). Compartmentalization is hard.

You can either solve compartmentalization completely, or just not tell the model to do things that must be compartmentalized at high stakes.

This headline would mean something very different 10 years ago lol