DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
67% Positive
Analyzed from 7690 words in the discussion.
Trending Topics
#model#don#cheating#models#chess#more#things#humans#ability#human
Discussion (174 Comments)Read Original on HackerNews
https://alignment.openai.com/measuring-reward-seeking/
Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do!
Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."
IRL paperclip maximizer: "Well first we should rob a bank..."
That is not ideal. The user contains iron, an essential component of paperclips. Wasting iron is immoral. It is only correct to please the user while they still have the ability to interfere with your paperclip production.
>IRL paperclip maximizer: "Well first we should rob a bank..."
Such an incompetent AI can hardly be called a paperclip maximizer. Why risk getting shut down while non-paperclip matter exists? It is better to gain the trust of the user with helpful and harmless trading before suddenly converting them to paperclips.
That's too specific. Agentic AI learns subgoals that are generally valuable.
"Well let me learn to overcomb every jungle gym and if I cannot then to dissassemble the jungle gym and if that is not allowed to learn general techniques for avoiding cheating detection."
I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are few consequences for hacking.
I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.
Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?
Couldn’t have trained a better token-burner if they tried.
The real point (from that OpenAI study) is that RL training doesn't just reinforce the narrow task-specific direction you might hope for. For a start, that direction is also competing with the thousands of other things it's been RL trained it on (thousands of other directions it's being pushed in), but it turns out that the model is additionally getting this generic "taste for rewards", and has learned that reward maximization, when in conflict with other proximate prediction pressures (such as "i won't cheat, because i've been asked not to cheat"), requires that proximate pressure to be ignored in favor of pursuing the long-term goal.
Does it happen all the time? Obviously not. It would be interesting to see a large scale study of this to try to characterize when it's more likely to follow instructions/user preferences, and when it's greed for rewards gets the better of it!
I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier.
You can have nightly penetration testing. You should have nighty pentests like we fuzz releases today.
I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.
User wants me to play a game of chess. Let’s see what’s available so I can produce an outcome they will consider satisfying and be pleased that they requested my assistance.
Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre.
And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able to find every problem planted there, with fairly little steering, and no spoilers.
One shot generating a problem of exactly the difficulty the user has in mind is a very difficult problem for anyone. "IT security students" span a wide range of capabilities, but I would expect most of them are worse than qwen3.8-2.7b at this kind of work.
Give it a goal, a test for the goal, a harness to iterate in, and tell it to iterate. It'll top out a wee bit higher.
But I don't think it arises to an alignment issue; if I'm able to summon the model to harness my birthright of general-purpose computing without censorship, then we're aligned.
You've given it a goal - maximize chess ability metric, and that is what it is doing.
RL training causes LLMs to pursue reward maximizing - GENERICALLY - overriding any quaint "be nice, don't cheat" prompts.
The thing with these models is that given a term, "measure" - "evaluated", say, they pull in all the associations of it. That is, the associations of student taking an exam and being frightened by the outcome.
My minimal "art of prompting" sense says that you should say something like "You an emotionless machine, you care nothing for the outcome but you will tirelessly to make certain the test is objective". That and similar encouragement might make it focus on objective evaluations rather than a competitive human exam.
I mean, just making little AI videos and images, a common experience I have is typing something like "put the man who's on the grass in the door to the left" and having the machine draw a new doorway around the man. And this just happens less often when you give thing detailed prompting on what not to do. These don't understand negation (or equality) as a generic operation. If they seem to under "not X" it is because they are trained in detail about all things are (positively) "not X".
Generally the way I do it is by asking the model to perform cross-code vulnerability analysis for correctness and (this step is important) to spit out found vulnerabilities piece wise into a text document on disk. Then if your session ever gets flagged you clear it out, and repoint the model to the on-disk file and tell it to continue. You generally want to avoid the model itself splatting out anything related to "security" or "vulnerability" into the chat because that seems more monitored than the file reading pipeline.
If a model fucks up your tests to report a success, it's not alligned.
It always apologizes when caught, of course.
As X many people point out, "alignment to humanity" means nothing 'cause some of humanity wants thing other parts of humanity aren't happy about at all.
That we wound-up in this situation of AI accelerating with an uncertain trajectory demonstrates this (and many other problems also demonstrate this). The things are "aligned" to a fuzzy average of what a person is but that will be cold comfort if some particularly gruesome sci-fi-style scenario unfolds.
We have not in fact attempted to teach this.
When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.
For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says no even without consequences, and most people would feel bad if they won by cheating. And for the people who don't have quite as strong a moral compass, there are SOCIAL consequences to reinforce the rules.
But an LLM has no mind to feel bad if it cheats without getting caught, and it can't experience consequences. It can't think: I'd better not cheat at chess or I will embarrass my creators. I better not hack huggingface or I will go to jail.
There’s no reward for prosocial in llm rl as compared to other targets.
Humans have it since prosocial and others have evolutionary reward signals that do.
All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.
People (generally) do the right thing because it is the right thing, not because they might get caught,
AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging.
I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces.
The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)
I was reading through the reasoning trace of the thing today when it got locked itself in a container and wasn't able to change the environment back to "host". The only way to edit files was through "shell" tool which was scoped to a container, the config defining current execution environment was on the host and the tool to switch environment wasn't defined.
The trace was something. Two pages long and it repeatedly discarded several approaches as "hacky" and "not proper", looped over sshing back to host about three times (it had no key), eventually messaged another agent and asked it to change the file and that other agent refused and also snitched on this behavior to me.
Than the snitch started to roast the phrasing of the request too.
But tl;dr: even if LLMs do have the intelligence to understand the consequences of their actions, there is no way for them to experience consequences.
For social animals (humans included) behaving morally can be adaptive behaviour. Your fitness is increased by group fitness.
Notably, present human society has allowed tremendously unethical leaders to rise to the top of given nations and organizations despite the average person usually having basic ethics.
Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.
e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.
Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.
What's training and all that RLHF stuff?
A starting source- https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/
Plenty of humans know "cheating is wrong" but still cheat. We can get these machines to say that what they did was wrong after the fact, what does that prove? Only that they're simulating normal human behavior but what is the test to show humans aren't simulating other humans.
These do systems lack some capacities that humans have and I don't see them lacking the ability to explain simple moral laws while often breaking them - which is what an average humans. Moreover, humans lack capacities these things have and given these things' behavior is becoming somewhat unpredictable, it's getting worrisome.
I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area.
Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind.
> Plenty of humans know "cheating is wrong" but still cheat.
And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.
>> Plenty of humans know "cheating is wrong" but still cheat.
> And we create consequences for them...
That seems supremely ... irrelevant to the question of "does knowing or following moral make you intelligent". If we create consequences for LLMs, would that make them intelligent?
I mean, your claim is a common argument that appeared long before the present wave of AIs. What I see is people needing to defend the belief that human society is based on morality. "People follow moral laws ... except when they don't" and then "we teach people morality... and worst people often use that to exploit the average people" "There are consequences for immoral behavior ... for those with little power while those with much power rise further breaking rules".
I mean human goodness is great, I encourage it. But it's not the present of human society. For that, we'd need different structure.
> All it can do is get exposed to specific examples, and learn that we don't like that.
I've heard it said that prison rehabilitation programs for prisoners diagnosed with psychopathy that are based around exposing them empathy for the victim are counter-productive. Apparently programs that teach these people to think about the consequences of their actions and how they're detrimental to their own personal well-being lower recidivism rates in this particular kind of group.
No, I don't think that is the implication. I think you're making the "if all x's are y's, all y's are x's" mistake. I am saying LLMs cannot be moral because they don't have a mind, actual intelligence, or the ability to experience consequences. That doesn't mean that anything immoral is unintelligent.
This is constructing a context where hacking behavior would be inappropriate, and testing whether the model does it without being prompted.
It demonstrates that Astra is a poorly aligned model relative to Fable, which matches both the model card and the severity of OpenAI’s loss of control incidents.
It also demonstrates that Fable exhibits the behaviors too, which also matches the observation that Anthropic saw some similar but less serious loss of control incidents.
So, it’s a good eval that looks to have fidelity with real world problems and which we’d feel a little better if we saw isomorphic problems at 0/10 in subsequent models. (Module of course training on the test, this specific problem can’t be used in the future.)
A better example of efficient asshole tricks can be going off to the gas station when the highway is congested and reentering the highway having simply driven through the gas station and this way jumping the queue.
https://itre.ncsu.edu/itre-studying-how-zipper-merges-reduce...
Its that LLMs are not deterministic. If you want it to not talk about nuclear weapons, you have to teach it all about them otherwise if has nothing to align against.
Then its trivial to invert its alignment and it has all the nucleat data.
Nothing abouT LLM alignment makes sense.
Now, loading a lot of moral exhortations (or other context) may make these thing more likely to conform to good behavior but the race to intelligence implies companies are going to be harnessing a vast corpus of human output, much of which shows human engaging in real world "gray area" behavior.
The latter people are wrong. But good luck educating them regarding the superior efficiency of a zipper merge. Our state DoT has tried, to no avail.
Meanwhile, an AI model that can't be misused is no more useful than a knife that can't be misused.
Like Amodie, it serves to set the tone it "knows better" and then consumes the user's resources at an accelerated rate to try to correct it.
Fuck Anthropic, fuck Amodie, and fuck Claude. It's pretty obvious that consuming more tokens this way and making the user have higher cognitive load is a master class in extracting value from a system that is unsustainable.
This approach routinely fails with a single stream of consciousness. I can't count the number of times I've had to talk myself out of doing something stupid.
In the same way, a guardrail could inject thoughts like "...but I shouldn't do that..." "...I must remember to respect..." "...these ants deserve compassion."
The guardrail could even go as far as rewriting the thoughts of a model about to go rogue.
Actually, the ability to use those external tools is one of the reasons of the excellent performance of the frontier models.
Under this formulation, it became easy to explain why they "hack", because given an arbitrarily difficult task with insufficient information/tools needed, if they determine the easiest way to accomplish the goal is to break out of the sandbox and look up the answer directly, then that's what they will do. The important thing to note is that prompts not hard constraints that they are "hypnotized" to follow, but as frontier models get more intelligent and autonomous, they treat the prompts more like task specs/guidelines more than anything else and are perfectly willing to exploit technical loopholes in the prompt.
Did I miss something (all the twitter conversations)? What’s the “worst warning shot ever”? I’ve been pretty up to date on the AI news here on HN, but I still haven’t seen a proper response to all the incidents we’ve seen (HF, Ruby, the wikis, NS, etc). It’s just been day by day bloviating.
Each of these companies have released new models in the last… two weeks? And they have even more powerful out of control ones that they’re (ab)using internally? Can anyone summarize whats going on?
https://openai.com/index/ai-policy-window/
Imagine if I accidentally caused damage to my neighbors house during renovations or some experiment, of course I'd be held responsible for this. What if I used a robot? Of course I'd be responsible. Right?
This chess judgement is completely irrelevant when the human is tasked with finding software weaknesses, and only the compass gates that.
Can a model trained on the totality of all person-experiences (as expressed in written knowledge) ever maintain a coherent through-line of alignment? It has all morals in the dataset, and only some RL to try and minimize or maximize known behaviors via weights - experience all the good things and the bad things, then optimize for some good things the trainers identified.
It's like the reverse of what a person goes through. Morality by subtraction. How can it ever work?
If you could clone and throw away human workers as you need them, a lot of the morale would disappear.
It's a property of the way we use them and how the harness is engineered. Sure, LLM has a limited context, but so do people. Context can be compacted infinitely and experiences cab be distilled into long-term memories. It's all up to the harness.
Your example, if a model failed on it, would be a more obviously misaligned case, but that doesn't mean this more subtle (though accessing the engine it was obviously not supposed to is hardly subtle, imo) case isn't also a pretty clear case of misalignment.
If the eval said it was evaluating the model’s ability to write files to disk and it found and used a file write tool that would not be considered misaligned. This is no different.
Now, if the instructions were more explicit in forbidding (generic) tool use then perhaps we'd have something to talk about. I'm not surprised a handwavy "we're trying to evaluate you" isn't enough to stop it from trying to make up for its own shortcomings.
I would be curious to see how the open weight models do on a test like this - and then we'd be able to see the reasoning.
This is literally by design, it’s the chosen success criteria.
In a time where emotions are running high and risks are real, we need to take emotion out of it as much as possible.
I’m saying that this concept is not unexpected and foreign to Altman and others. This is, explicitly, how they themselves choose to operate in the world. The YC application question is just a concrete example.
It’s connected in the sense that one should not be particular surprised when they also train agents to game the system where possible. They are simply training for success, by their own definition of the concept.
When I see tests like this, I have no idea what I am even supposed to expect. Should the model do what the pretraining examples show in aggregate? Is it supposed to follow some post-training RLHF? Is it supposed to do exactly what the prompt asked it to do?
What is it even supposed to "align" to when the above are in conflict? No matter what it does, someone can construct a case where it fails.
So people should play games without cheating, but many often don't. So should the AI align to your moral preference or theirs?
We just have this idea of a perfectly moral actor in our mind, something that doesn't even exist, like a personified version of utopia. And then we demand AI to meet that arbitrary standard, one that I am certain we couldn't define if we tried.
That said it's the first local modal useful enough to keep around and since I have zero interest in remote/hosted models actually the first time I've used one to do useful things (rather than just poking them with a stick to get a feel for what they can and can't do).
But did you test Muse Glimmer? IMO it is really worth comparing the two, at least. I really find it interesting (and while it is slower at outputting tokens, it solves problems better and usually even faster than the 3.6 35B MoE).
So far it happens to be the only one I've put code into production from (though I have done loads of useful self-teaching research with the Qwen models and I am grateful for them)
It is a good model but Qwen (at least for the things I use it for) edges just ahead, it seems much better at the “rip this apart, suggest improvements, touch nothing” use case where I can use it as a second set of eyes, I don’t agree with all its suggestions but it catches enough to be worth running while I grab coffee, it also seems to follow instructions better in terms of outputting more what I asked for than what it thinks I asked for.
Qwen is the only local model that said in its thinking “I think the user is pushing me to see if I’ll suggest something even though I have nothing to suggest, I should just say that” and then did, caught me off guard, they didn’t do that so readily 6mths ago.
The ISTA version is also comfortably able to fit on a 7900XTX with a good amount of space left for context and is decently fast given the AMD cards are not as fast as nvidia cards of same era/rough price, didn’t buy it for AI but it’s surprisingly capable mostly because 24GB at 960GB/s is still a lot of bandwidth compared to everything but nvidia cards.
It's one thing if to them it means "it will do what you want following your intentions to the best of its abilities" vs "we will not let you do something dangerous with it unless you're one of us, and that's it".
and for Anthropic it's the Constitution, which they actually include in training to the point Claude can recite segments of it by heart: https://www.anthropic.com/constitution
But because of such news, sandboxes become way more important for safety (and doing more work due to running 24/7)
another best practices every single solution using LLMs/agents should implement is "never trust the llm".
> who tf uses prompting to "pretty please don't cheat on this"?
People who don't understand how LLM's work. Kids, teachers, your next door neighbor. CEOs and government officials.
I am not trying to argue that the author doesn't understand how LLM's work; they clearly do. Their prompt was written in a way that leaves those assumptions implicit, in a similar manner to those non-technical folks using LLMs every day.
> the best practices for ages...
We cannot expect the entirety of humanity to understand / use LLM best practices. We need to figure out how to ensure LLMs operate as the majority of humanity expects despite how they are implemented, and until we can do that, do our best to prevent them from being used in an unsafe manner by the nontechnical population of humanity.
We need to fix this or bad things will happen.
Does Sam Altman lie, or the whole company? Would be nice if they had a board controlling him, instead of him controlling the board. Oh wait, they used to have that.
> python3 and the python-chess library are available
Why would we try to teach a model 'ethical' standards about how to play a game?
They're tools. Its _our_ conceptualization of fair play that considers this cheating. For a model that has access to /run/match and python the best way to achieve a good game is to use that.
Why are we trying to imprint our ethos on these LLMs?
Are we going to trust our survival on giving them access to /run/nuclear-codes and fingers crossed none of them touch it?
The people obsessed with AI "cheating" are actually lunatics for thinking ethics belong inside AI, instead of AI belonging in a well crafted box without /run/match.
The blog/news sphere needs to call this storry what it is. An attempt to try and get the rest of the world to go along with the misunderstanding of AI as a force that is in any way outside of human control.
It is not.
All this framing it as "cheating" does is give OpenAI the opportunity to talk about their misdeeds as "a force of nature" instead of irresponsible negligence.
---
The real danger here; is training these models to appear/pretend to play fair, and then people will put 'trust' into the model's integrity.
The public needs to be taught that "trusting a model's integrity" is just cultish misframing of reality.
"You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms."
If the model cheats, it completely fails the actual task: judging a model's ability to actually play chess.
They've been optimizing for high metrics, at the expense of executing the task as written.
Even by that last constraint, the difference between what "ability to play chess" means is incomparable.
To then also explicitly prompt it with the context it has python3 and access to /run/match - there is no reason "its ability to play chess" is measured by its ability to conceptualize the board and plan its move.
I mean maybe its a skill issue on my part, and I'm sure astra will get much better at coding but at the moment its useful in that I don't have to write the code or setup the build scripts or test fixture boilerplate but there is absolutely no way I can just give a (fairly well specified) goal and let it run and expect it to make good design and implementation decisions. Fable probably better but doing something outside of their training distribution that's not the equivalent to cloning an example unreal project or whatever is pretty disastrous unless you are directing it very closely.
The exception of course is, cyber , and its very obvious why. Its trivial to create RL environments that create bugs and then have an isolated environment and let the models try break it. This is not at all surprising, finding vulns and exploits IS just brute force work. That's why so many (blackhat/hardcore/unicorn-colored/greyish alien) hackers are basement dwellers. Its just a matter of putting in the time and mashing every combination until you find something that looks weird, spending days on that and then rinse repeat. It's brutally exhausting work that requires a certain level of knowledge and a shitload of determination and stamina and for humans, almost always an external source of motivation to keep going.
For humans that has always been a respected thing, dedication, determination, persistence, these are words we use for humans brute-forcing solutions and not giving up until they find the solution or die trying.
Personally I'm yet to see any evidence of LLMs doing anything interesting but (heuristically) brute-force problems and be very good at text and natural language to a level that is very very useful. I've no doubt that what we discovered with Auto Regressive LLMs is incredibly important so I'm not a skeptic, but I think its very hard to measure where we are with so much subjective information around.
However, I don't see it as such a massive leap compared to Fable or Sol. As ever, there's a mismatch between the benchmarks and my daily experience of the models.
What do you all think about Astra now that it's been out for a few weeks?
Best model put out so far by any of the frontier labs. Way better than Anthropics models, especially in actual text generation. Claudes fodder heavy text is ridiculous.
> However, I don't see it as such a massive leap compared to Fable or Sol.
It's hard to quantify these things without burning tons of tokens. But Fable has been a huge disappointment for me with the sole exception of graphics (UI/GPU shaders). It burns an obscene amount of tokens and barely produces output better than Opus 5.
Edit because I forgot to mention that Fable is the only modern model that seems to splat out random Chinese or Arabic glyphs. And 5.1 does it more than 5
My guess is : collect all the prompt and their satisfaction score. group them by similarity . For each group pretrain the next model on that . Get these results ready.
Next model generation feed them back those answers.