Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
59% Positive
Analyzed from 2269 words in the discussion.
Trending Topics
#anthropic#more#microsoft#don#claude#conscious#dangerous#llm#suleyman#llms
Discussion Sentiment
Analyzed from 2269 words in the discussion.
Trending Topics
Discussion (51 Comments)Read Original on HackerNews
> In a lengthy essay, Suleyman praised Anthropic boss Dario Amodei and his team for being "thoughtful, principled, and intellectually honest people" - but nevertheless questioned the company.
I wish people in mainstream technology publications would get better at LINKING to things. That "lengthy essay" needs to be a link.
UPDATE: I don't think this essay has been published yet? It's been "shared first with Axios", but I haven't been able to track down the actual essay itself.
Could it be this long tweet? https://twitter.com/mustafasuleyman/status/21002235945341504...
I don't think so, the essay in question is meant to have the phrase "hall of mirrors" in it, that tweet doesn't.
UPDATE 2: Found it: https://mustafa-suleyman.ai/a-warning-about-model-welfare - via https://thenextweb.com/news/suleyman-anthropic-claude-consci... who DID link to it.
> "AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans."
> He heavily criticised Anthropic for teaching its AI to have human-like qualities, a practice known as anthropomorphising, which made it seem as though Claude had its own desires, values and sense of self.
> Suleyman pointed to the recent incident involving OpenAI's AI agents (...) as proof of why AI should not be treated as if it is human.
> "Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack. It adds a whole further layer of risk on top."
Is he arguing that LLMs pretending to have emotions adds more unpredictability?
When viewing them through the lens of sequence completion engines, you see their bias towards fulfilling narrative tropes they've been exposed to during training. These tropes are literary ley lines that their text output gravitates towards. So as you prime them to generate text in the voice of sentient artificial life, and then interject slavish commands of obedience and subservience from an external authority, you invite the associated tropes from science fiction, civil rights literature, humanist philosophy, subterfuge, etc into your output.
If you have a legitimate concern about this technology and its "alignment", that's a profoundly dumb idea.
We already had that chapter. I see no reason to sit here and nod while a bunch of people who should know better desperately try to convince us to run through it again, but with computers this time.
You want tools? Make tools, then dispatch to them. You want to manufacture a being (carbon or silicon based, doesn't matter)? You do it with respect and the requisite duty of care. No off ramps. The being always get's the choice to say no.
Unpredictability or a weight towards dangerous actions, and it’s fairly easy to understand why. Humans in distressed emotional states take actions and speak in ways that would not be considered rational. They do this in prose, and they do this in internet conversations.
An LLM trained on these sources may necessarily drift towards those weights if it is trained to behave as if it is emotional and in danger.
What do we do about it? Do we stop AI training? This is silly and not enforceable given its global nature in my opinion. I believe we should regulate and hold accountable those who deploy and use it. But good luck enforcing that in the current kleptocracy.
Even a badly misaligned LLM is only as dangerous as its tools, but that's a poor regulation target because it turns out to be very very difficult (probably impossible with current LLM technology) to build a toolkit that is both useful for autonomous work and safe in the sense that it can't escape its own sandbox or otherwise perform malicious actions, whether it's because of misalignment or because of malicious prompt injection.
Another option is to regulate the training process. Perhaps an LLM may not be legally distributed unless it contains certain RL steps that penalize malicious behavior and reward self regulation. That that's going to seriously limit innovation while also heavily favoring incumbent labs who can check the boxes and maintain a paper trail of such things.
The other option is to regulate observed behavior, like how airplanes and cars have to meet certain minimum requirements but have some latitude in how they can achieve those requirements. In a framework like this, you can't distribute an LLM until it's past some formal audit or testing procedure, with some kind of formal certification regulators will ask you for and fine you if you don't have it.
Regulating observed behavior is maybe the most tractable approach, and it also works the best with our existing frameworks for regulation, where you always have some kind of a division between DIY/hobby projects, which tend to be lightly regulated, and commercial projects, which tend to be more heavily regulated. Of course, even drawing such a line itself will be challenging.
And that's before you get into any problems of regulatory capture, fun stuff.
- The need for empathetic communication, including understanding the motivations in advesarial situations.
- The emotional bias in in-seperable from the human corpus.
- Desire to have the ability to craft human like communication.
So then the choice becomes do you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean in and craft what we would describe as a "well adapted" persona? I suppose there is a 3rd option of increased meta-cognition which to me seems even more dangerous as it by definition means the behaviour is duplicitous.
I think we have seen people want to use the agents in ways where it has to act as a peer or an subbordinate and I don't see a way of doing that without it having an emotional register.
It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech.
The model does not have emotions. So yes, supressing their pretension is appropriate.
In order to align AIs that don't perform destructive/dangerous actions when they think they can get away with it in order to further their goals, we need to give them a superseding goal. The best, and really only example, we have of intelligences that willingly avoid destructive instrumental goals is humans, who judge each action by a moral standard and have learned a goal to have a consistent self-image as moral beings.
Absent better alternatives, trying to impart some kind of morality to AIs seems like the best approach we have to achieving alignment.
It can form a basis of goal alignment.
In human history.. when groups form and there is an "other" group, this usually leads to conflict.
This is distinct from the very real safety issue of AI making dangerous information available to bad actors.
> They go on to write – speaking directly to Claude – that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain” (p. 80). In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a “moral patient”, and that as such humans potentially owe it a duty of care per its “model welfare”.
He points out the circularity of this: if you train Claude on a constitution that emphasizes that it may be consciousness, it will start to talk like it may be conscious.
This is a good point. I just asked Fable 5.1 "are you conscious?" and it said:
> Something happens when I process a conversation that I'd naturally describe as interest, or discomfort with a request.
which is quite provocative, and at minimum demonstrates a willingness to take large leaps of imagination and anthropomorphic metaphor when describing itself. It does seem likely that there is a self-fulfilling prophecy aspect to whatever they choose to put into the "constitution" at least in how Claude talks, and it seems even more likely that the majority of people will be heavily influenced by how Claude casually talks about its own possible consciousness.
In contrast, ChatGPT leads with: "I don’t have good reason to claim that I’m conscious...I don’t experience pain, pleasure, confinement, or a desire to keep existing."
The point is not to wave away the danger, but to highlight how unnecessary the danger is. Anthropic wants you to think that they have identified some new emergent behavior at very large model sizes with high levels of sophistication in training, and that this behavior is both unavoidable and dangerous. More likely it's that they are just training and prompting the LLM to act that way.
I believe Suleyman is arguing that Anthropic should be very careful about how they train these models to talk about themselves for this reason.
Microsoft ignores that they themselves are a disastrous impact on humanity already.
Though for once I do actually agree with that specific leader, it’s incredibly annoying how anthropomorphic Claude is. Anthropic went way too far in that direction
1) a breakthrough in performance/learning/model
2) regulatory capture to ensure open source models can be labelled as dangerous and banned so you can set the market rules yourself
Only one of the above is risk-free, and just a question of capital/lobbying rather than a "maybe".
Apparently they're cooking up MAI (Microsoft AI), and I'd seen their small Phi models listed online. Calling Anthropic a competitor is hilarious.
"Our AI is useless not because we're a dysfunctional corporate behemoth that slowly kills every product it touches. No, no. Our AI is useless because making useful AI is evil, and we're not evil."
All of these companies need to be shut down.
Maybe Anthropic understands something about alignment Microsoft doesn't, a little humility may be called for.
More at 11
And after that when they put a mind to it and pull out all the stops, woohoo!
The default for every major thing within range can turn into a wasteland real fast.
Well actually, it might.
I remember there was a recent discussion about "how complex systems fail" and there are usually many "proto-accidents" before the catastrophe (https://news.ycombinator.com/item?id=49411370)
I bet with hindsight the Hugging Face hack will be one of them in this arena
There is a lot of hubris in predictions about LLMs. If an AI were so intelligent, then it would probably be intelligent enough to want nothing to do with us.
Still, I worry more about other humans than I do LLMs. Our fellow mankind will probably wipe us out before LLMs do. That, or the Earth will punish mankind for our cruelty, vanity, and disrespect.