What is Nueralese and Why is it Bad

If the model can understand neuralese why can it not convert it into English for monitoring or review purposes?

Or we believe the model will encode secret messages like "don't reveal this information" into the neuralese. But as the author mentions, they could have been doing that all along

> Models can omit key information in their visible thoughts, as this Anthropic 2025 paper shows. We are also worried about steganography

The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT.

That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.

Title is misspelled -- "ue" for "eu". In a neologism, some nuisance.

What about the trend of summarizing or eliding reasoning from the visible model response, ostensibly to make distillation by competitors harder?

Can we take a step back here? Both OpenAI’s and Anthropic’s models think in encryptedese, and it seems thoroughly absurd to think that the entire world should trust those two companies to adequately monitor the plaintext or, for that matter, to have their monitoring systems aligned with what is actually good for the world.

If you want to monitor your model, you need to start with an inference provider that gives you the entire output and possibly even run it yourself to get access to the internal states. And if you think the KV cache and (when present) the recurrent state don’t encode a lot of “thought”, you are fooling yourself.

FWIW, I think most model architectures at least have the property that latent state can’t propagate from higher layers to lower layers by any route other than the output tokens. But even a two-iteration structure could be designed so that the last layer produces a vector that enters the first layer, once per token, and I bet it it would be very easy to train such a model to “think” in silence in the sense that the output tokens while thinking would all be one particular null token.

My understanding is, that we do not know if the chain-of-thought actually matters in the way we assume for the result.

I think there were experiments where seemingly relevant parts of the CoT were ablated and it did not change the result.

For all we know it might be somewhat human parseable neuralese.

What exactly prevents anyone worried about this to build an LLM that can decode the neuralese to english and use it to monitor what the model is doing?

Or is the neuralese some sort of irreversibly encrypted data set that only an LLM can "understand" and that can never be translated back to English?

I don't think their architectural view of CoT works is correct is it?

Can we reframe the question. Instead of "why is neuralese bad?", I think we should first answer "is CoT monitoring effective?"

surely Neuralese interpreters can be made that turn the chain of numbers into an English description?