A global workspace in language models

> I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.

Too bad the frontier models are closed weights.

Maybe the research community and whole rest of the world will build on open and all the advances will happen in open ecosystems instead.

If dirt-simple type operations like copy-paste yield useful improvements with even a small probability that would seem to open things up for adaptive reconfiguration and whole other classes of optimizations like genetic algorithms.

Found it: https://news.ycombinator.com/item?id=47500709

Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/

tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers

Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts

Source for those interested

https://dnhkng.github.io/posts/rys/

Yeah! I still think about that sometimes. Mind-blowing that worked at all, let alone improved performance.

Such an inspection capability might also be used to target ads to LLMs, which would then be more likely to mention or recommend those products and services.

> I'm now considering the architecture of the service

What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.

Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it.

There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors

Note that Neel Nanda replicated the results on a Qwen model.

Presumably the rationale for the decision to abridge the thinking traces will ensure that they don’t; if this is real (and there’s no good reason to trust that it is yet) then it is the secret sauce.

Anthropic aren't even willing to expose the CoT of their models. You will have to rely on them to build those sorts of things into dedicated signals.

Anthropic won't do it, but they published the j-lens to introspect the model- from what I understand it's roughly simply feeding a chosen layer straight into the final layers of the LLM for decoding into language:

https://github.com/anthropics/jacobian-lens

Looks like it should be easy to use on open weights models.

But J-Space is much catchier. This is not a scientific paper, it's a promotional essay.

There was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers and repeat them, and you get a stronger model. Those middle layers would be what Anthropic calls the J-Space, and their J-Lens maps activity in those layers back to tokens that trigger similar activity (with a technique they only drop hints at)

The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them

It's not in the weights. Sounds to me like jspace is the "positive cone" over relevant (large norm) j-lenses, and j-lenses are gradients wrt tokens on the residual stream when you average over some training data.

Anthropic theorize that middle layers in an LLM is a "J-Space" used to "think" about the future answer or about abstract concepts.

Their method is used to identify which tokens can appears in which layers of the model.

It's been shown that LLMs use their outer layers to decode from and encode to language, while their middle layers deal in language-independent abstract concepts. This means that the same question or statement in different languages activates the outer layers differently but produces the same patterns in the middle layers. Check this article with cool visualizations (btw, this is one of the articles mentioned also by a sibling answer):

https://dnhkng.github.io/posts/sapir-whorf/

The middle layers also perform reasoning on the abstract concepts, to the point that you can replicate some blocks of inner layers (thus giving the LLM more internal "reasoning space") and by this increase the model's reasoning abilities. The video in this article shows that when performing a sequence of arithmetic operations (without CoT, i.e. the result is spit out directly), internally the intermediate calculations are spelled out, and this can only happen in the depth direction of the LLM (since no new token is added to the sequence). So this "jspace" can only be situated in the middle layers, probably in circuits that repeat nearly identical across several layers.

Tokens that are activated but not present in it's output maybe?

I too have confusion.

Your comment seems to have little to do with the article?

Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source:

"In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."

Lying involves intent whereas hallucinations and mistakes are an artifact of how they work. Humans hallucinate, make mistakes, and can actually lie. We've been dealing with this forever. What's the value in requiring the llms to have 100% accuracy? (I don't think it is possible)