Cache-to-Cache: Direct Semantic Communication Between Large Language Models

A few months ago I asked why semantic representation rather than text wasn't used, since natural language seems quite a lossy representation for semantic concepts:

https://news.ycombinator.com/item?id=47195212

I wouldn't have thought to use it for LLM-to-LLM communication, though

I feel like multimodal models that can read images should work differently than they do. My understanding is that multimodal models basically first generate an image embedding and then the model is trained to interpret that embedding, but in the same way that text is lossy, it seems like the embedding would be as well. Why don't multimodal models learn to interpret images themselves without an embedding? Or e.g., by passing some "prompt" to the embedding model?

So the models will not only be using more and more Neuralese in their CoT (like GPT-6), but different agents will also be able to communicate with each other in Neuralese. It's not looking good for monitorability.