Agent Memory as a File Format

I find once there is one poisoned line of text it negatively affects everything else downstream. Instead, I use a temp/ folder with documents and use different files for different agents and models. Then I have to constantly prune and delete the files. Any information that can be extrapolated is just noise which negatively affects the agent. If you have a definition of a database structure and it has been implemented, that information should not be contained in any text document -- it is noise, will drift, and be impossible to debug why the agent keeps producing undesired behavior.

I have a ~/Projects folder. For example, I use Playwright with Chrome DevTools Protocol in order to do performance testing and leak detection. There is a script that handles this. My prompt is "Search ~/Projects for perf testing with CDP and Playwright and implement here". Point being, if I need anything I point to a resource or ask to search a resource and it will find it quick and, most importantly, tends to improve it every iteration.

If I was in an institution, I would have a repository and would rather just point the resource and say use that than have memory of it locally.

- memory systems are a specific type of knowledge base where you generate all the documents. You might as well generate them to be less than your embedding token limit to obviate the need for chunking.

- embedding models are getting better and are no longer just semantic averaging.

- small models are getting dirt cheap, making parallel reads cost manageable

What they describe is sort of the simplest architecture that takes advantage of these observations. I believe them when they say it works well.

I do suspect though that things like keyword lookup will completely fail if every memory is just a vector. Hence why something like Typesense hybrid search can still be useful.

I do think that having a set of token that are highly personalized to your project and to way you work is beneficial. I also think that the idea that this set of token will be constructed in the background without any work from the user is really appealing. So it's understandable that the 'memory' analogy became so popular.

But in my experience having a really good AGENTS.md file almost always produce better results than enabling memory.

Maybe we should start to think about how we 'train'/'onboard' agents into our projects, in a similar way that we do for new co-workers. Imagine if we could send the agent to our repo and ask it to learn our patterns and in the end we could quiz the agent to gauge how much it actually understood the project. Once he 'understands' the project we can start to use it to help with development.

In a very small scale (example, individual new features) I will sometimes ask the agent to explain me how things work (even though I already know how it works) so I can 'prime' the agent context with good data before starting any real work. But I'm not sure if this approach could be reliably scaled to work with any repo for any kind of work.

I wonder how much my system needs something like this. Between the invisible system memory of my random chats with Gippity, my Matt Pocock skills saving terminology and plans, and whatever else Cursor and Codex do, I don't think I feel a need for more agent memory. I do like how it's exposed and searchable, and not invisible. But I honestly just send my questions/tasks away to my magic agent and eventually it gets it right anyway; do I need more discrete memory my team has to maintain? (That's an earnest question, not disregard for this)

I just had a thought about similar thing - how to track human decisions on the codebase? Consider you are writing code together with AI, how you understand which code change happened because human asked for it

Incredible to watch things come full circle. Next thing you know, someone is going to figure out a binary encoding.

Both can be true: - It's useful to anthropomorphize agents when predicting behavior and - we have to use specific language to specify what we mean.

What does the author mean by "confuse the models" ? Are they talking about not picking right information? Picking the wrong information? Losing their previous context / task?

Part of setting up a proper eval is also deciding what we actually mean ourself. What are we actually optimizing for? It's not, e.g. % confusion, %rubbish, etc.

The article does point to it: retrieval latency, accuracy, etc.

> This is a common fear with memory systems but doesn't really apply to memoryfields. Irrelevant material is simply never surfaced by the semantic search.

This is so wrong. The Achilles' heel of this approach is the RAG. What makes it worse is having lots of memories that are outdated, wrong, hallucinated, or irrelevant.

Nothing beats curated data. Memory should be regularly reviewed, compacted, and cleaned up if it's no longer valid.