RAG Is Simpler Than You Think

I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search - at which point you’ll go the last mile and do reranking etc etc all the while having to support the operational burden of vector search. Then you turn around and build a search query with 500 keywords and sure it’s painful but it just works, accommodates all use cases, scales and is overall less annoying to maintain.

I have a particular antipathy for articles too lazy to spell out acronyms on first use.

So: https://en.wikipedia.org/wiki/Retrieval-augmented_generation

Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats with summary and keywords. Unless you have entire libraries of books to embed it’s going to be a few hundred dollars of API calls.

Then, throw it all in BigQuery. Handles all the vector stuff natively.

Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.

I assume other vendors than Google have a similar batteries-included approach you can just plug in.

More LLM-generated text about LLMs.

Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.

I would like to see how each recipe performs against its corresponding evals. Some sort of ranking would be useful.

Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.

There have been many blogs like this over the last years.

Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.

90% of "document" based RAG projects should view semantic search with embeddings as their primary method.

It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.

Agentic query rewrite on top of good old fashioned Lucene is the end game. This is effectively providing a lot of the same magic you get with the semantic approach. Allowing the agent to query the document store iteratively is where the capabilities become unbounded.

Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.

If someone has a Postgres db and want very simple RAG:

https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )

The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.

Maybe I'm old but where exactly are the "dragons"?

How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?

If so, I'd like to see more design patterns around existing search problems:

- Correcting or backtracking based on feedback.

- Measuring relevance.

- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?

Chunk size matters way more than the retrieval model in my experience. Get that wrong and nothing else helps.

> Why this is more flexible than embeddings

Oh boy...

The article sounds like AI slop with some predictable tells like short punctual sentences, bizarre jargon, and titles like "Recipe 4: On-The-Fly Embedding (The Fresh Data Play)"

Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"

i want to ask that, if a user want to search sth, but he doesnt know the exact name(keywords), just some description. at this moment, whether the text serach fail?

OT but its interesting that none of the harnesses today use embeddings but just simple grep. I would not have predicted this