RAG Is Simpler Than You Think

I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search - at which point you’ll go the last mile and do reranking etc etc all the while having to support the operational burden of vector search. Then you turn around and build a search query with 500 keywords and sure it’s painful but it just works, accommodates all use cases, scales and is overall less annoying to maintain.

Does anyone have experience using SMLs for RAG (either as query rewriter or as generator for the final answer)?

I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.

I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."

Any info or advice would be appreciated.

RAG is basically good old information retrieval with LLMs doing the querying. This can include vector search but it works without that as well. Treating vector search as magic pixie dust that makes search great without effort is not necessarily going to work that well. Also, it can add a lot of cost and complexity to the equation. And if not tuned properly, you don't necessarily get good results.

The key thing with RAG is to get the right information in the context with as few queries as possible. That requires good recall (ensuring that if it is there it can be found with a reasonable query) and precision (ensuring the best stuff is on top and minimizing false positives).

With search, and by extension RAG, the principle of shit in, shit out applies. Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG. And if you mess that up, search is not going to be working that well and no amount of AI can compensate for that or only at great cost in tokens and time. So, having an ETL pipeline to pre-process what you index, testing & benchmarking search quality, etc. are all helpful.

The good news is that you don't need that much skills with agentic coding to build something half decent for this. This code almost writes itself. And even a little bit of effort on extracting structure before indexing can make a big difference.

I have a particular antipathy for articles too lazy to spell out acronyms on first use.

So: https://en.wikipedia.org/wiki/Retrieval-augmented_generation

More LLM-generated text about LLMs.

Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.

Start with BM25 and only add embeddings when keyword search actually fails you. Saves a lot of pain.

Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats with summary and keywords. Unless you have entire libraries of books to embed it’s going to be a few hundred dollars of API calls.

Then, throw it all in BigQuery. Handles all the vector stuff natively.

Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.

I assume other vendors than Google have a similar batteries-included approach you can just plug in.

Althought I agree with the first point of the author that FTS is underrated in this new RAG-first framework, the whole article really hides all the problems with RAG-pipeline and kind of hand wave everything.

If you are building a RAG pipeline for your company and are struggling like me, I would recommend this author that has whole series on entreprise documents (start with the one from May 22nd): https://towardsdatascience.com/author/angela.shi/page/4/

Note: I am not the author, just got her article in my newsletter and found it useful.

What RAG means for AI is what a library means for human beings.

It's necessary and would be good for you if you want to learn something systematically.

But for most of the normal issues, we can not rely a lot on it.

There have been many blogs like this over the last years.

Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.

90% of "document" based RAG projects should view semantic search with embeddings as their primary method.

It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.

RAG is so 2024.

Agentic query rewrite on top of good old fashioned Lucene is the end game. This is effectively providing a lot of the same magic you get with the semantic approach. Allowing the agent to query the document store iteratively is where the capabilities become unbounded.

Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.

If someone has a Postgres db and want very simple RAG:

https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )

The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.

I would like to see how each recipe performs against its corresponding evals. Some sort of ranking would be useful.

Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.

Im curious whether the $10,000 figure includes unstated migration costs, since the raw embedding API cost under the earlier assumptions comes to $10.

Maybe I'm old but where exactly are the "dragons"?

How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?

If so, I'd like to see more design patterns around existing search problems:

- Correcting or backtracking based on feedback.

- Measuring relevance.

- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?

The article sounds like AI slop with some predictable tells like short punctual sentences, bizarre jargon, and titles like "Recipe 4: On-The-Fly Embedding (The Fresh Data Play)"

Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"

> Why this is more flexible than embeddings

Oh boy...

Chunk size matters way more than the retrieval model in my experience. Get that wrong and nothing else helps.

OT but its interesting that none of the harnesses today use embeddings but just simple grep. I would not have predicted this

i want to ask that, if a user want to search sth, but he doesnt know the exact name(keywords), just some description. at this moment, whether the text serach fail?

Wow! Terrible layout. Shouldn't fully justify on a small screen.