Over the last two days after my claim "long context will replace RAG", I have received quite a few criticisms (thanks and really appreciated!) and many of them stand a reasonable point. Here I have gathered the major counterargument, and try to address then one-by-one (feels like a paper rebuttal): - RAG is cheap, long context is expensive. True, but remember, compared to LLM, BERT-small is also cheap, and n-gram is even cheaper, but they are not used today, because we want the model to be smart first, then makes smart models cheaper -- history of AI tells it is much easier to make smart models cheaper than making cheap model smart -- when it is cheap, it's never smart. - Long context can mix retrieval and reasoning during the whole decoding processing. RAG only does the retrieval at the very beginning. Typically, given a question, RAG retrieves the paragraphs that is related to the question, then generate. Long-context does the retrieval for every layer and every token. In many cases the model needs to do on-the-fly per-token interleaved retrieval and reasoning, and only knows what to retrieve after getting the results of the first reasoning step. Only long-context can do such cases. - RAG supports trillion level tokens, long-context is 1M. True, but there is a natural distribution of the input document, and I tend to believe most of the cases that requires retrieval is under million level. For example, imagine a layer working on a case whose input is related legal documents, or a student learning machine learning whose input are three ML books -- does not feel as long as 1B right? - RAG can be cached, long-context needs to re-enter the whole document. This is a common misunderstanding of long-context: there is something called KV cache, and you can also design sophisticated caching and memory hierarchy ML system working with kv cache. This is to say, you only read the input once, then all subsequent queries will reuse the kv cache. One may argue that kv cache is large -- ture, but don't worry, we LLM researchers will give you crazy kv cache compression algorithms just in time. - You also want to call a search engine, which is also retrieval. True, and in the short term, it will continue to be true. Yet there are crazy researchers whose imagination can be wild -- for example, why not letting the language model directly attend to the entire google search index, i.e., let the model absorb the whole google. I mean, since you guys believe in AGI, why not? - Today's Gemini 1.5 1M context is slow. True, and definitely it needs to be faster. I'm optimistic on this -- it will definitely be much faster, and eventually as fast as RAG Let's see how things go, shall we?