Koorosh Aslansefat

Don't do RAG Proposes cache-augmented generation (CAG) to eliminate retrieval latency and minimize retrieval errors. What is CAG? CAG aims to leverage the capabilities of long-context LLMs by preloading the LLM with all relevant docs in advance and precomputing the key-value (KV) cache. The preloaded context helps the model to provide contextually accurate answers without the need for additional retrieval during runtime. When to apply CAG? It's a useful alternative to RAG for cases where the documents/knowledge for retrieval are of limited, manageable size. My thoughts: As LLMs advance in capabilities, I suspect that what we know as RAG today could change significantly either architecturally or how it's optimized. CAG is one in a growing list of developments and new ideas that have emerged recently to address limitations like poor retrieval relevancy and latency. There could also be hybrid methods that combine preloading with selective retrieval. Don't sleep on long-context LLMs. They are here to stay.