Previously, RAG systems were the standard method for retrieving information from documents. However, if you are not repeatedly querying the same document, it may be more convenient and effective to just use long-context LLMs. For example, Llama 3.1 8B and Llama 3.2 1B/3B now support up to 131k input tokens, making them viable alternatives if your hardware can handle it. In either case, one big limitation of using LLMs for question answering here is their inability to provide citations. I just read an interesting paper that aims to address (or improve) information retrieval with fine-grained citations, "LongCite: Enabling LLMs to Generate Fine-Grained Citations in Long-Context QA" (arxiv.org/abs/2409.02897). In this paper, the researchers use off-the-shelf LLMs to generate a dataset consisting of long-context QA instances with precise sentence-level citations and then use that dataset to finetune an open-weight LLM to generate answers with citations. The resulting LongCite 8B and 9B models are surprisingly good compared to GPT4o, Llama 3.1, etc. How does this work? Here is the 4-step procedure for generating the dataset for instruction-finetuning: (a) Starting with long texts or documents, their method uses an existing LLM to generate a Q&A dataset (a query and its associated answer) using Self-Instruct (Wang et al. 2023; discussed in one of my previous posts). (b) Next, they use the answer to retrieve several 128-token chunks from the input text for coarse-grained citations. (c) The LLM then looks for relevant sentences within these chunks to provide more fine-grained sentence-level citations (d) The researchers filter out Q&A pairs where less than 20% of the statements in the answer don't have citations The resulting dataset is then used to train an LLM in a conventional (SFT) fashion.