Quick rant bc this triggered me: this is unfortunately wildly misleading and incorrect. In this eval, the candidates retrieved by each model are never evaluated to determine which returned the most relevant results... Synthetic queries were generated with GPT 3.5 turbo, GPT-4-0613 (from 2023) was used to ensure that the chunks GPT 3.5 turbo used to generate those queries were actually factually accurate. And then it's assumed that there are no other chunks in the corpus that are better answers to the generated query. You can't even use this approach to produce good training data for an embedding model bc way too much of the time there's a better positive chunk in the corpus. You have to use better llms, a better synthetic query generation pipeline, and apply sensible filtering on those just to get usable training data, much less an eval... These synthetic queries are also unusual in that there's clearly v high lexical overlap between the synthetic queries and the document chunks they were generated from (you can see this in how well bm25 does on the problematic eval). And then none of the strongest "single vector" embedding models are included in the benchmark. Voyage-4-nano is good for its size but not being evaluated properly here. When you fix this, every conclusion flips. The models at the bottom of this plot all start out performing the late interaction models and bm25. I've never been able to actually recommend a late interaction model to anyone bc, without exception, they have underperformed the state of the art in dense embedding models every time I've benchmarked them. I'm not talking about a plausible future state in retrieval. I'm not talking about what might someday happen if people train better late interaction models or a paper where they use the same training data, same compute and same param count. I'm talking about you building an app that needs good search where you're asking me to help you pick either a base to fine tune or the best off the shelf model for your task. There's almost a 0% chance you should consider using a late interaction model today. There are exactly 0 competitive open source late interaction models when you compare to the current state of the art "single vector" embedding models and it's not remotely close folks. Pls don't be confused by these terrible benchmarks. It's not that I don't think late interaction models could someday be good if someone training one on a large, strong dataset. It's a cool idea. We need lots of different ideas to blossom but then we need to accurately assess which techniques are currently going to give builders the best results. But today, the available open source late interaction models are not good.