Smaller subset of 200 queries (prior post was >1k) but jina-reranker-v3.5 doesn't do well here and more notably, voyage rerank-3 is a good baseline bc it's ~Luna quality (Luna (low) with blitzrank is wayyyy higher latency to be clear, this is a method for prepping a dataset not a reranker you'd use on prod). If anything Luna should have an advantage here bc it's likely distilled from/trained on much of the same data as Sol's preferences which is my ground truth here so in that sense your assumption should be that this eval makes Luna look better that it should. So here again, Jev is outperforming Luna at lower cost and much much lower latency.

Quick rant bc this triggered me: this is unfortunately wildly misleading and incorrect. In this eval, the candidates retrieved by each model are never evaluated to determine which returned the most relevant results... Synthetic queries were generated with GPT 3.5 turbo, GPT-4-0613 (from 2023) was used to ensure that the chunks GPT 3.5 turbo used to generate those queries were actually factually accurate. And then it's assumed that there are no other chunks in the corpus that are better answers to the generated query. You can't even use this approach to produce good training data for an embedding model bc way too much of the time there's a better positive chunk in the corpus. You have to use better llms, a better synthetic query generation pipeline, and apply sensible filtering on those just to get usable training data, much less an eval... These synthetic queries are also unusual in that there's clearly v high lexical overlap between the synthetic queries and the document chunks they were generated from (you can see this in how well bm25 does on the problematic eval). And then none of the strongest "single vector" embedding models are included in the benchmark. Voyage-4-nano is good for its size but not being evaluated properly here. When you fix this, every conclusion flips. The models at the bottom of this plot all start out performing the late interaction models and bm25. I've never been able to actually recommend a late interaction model to anyone bc, without exception, they have underperformed the state of the art in dense embedding models every time I've benchmarked them. I'm not talking about a plausible future state in retrieval. I'm not talking about what might someday happen if people train better late interaction models or a paper where they use the same training data, same compute and same param count. I'm talking about you building an app that needs good search where you're asking me to help you pick either a base to fine tune or the best off the shelf model for your task. There's almost a 0% chance you should consider using a late interaction model today. There are exactly 0 competitive open source late interaction models when you compare to the current state of the art "single vector" embedding models and it's not remotely close folks. Pls don't be confused by these terrible benchmarks. It's not that I don't think late interaction models could someday be good if someone training one on a large, strong dataset. It's a cool idea. We need lots of different ideas to blossom but then we need to accurately assess which techniques are currently going to give builders the best results. But today, the available open source late interaction models are not good.