SOTA models and SOTA blogpost 🔥
We surfed the dense and late-interaction waves… now it's time to SlideOn sparse retrieval 🐕
🚀 Meet SlideOn and mSlideOn, fully open SOTA sparse retrievers at 174M and 339M parameters, trained by @LightOnIO R&D
Models:
huggingface.co/collections/l…
Blog: huggingface.co/blog/lightona…
This is a great day for sparse retrieval enjoyers. A very strong release with an Apache license, efficiency very similar to SPLADE-v3, and what looks like a way to avoid custom LexMAE/RetroMAE/CoCondenser pretraining - just multi-vec pretrain!
Introducing SPARSEUP: the first model release from @Linkup_platform , and the missing sparse companion of DenseOn and LateOn.
Same backbone, same data, <150M, 56+ on BEIR-13. The model is open-source, use it!
Blog: linkup.so/blog/introducing-s…
Model: huggingface.co/Linkup-Platfo… (Apache 2.0)
The most recent OpenSearch model based on Alibaba's NewModel already had a RoPE-enabled context length, but for workflows still on a BERT-based sparse encoder, ModernBERT's extended max sequence length might come in handy too
Thanks for great model and blogpost @thibault_formal 🫡
French open-model supremacy is getting out of control (but nothing new 🇫🇷 )
jonah retweeted
PLAID-style late-interaction search has always had a slow half: exact rescoring. I made it nearly free in next-plaid with one int8 operation. Works on existing indexes, CPU only. Top bar is 1.6.5, bottom is 1.7.0. The "decompress" segment is gone. Thread.
jonah retweeted
Hierarchical pooling is very strong to reduce the footprint of late interaction without degrading results
We recently showed that training for MUVERA/SMVE improved performance retention
What if we trained for hierarchical pooling?
We get even stronger (5×) lossless compression!
jonah retweeted
Rescaling MLM-Head for Neural Sparse Retrieval
@yjoonjang et al. find that pretrained encoders with large MLM-head scales face degradation in sparse retrieval, and introduce a zero-cost rescaling correction to stabilize training.
📝 arxiv.org/abs/2606.18811
Bringing some of the mystery out of stable LSR training
If you’ve struggled training SPLADE models across backbones (noob lol who would think it’s difficult DEFINITELY not me) give this one a read :)
The scale of MLM-Head may be the problem of your SPLADE models. Rescaling the head could resolve this problem. Check our new paper about this!
arxiv.org/abs/2606.18811
jonah retweeted
Whether you are GPU poor or GPU rich, today's release of PyLate has something for you!
GPU maxxers: MaxSim kernels greatly speed up training while lowering the memory requirements
CPU enjoyers: TACHIOM enables lightning fast multi-vector indexing and search directly on CPU
- sparse maxsim
- contrastively-trained sae
- strong results on BERT and 8b nemo
this is a great one
No More K-means:Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval
@Veritas2026 et al. replace vector clustering with efficient sparse autoencoders & natural inverted indexing to accelerate multi-vector retrieval.
📝arxiv.org/abs/2605.30120
👨🏽💻github.com/Y-Research-SBU/SS…
jonah retweeted
Introducing mxbai-rerank-v3-listwise: reranking that goes beyond binary relevance.
It reads the whole candidate set, resolves conflicts, and ranks by directives like recency, source priority, and multi-step rules.
+11% NDCG@10 on average across multiple domains, modalities, and languages in runs with Wholembed v3.
Available today in preview in Mixedbread.
Yes, the same team behind the SoTA Sparse index SEISMIC is also behind what appears to be SoTA multi-vector index. what is in the water in Pisa
Efficient Multivector Retrieval with Token-Aware Clustering and Hierarchical Indexing
Presents a multivector retrieval system that uses token-aware clustering to allocate centroids based on token frequency & semantic variance.
📝arxiv.org/abs/2604.28142
👨🏽💻github.com/TusKANNy/tachiom
Researchers in Asia have something incredible to wake up to tomorrow, glad I stayed up :D
Amazing release. PhD students around the world should rejoice the open dataset, it is really really impressive. Great work goats 🫡
The new generation of open state-of-the-art single and multi-vector retrieval models is here
It's time, DenseOn with the LateOn 🎶
@LightOnIO releases models that leap past existing ones, and everything you need to do the same!
They even released the base bidirectional models 😍
Great release, thanks for all the checkpoints ♥️
🚀 New model family release with an OMNIMODAL version !
After Eurobert, I'm excited to introduce BidirLM, a family of 5 frontier bidirectional encoders including an OMNIMODAL encoder at just 2.5B parameters.
🧵👇
huggingface.co/BidirLM
Whole grains are good for health and for SoTA retrieval 🍞🍞
jonah retweeted
I'm personally very bearish on MUVERA due to its many, many, failure cases, but I have a lot of respect for the @weaviate_io folks, so I gave this another deep read to see if there were things that could change my mind.
However this has me a bit puzzled, if I'm reading the graph below right, it means that MUVERA itself produces a ~50+% incompressible performance degradation at commonly used indexing parameters, and still a ~20% degradation at near-bruteforce search tier parameters (ef=1024), meaning that the degradation would be purely due to MUVERA itself.
For most retrieval uses, this would make the method completely unusable, as this degradation for many workflows is almost similar to the one we'd experience from replacing semantic search with pure bm25/keyword search.
I feel like I'm missing something here so I'm very happy to be corrected if I'm misinterpreting the results!
Multi-vector embeddings (ColBERT, ColPali) are budget killers.
But MUVERA can cut your memory footprint by 70%.
Multi-vector models offer incredible retrieval but suffer from massive memory overhead and slow indexing. MUVERA (Multi-Vector Retrieval via Fixed Dimensional Encodings) compresses these into single, fixed-dimensional vectors.
How it works:
MUVERA condenses a sequence of vectors (e.g., 100x96d) into one vector via:
1️⃣ Space Partitioning: Groups vectors into buckets using SimHash or k-means clustering.
2️⃣ Dimensionality Reduction: Applies random linear projection to compress each sub-vector while preserving dot products.
3️⃣ Repetitions: Repeats the process multiple times and concatenates results to improve accuracy.
4️⃣ Final Projection: Optional final compression (not used in Weaviate's implementation).
The impact (LoTTE benchmark):
- Memory: 12GB → <1GB.
- Indexing: 20+ mins → 3-6 mins.
- HNSW Graph: 99% smaller.
There’s a trade-off:
You trade a slight dip in raw recall for massive efficiency gains. However, by tuning the HNSW `ef` parameter (e.g., `ef=512`), you can recover 80-90%+ recall while keeping costs low.
When should you use MUVERA?
→ Large-scale production RAG
→ Systems where memory/infrastructure costs are the direct bottleneck
→ Use cases requiring fast indexing
MUVERA in @weaviate_io 1.31+ takes just a couple of lines of code. You can tune three parameters (k_sim, d_proj, r_reps) to balance memory usage and retrieval accuracy for your specific use case.
Read the full technical deep-dive here: weaviate.io/blog/muvera?utm_…
