I just finished my small experiments comparing different encoder models on retrieval tasks. The goal was to check whether MLM is better than RTD for these tasks. I compared Electra's small models, both generator and discriminator, that have the same size. Additionally, it was tested DeBERTa v1, which was pre-trained with MLM and DeBERTa v3, which was pre-trained with RTD on a two times larger dataset. As a baseline ModernBERT was evaluated as well. Models were fine-tuned on 500k examples from the MS-MARCO dataset (huggingface.co/datasets/sent…). For benchmarking, the NanoBEIR evaluator was used. You can see the average ndcg@10 plotted below. It's clearer that MLM-trained models produce better discriminative features, however, more detailed experiments are needed for more accurate conclusions. @antoine_chaffin @tomaarsen