@gisship

PhD student @CentraleSupelec. Previously @psl_univ, @MIT and @HECParis.

Joined August 2013
  • Tweets17
  • Following65
  • Followers68
7Media
🚨 New paper drop: Should We Still Pretrain Encoders with Masked Language Modeling? We revisit a foundational question in NLP: Is masked language modeling (MLM) still the best way to pretrain encoder models for text representations? 📄 arxiv.org/abs/2507.00994 (1/8)

Should We Still Pretrain Encoders with Masked Language Modeling?

Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence...

arxiv.org
1
4
1
26
3,393
more replies
There is plenty more in the paper, go check it out! arxiv.org/abs/2507.00994 You can also visit our project page on Hugging Face: huggingface.co/MLMvsCLM (7/8)

Should We Still Pretrain Encoders with Masked Language Modeling?

Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence...

arxiv.org
1
4
252
Huge thanks to the dream team! @N1colAIs , @ManuelFaysse , @DuarteMRAlves, Emmanuel Malherbe, @andre_t_martins , Céline Hudelot, and @PierreColombo6 🙏 (8/8)
5
212
📢 Excited to share my latest work! Introducing "Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism". Check it out! arxiv.org/abs/2402.12997 Key takeaways down below 👇

Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism

Neural Information Retrieval (NIR) has significantly improved upon heuristic-based Information Retrieval (IR) systems. Yet, failures remain frequent, the models used often being unable to retrieve...

arxiv.org
1
1
2
5
891
more replies
Feel free to comment or message if you have any questions! Special thanks to @ManuelFaysse, Emmanuel Malherbe, @CelineHudelot and @PierreColombo6 for their precious advice and contributions.
2
2
173
Check out our code on GitHub: github.com/artefactory/abste… #Research #MachineLearning #NLP #LLM #InformationRetrieval

GitHub - artefactory/abstention-reranker

Contribute to artefactory/abstention-reranker development by creating an account on GitHub.

github.com
1
134
Load more