Research Scientist @MSFTResearch Prev: @Huawei @EPFL @ICepfl @HD_NLP
Bengaluru, India
Joined April 2012
- Tweets578
- Following2.2K
- Followers907
- Likes5.2K
Pinned Tweet
Can AI agents truly synthesize information across multiple sources?
🏆 Best system: 8.97 F1
❌ Most systems: 0% exact match
🚀 Introducing DEEPSYNTH, our new benchmark accepted at #ICLR2026!🇧🇷
📄 Paper: arxiv.org/abs/2602.21143
🤗 Data: huggingface.co/datasets/Deep…
20 🧐 submissions seems really high.
Additionally, per-author quotas are introduced, where authors are capped at 20 total submissions and 5 first-author (including shared first-author) submissions per cycle. More details wil be posted on the ACL web and here shortly. #NLProc
Debjit Paul retweeted
Over the past few days, I've taken the time to summarize my thoughts on the recent incidents involving agents’ misaligned behavior. We don't know with certainty what comes next, but we know where these issues originate, and this can help us plan the path forward.
Please feel free to ask your questions in the replies, and I’ll try to answer some of them in the coming weeks.
yoshuabengio.org/en/publicat…
🚀🚨We're hiring at Microsoft Research India!
We're looking for Researchers, Senior Researchers, and Senior Applied Researchers to join our team in Bengaluru.
📧 Send your CV to [email protected]
#Hiring #MicrosoftResearch #InformationRetrieval
Debjit Paul retweeted
We just got an award at #ACL2026 for our Parity-Aware BPE paper!
When AI systems process text, they first break it into chunks called tokens. The standard method for deciding how to chunk text (called BPE) learns from frequency: chunks that appear often in the training data get treated efficiently.
Since training data is dominated by languages like English, those languages get compact, sensible tokenizations, while lower-resource languages end up chopped into many more tokens, often in ways that ignore how the language actually works.
This matters practically: more tokens means more computation and higher costs, so speakers of underrepresented languages literally pay more to use the same AI systems.
We propose a fix called Parity-aware BPE. Instead of always picking the chunk that helps compression overall (which mostly benefits dominant languages), our version repeatedly asks: which language is currently getting the worst deal?
Then it picks the chunk that most helps *that* language. It sacrifices a tiny bit of overall efficiency in exchange for fairness across languages.
The results: inequality between languages (measured with the Gini coefficient, the same statistic used for income inequality) drops by up to 89%, while overall compression barely suffers and the AI models trained with it perform just as well.
Great to have been part of an amazing author team: @negarforoutan, @clara__meister, @DebjitPaul2, @sina_ahm, @ABosselut,
@RicoSennrich
Debjit Paul retweeted
🔬 Hiring 2 undergrad research interns (6 months) at Microsoft Research India.
The transformer has been the default encoder for dense retrieval. But under the low-latency constraints of real production systems, it becomes a serious bottleneck on retrieval performance, deep encoders are accurate but too slow, shallow ones are fast but lossy.
So we're asking fundamental questions:
→ What assumptions are we baking in when we reach for a transformer to solve a task?
→ What alternative scalable encoder architectures can exploit the natural biases of retrieval better than the transformer does?
What interns will actually work on over 6 months:
→ Critically analyzing where transformer-based dense encoders fall short under production retrieval pressure
→ Exploring alternative architectures that preserve deep-encoder accuracy at a fraction of the inference cost
→ Data + compute efficient training algorithms for large dense encoders
Strong Python + PyTorch. Bonus if you've trained an encoder or built a retrieval pipeline end-to-end.
For undergrads who treat "why is the architecture shaped this way?" as a real question.
Apply: forms.office.com/r/G1TyJZCFG…
DMs open.
#InformationRetrieval #MLSystems #NLProc
@MSFTResearch
Debjit Paul retweeted
The better LLMs get at reasoning, the longer their traces get—thousands of tokens, dozens of tool calls. But in law, medicine, and agentic AI, "usually correct" isn't good enough: answers must be verifiably correct.
We built interwhen at @MSFTResearch to make that tractable. And it's now open source.
Across benchmarks, plugging interwhen into an LLM yields:
✅ 100% soundness (with full verifiers)
📈 up to 15% accuracy gain
⚡ ~ 1.5× compute cost
🧵
Debjit Paul retweeted
📢 Open-sourcing the Sarvam 30B and 105B models! Trained from scratch with all data, model research and inference optimisation done in-house, these models punch above their weight in most global benchmarks plus excel in Indian languages.
Get the weights at Hugging Face and AIKosh. Thanks to the good folks at SGLang for day 0 support, vLLM support coming soon. Links, benchmark scores, examples, and more in our blog - sarvam.ai/blogs/sarvam-30b-1…
Debjit Paul retweeted
Your AI agent scores 95% on benchmarks but can it actually synthesize information from multiple sources to answer a real question?
DEEPSYNTH, a new benchmark from @DebjitPaul2, Daniel Murphy, Milan Gritta and team at Huawei Noah's Ark Lab (@HuaweiNoahsArk), @imperialcollege, @ucaboratory, and @Cambridge_Uni, tests exactly this. 120 tasks across 7 domains, 67 countries, requiring agents to gather data, cross reference sources, and produce structured insights.
The results are sobering: 11 state of the art LLMs and deep research agents top out at 8.97 F1. The best LLM judge score reaches only 17.5. Models hallucinate freely and collapse when reasoning over large information spaces.
The gap between answering trivia and doing actual research remains massive. Accepted at ICLR 2026.
Debjit Paul retweeted
Can AI agents truly synthesize information across multiple sources?
🏆 Best system: 8.97 F1
❌ Most systems: 0% exact match
🚀 Introducing DEEPSYNTH, our new benchmark accepted at #ICLR2026!🇧🇷
📄 Paper: arxiv.org/abs/2602.21143
🤗 Data: huggingface.co/datasets/Deep…
🔍 Key findings: Current agents “struggle with hallucinations and reasoning over large information spaces”. In other words, they tend to make unsupported claims or miss links across sources when answering complex questions.
Work done in collaboration with Daniel Murphy, @milangritta, Ronald A. Cardenas, @victor_p91, Jun Wang, @glampouras_NLP, and an amazing annotation team.
Project Page: agentdeepsynthesis.github.io…