Research Scientist at Google DeepMind working on multilinguality
San Francisco
Joined September 2012
- Tweets94
- Following92
- Followers600
- Likes140
Pinned Tweet
Excited to receive an Outstanding Paper award for this work at @emnlpmeeting! Thanks to my co-authors George Foster and @markuseful! Updated version available here: aclanthology.org/2023.emnlp-…
LLM-based metrics like GEMBA predict many ties, but the way that ties should be handled in Kendall’s tau for meta-evaluating metrics has been a longstanding issue. We propose an update to the meta-evaluation methodology to handle ties.
arxiv.org/pdf/2305.14324.pdf
Dan Deutsch retweeted
We were asking one question a year ago: "Can we trust LLMs to build and grade their own benchmarks"? We tested frontier models on automated LLM benchmarking. The result? Every single model ranks itself #1 on benchmarks it generates (arxiv.org/abs/2509.26600). Our work has been accepted to EMNLP 2026 Main!
Dan Deutsch retweeted
Happy to share CollabEval, recent(-ish) work from @GoogleDeepMind and @GoogleResearch that can be used to improve the statistical efficiency of model evaluation by exploiting correlations between historical evals of different models on the same tasks.
arxiv.org/pdf/2607.05046
Dan Deutsch retweeted
Machine translation is tough to evaluate, partly because most of what you throw at is too easy. That doesn't at all mean that translation is solved; we're just not doing a good job finding interesting inputs.
Dan Deutsch retweeted
Come do a PhD with me at Columbia!
My lab tackles basic problems in alignment, interpretability, safety, and capabilities of language systems. If you love adventuring in model internals and behaviors---to understand and improve---let's do it together!
pic: a run in central park
Dan Deutsch retweeted
🗺️ Are we making our #LLMs multilingual, or anglocentric?
Much work brings languages closer to English, but that comes at the cost of crucial #cultural nuance.
@h__j___han tackles this trade-off with surgical steering, adapting LLMs to cultural contexts at inference time.
Dan Deutsch retweeted
Our Google Translate team is bringing a strong presence to #ACL2025 in Vienna this week! 🇦🇹 My group is excited to present several of our latest papers. 👇 Don't miss them!
Dan Deutsch retweeted
Two new datasets from Google Translate targeting high and low resource languages!
WMT24++: 46 new en->xx languages to WMT24, bringing the total to 55
SMOL: 6M tokens for 115 very low-resource languages
WMT24++: huggingface.co/datasets/goog…
SMOL: huggingface.co/datasets/goog…
Dan Deutsch retweeted
😼SMOL DATA ALERT! 😼Anouncing SMOL, a professionally-translated dataset for 115 very low-resource languages! Paper: arxiv.org/pdf/2502.12301
Huggingface: huggingface.co/datasets/goog…
🚨New machine translation dataset alert! 🚨We expanded the language coverage of WMT24 from 9 to 55 en->xx language pairs by collecting new reference translations for 46 languages in a dataset called WMT24++
Paper: arxiv.org/abs/2502.12404v1
Data: huggingface.co/datasets/goog…
This project was a highly collaborative effort with many people contributing translations, evaluations, analyses, etc., so I want to thank all of my co-authors! @ebriakou @iseeaswell @marafinkels Rebecca Galor @JurikJuraska @gezakovacs Alison Lui @RicardoRei7 @jasonriesa
@shrutirij @prk_riley @esalesk @FirasTr88060642 Stephanie Winkler @BZhangGo @markuseful
#nlproc #nlp #ai
Dan Deutsch retweeted
Thrilled to share our latest findings on data contamination, from my internship at @Google! We trained almost 90 Models on 1B and 8B scales with various contamination types using machine translation as our task and analyze the impact of contamination.
arxiv.org/abs/2501.18771
Dan Deutsch retweeted
🚀 We have just released bfloat16 variants of all 3 MetricX-24 models, offering nearly identical performance to their float32 counterparts, but with a 50% smaller memory footprint. ✨ We hope this makes the XL and XXL models more accessible!
🔗 GitHub: github.com/google-research/m…
🌐 Meet MetricX-24, our SOTA machine translation evaluation metric and a successor to the successful MetricX-23. 🚀 Now open-source in PyTorch/Transformers! 🎉 Ready to take this top performer in the WMT24 Metrics Shared Task for a spin?
🔗 Code: github.com/google-research/m…
Dan Deutsch retweeted
🌐 Meet MetricX-24, our SOTA machine translation evaluation metric and a successor to the successful MetricX-23. 🚀 Now open-source in PyTorch/Transformers! 🎉 Ready to take this top performer in the WMT24 Metrics Shared Task for a spin?
🔗 Code: github.com/google-research/m…
Super simple and effective way of significantly increasing the performance of your evaluation metric!
LLMs are typically evaluated w/ automatic metrics on standard test sets, but metrics + test sets are developed independently. This raises a crucial question: Can we design automatic metrics specifically to excel on the test sets we prioritize? Answer: Yes!
arxiv.org/abs/2411.15387
New application link! google.com/about/careers/app…
I am at EMNLP/WMT this week. Please come find me if you want to learn more about this role!
Interested in doing research on Google Translate and Gemini? Good news! I’m hiring for full-time roles on the Google Translate Research Team! Apply here: google.com/about/careers/app…