Research Scientist at Google DeepMind working on multilinguality
San Francisco
Joined September 2012
- Tweets94
- Following92
- Followers599
- Likes140
Dan Deutsch retweeted
We were asking one question a year ago: "Can we trust LLMs to build and grade their own benchmarks"? We tested frontier models on automated LLM benchmarking. The result? Every single model ranks itself #1 on benchmarks it generates (arxiv.org/abs/2509.26600). Our work has been accepted to EMNLP 2026 Main!
Dan Deutsch retweeted
Happy to share CollabEval, recent(-ish) work from @GoogleDeepMind and @GoogleResearch that can be used to improve the statistical efficiency of model evaluation by exploiting correlations between historical evals of different models on the same tasks.
arxiv.org/pdf/2607.05046
Dan Deutsch retweeted
Machine translation is tough to evaluate, partly because most of what you throw at is too easy. That doesn't at all mean that translation is solved; we're just not doing a good job finding interesting inputs.
Dan Deutsch retweeted
Come do a PhD with me at Columbia!
My lab tackles basic problems in alignment, interpretability, safety, and capabilities of language systems. If you love adventuring in model internals and behaviors---to understand and improve---let's do it together!
pic: a run in central park
Dan Deutsch retweeted
🗺️ Are we making our #LLMs multilingual, or anglocentric?
Much work brings languages closer to English, but that comes at the cost of crucial #cultural nuance.
@h__j___han tackles this trade-off with surgical steering, adapting LLMs to cultural contexts at inference time.
Dan Deutsch retweeted
Our Google Translate team is bringing a strong presence to #ACL2025 in Vienna this week! 🇦🇹 My group is excited to present several of our latest papers. 👇 Don't miss them!
Dan Deutsch retweeted
Two new datasets from Google Translate targeting high and low resource languages!
WMT24++: 46 new en->xx languages to WMT24, bringing the total to 55
SMOL: 6M tokens for 115 very low-resource languages
WMT24++: huggingface.co/datasets/goog…
SMOL: huggingface.co/datasets/goog…
Dan Deutsch retweeted
😼SMOL DATA ALERT! 😼Anouncing SMOL, a professionally-translated dataset for 115 very low-resource languages! Paper: arxiv.org/pdf/2502.12301
Huggingface: huggingface.co/datasets/goog…
@shrutirij @prk_riley @esalesk @FirasTr88060642 Stephanie Winkler @BZhangGo @markuseful
#nlproc #nlp #ai
This project was a highly collaborative effort with many people contributing translations, evaluations, analyses, etc., so I want to thank all of my co-authors! @ebriakou @iseeaswell @marafinkels Rebecca Galor @JurikJuraska @gezakovacs Alison Lui @RicardoRei7 @jasonriesa
As a bonus, we also release screenshots of the original English URLs to support future work on multimodal translation. huggingface.co/datasets/goog…
We also find that MT systems/LLMs produce better translations than the human translators (at least according to automatic metrics), a result that should be confirmed with a follow up study (hint: future work coming soon!)
We collect machine translations from a variety of different MT systems/LLMs and rank them with automatic metrics. Takeaway: LLMs get the best scores across all languages! Data coming to MTME soon github.com/google-research/m…
Each language has a human-written reference and subsequent post-edit (including for 8 of the original WMT24 languages)
🚨New machine translation dataset alert! 🚨We expanded the language coverage of WMT24 from 9 to 55 en->xx language pairs by collecting new reference translations for 46 languages in a dataset called WMT24++
Paper: arxiv.org/abs/2502.12404v1
Data: huggingface.co/datasets/goog…
Dan Deutsch retweeted
Thrilled to share our latest findings on data contamination, from my internship at @Google! We trained almost 90 Models on 1B and 8B scales with various contamination types using machine translation as our task and analyze the impact of contamination.
arxiv.org/abs/2501.18771
Dan Deutsch retweeted
🚀 We have just released bfloat16 variants of all 3 MetricX-24 models, offering nearly identical performance to their float32 counterparts, but with a 50% smaller memory footprint. ✨ We hope this makes the XL and XXL models more accessible!
🔗 GitHub: github.com/google-research/m…
🌐 Meet MetricX-24, our SOTA machine translation evaluation metric and a successor to the successful MetricX-23. 🚀 Now open-source in PyTorch/Transformers! 🎉 Ready to take this top performer in the WMT24 Metrics Shared Task for a spin?
🔗 Code: github.com/google-research/m…
Dan Deutsch retweeted
🌐 Meet MetricX-24, our SOTA machine translation evaluation metric and a successor to the successful MetricX-23. 🚀 Now open-source in PyTorch/Transformers! 🎉 Ready to take this top performer in the WMT24 Metrics Shared Task for a spin?
🔗 Code: github.com/google-research/m…
Super simple and effective way of significantly increasing the performance of your evaluation metric!
LLMs are typically evaluated w/ automatic metrics on standard test sets, but metrics + test sets are developed independently. This raises a crucial question: Can we design automatic metrics specifically to excel on the test sets we prioritize? Answer: Yes!
arxiv.org/abs/2411.15387
