@_danieldeutsch

Research Scientist at Google DeepMind working on multilinguality

 San Francisco
danieldeutsch.github.io/
Joined September 2012
  • Tweets94
  • Following92
  • Followers599
16Media
Dan Deutsch retweeted
Wenda Xu
@WendaXu2Sep 4
We were asking one question a year ago: "Can we trust LLMs to build and grade their own benchmarks"? We tested frontier models on automated LLM benchmarking. The result? Every single model ranks itself #1 on benchmarks it generates (arxiv.org/abs/2509.26600). Our work has been accepted to EMNLP 2026 Main!
2
8
24
1,405
Dan Deutsch retweeted
Adam Fisch@adamjfischJul 8
Happy to share CollabEval, recent(-ish) work from @GoogleDeepMind and @GoogleResearch that can be used to improve the statistical efficiency of model evaluation by exploiting correlations between historical evals of different models on the same tasks. arxiv.org/pdf/2607.05046
2
6
1
29
3,250
Dan Deutsch retweeted
Vilém Zouhar@zouharviMar 12
Machine translation is tough to evaluate, partly because most of what you throw at is too easy. That doesn't at all mean that translation is solved; we're just not doing a good job finding interesting inputs.
1
1
16
869
Dan Deutsch retweeted
John Hewitt
@johnhewtt19 Nov 2025
Come do a PhD with me at Columbia! My lab tackles basic problems in alignment, interpretability, safety, and capabilities of language systems. If you love adventuring in model internals and behaviors---to understand and improve---let's do it together! pic: a run in central park
12
128
4
948
79,888
🗺️ Are we making our #LLMs multilingual, or anglocentric? Much work brings languages closer to English, but that comes at the cost of crucial #cultural nuance. @h__j___han tackles this trade-off with surgical steering, adapting LLMs to cultural contexts at inference time.
HyoJung Han@h__j___han31 Oct 2025
Lots of work on cross-lingual alignment encourages multilingual LLMs to generalize knowledge across languages. But this push for uniformity creates a tension: what happens to knowledge that should remain local? We look into this trade-off of transfer and cultural erasure:🧵
11
50
9,092
Our Google Translate team is bringing a strong presence to #ACL2025 in Vienna this week! 🇦🇹 My group is excited to present several of our latest papers. 👇 Don't miss them!
1
5
52
3,298
Two new datasets from Google Translate targeting high and low resource languages! WMT24++: 46 new en->xx languages to WMT24, bringing the total to 55 SMOL: 6M tokens for 115 very low-resource languages WMT24++: huggingface.co/datasets/goog… SMOL: huggingface.co/datasets/goog…

google/wmt24pp · Datasets at Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co
2
22
3
82
15,646
😼SMOL DATA ALERT! 😼Anouncing SMOL, a professionally-translated dataset for 115 very low-resource languages! Paper: arxiv.org/pdf/2502.12301 Huggingface: huggingface.co/datasets/goog…
3
12
3
35
4,247
Thrilled to share our latest findings on data contamination, from my internship at @Google! We trained almost 90 Models on 1B and 8B scales with various contamination types using machine translation as our task and analyze the impact of contamination. arxiv.org/abs/2501.18771

Overestimation in LLM Evaluation: A Controlled Large-Scale Study...

Data contamination -- the accidental consumption of evaluation examples within the pre-training data -- can undermine the validity of evaluation benchmarks. In this paper, we present a rigorous...

arxiv.org
3
20
3
84
12,440
🚀 We have just released bfloat16 variants of all 3 MetricX-24 models, offering nearly identical performance to their float32 counterparts, but with a 50% smaller memory footprint. ✨ We hope this makes the XL and XXL models more accessible! 🔗 GitHub: github.com/google-research/m…

GitHub - google-research/metricx

Contribute to google-research/metricx development by creating an account on GitHub.

github.com
Jurik Juraska@JurikJuraska3 Dec 2024
🌐 Meet MetricX-24, our SOTA machine translation evaluation metric and a successor to the successful MetricX-23. 🚀 Now open-source in PyTorch/Transformers! 🎉 Ready to take this top performer in the WMT24 Metrics Shared Task for a spin? 🔗 Code: github.com/google-research/m…

GitHub - google-research/metricx

Contribute to google-research/metricx development by creating an account on GitHub.

github.com
2
2
366
🌐 Meet MetricX-24, our SOTA machine translation evaluation metric and a successor to the successful MetricX-23. 🚀 Now open-source in PyTorch/Transformers! 🎉 Ready to take this top performer in the WMT24 Metrics Shared Task for a spin? 🔗 Code: github.com/google-research/m…

GitHub - google-research/metricx

Contribute to google-research/metricx development by creating an account on GitHub.

github.com
1
5
3
17
2,352
[1/5] Are verbose #LLM translations skewing evaluation results? TLDR: Yes! Our recent work dives into the prevalence and impact of LLM verbosity in automatic and human evaluations. 📎 Paper: arxiv.org/pdf/2410.00863
1
11
48
4,509
Dan Deutsch retweeted
Wenda Xu
@WendaXu24 Aug 2024
I am excited to attend ACL2024 to present our proud paper "Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement" (We received scores of 5, 4.5, 4 and a meta-review of 5🤲). I am looking for full-time industry positions. Here is a list of my first-author publications and research interests:
5
16
1
98
20,727
📢 WMT24 Metric Shared Task Starts NOW! 🎉 Calling all researchers and metrics enthusiasts! We invite you to submit your best evaluation metrics to the WMT Metric Shared Task. Assign quality scores to all WMT submissions and translations by top LLMs like Gemini, GPT4o, Claude, LLambda, and more. Let's push the boundaries of MT evaluation together! #WMT #MachineTranslation #NLProc All details including information how to download the data: www2.statmt.org/wmt24/metric… Deadline for submissions is 12:00 pm July 31 (AoE) Let's go! :-)
1
11
24
2,431
Excited to announce that 110 languages got added to Google Translate today! Time for context on these languages, especially the communities who helped a lot over the past few years, including Cantonese, NKo, and Faroese volunteers. Also, a 110-language youtube playlist. 🧵
14
55
12
228
49,578
New paper alert! Designing reliable human evaluation is both crucial and difficult. Human raters can exhibit different behaviors when rating NLG outputs. These differences are not generally due to a rater performing the task incorrectly, but rather due to differences in harshness or leniency between raters: a Minor error to one rater may be a Major error to another. Consequently, decisions around which raters rate which items can alter the final system ranking. In our new paper, we analyse the impact of rater assignment on the final system ranking and show how you can design a replicable, reliable human evaluation by assigning the right raters to the right items. Take a look: arxiv.org/pdf/2404.01474.pdf
2
16
73
8,594
Exciting News! Introducing our new Quality-Aware Machine Translation model! 🚀 100x faster MBR decoding 🔝 Improved translation quality 🧠 Self-evaluation and guidance for top-notch translations Take a look: arxiv.org/abs/2310.06707
1
7
31
1,819
We're looking for a final-year PhD intern passionate about working on automatic metrics for machine translation and NLP in Mountain View. If interested, please send an email to me and @_danieldeutsch. The ideal candidate should have experience working on automatic evaluation and be familiar with metrics like Metric-X, COMET, BLEURT and human evaluation frameworks like MQM.
1
13
2
44
6,526
Dan Deutsch retweeted
Wenda Xu
@WendaXu216 Nov 2023
Can we not criticize LLM but pinpoint errors it makes and automatically guide it with fine-grained actionable feedback? Can we formulate iterative refinement into a local search problem, simulated annealing? My cool summer intern work @Google @_danieldeutsch @markuseful @ucsbNLP
8
24
1
124
19,074
Dan Deutsch retweeted
Eval4NLP@eval4nlp30 Oct 2023
Our accepted papers and program are now online: eval4nlp.github.io/2023/prog… eval4nlp.github.io/2023/acce… Moreover, we're excited to have @alexfabbri4 as invited speaker on the topic of "Re-Evaluating Summarization Evaluation in the Era of LLMs" See u tomorrow 9am (UTC+8), online only!
4
1
12
2,087
Load more