AI Researcher @CapitalOne AIF. Ex @TechAtBloomberg @BigScienceW @SFResearch @hkust. Working on multilingual and LLM #NLProc.
Joined December 2011
- Tweets1.3K
- Following863
- Followers1.9K
- Likes7.8K
Genta Winata retweeted
๐ฅณ Happy to share that PingPong has been accepted to EMNLP 2026 - Main Conference!
Special thanks to all of my collaborators: @hanifmz07, @faridlazuarda, Maryam**, @Shmuhammadd, @nedjmaou, @sathready, @ponguru, @KodaliPrashant, Sunayana**, @gentaiscool, @AyuP_AI, @AlhamFikri
โ ๏ธNew Paper Alertโ ๏ธ
Introducing PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues
Ever seen a 4-player ping-pong game with chaotic turn-taking? Thatโs a perfect analogy for multi-party polyglot in code-switching conversations.
We introduce PingPong, a benchmark for natural multi-party code-switching dialogues covering five language-combination variations, some of which are trilingual and one that involves a different writing script.๐๐
๐Paper: arxiv.org/pdf/2601.17277
๐คDatasets: huggingface.co/code-switchinโฆ
Huge thanks to all of my collaborators: @hanifmz07, @faridlazuarda, @Shmuhammadd, Maryam**, @nedjmaou, @gentaiscool, @AyuP_AI, @AlhamFikri.
Check this thread out! ๐งต
#PingPongDataset #CodeSwitching #Multilinguality #Polyglot #NLP #LLM #Benchmark
(1/n)
Genta Winata retweeted
-- Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection (Oral)
@niu_tianyi , @cyjustinchen , @gentaiscool , Shi-Xiong Zhang, Supriyo Chakraborty, Sambit Sahu, @zhan1624 , @EliasEskin
arxiv.org/abs/2601.09692
#ACL2026 Harbor D: Session 17: Oral Session G: Tues. July 7, 11:00-12:30 PT
xitter.cf/EliasEskin/status/2011โฆ
๐ข Introducing Routing with Generated Data (RGD), a new setting for annotation-free LLM routing. We study how routers can be trained without any ground-truth labels. We also introduce CASCAL, a novel label-free LLM router that identifies niche skills using consensus-voting and hierarchical clustering.
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
Genta Winata retweeted
2/ #ACL2026 Tues. July 7, 11:00-12:30
Harbor D: Session 17: Oral Session G
Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection
@niu_tianyi, @cyjustinchen, @gentaiscool, Shi-Xiong Zhang, Supriyo Chakraborty, Sambit Sahu, @zhan1624, @mohitban47
arxiv.org/abs/2601.09692
xitter.cf/EliasEskin/status/2011โฆ
๐ข Introducing Routing with Generated Data (RGD), a new setting for annotation-free LLM routing. We study how routers can be trained without any ground-truth labels. We also introduce CASCAL, a novel label-free LLM router that identifies niche skills using consensus-voting and hierarchical clustering.
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
Genta Winata retweeted
๐จ Excited to announce that RGD has been accepted to #ACL2026 Main!
Routing with Generated Data (RGD) is a new LLM routing paradigm where routers estimate skills of models using generated data, without ground-truth labels.
We further introduce CASCAL, a new router for RGD that discovers niche skills via consensus voting + hierarchical clustering, with no ground truth needed.
๐งต๐
๐ข Introducing Routing with Generated Data (RGD), a new setting for annotation-free LLM routing. We study how routers can be trained without any ground-truth labels. We also introduce CASCAL, a novel label-free LLM router that identifies niche skills using consensus-voting and hierarchical clustering.
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
Genta Winata retweeted
VLMs can easily get distracted by unrelated cultural cues. Happy to present our work on this soon at #CVPR2026๐ฅณ
Working on multilingual VLMs? Consider using our benchmark:
๐arxiv.org/pdf/2511.17004
๐คhuggingface.co/datasets/patrโฆ
Amazing work by @patrickamadeus_ and colleagues!
Excited to share that we have committed our paper โVision-Language Models are Confused Touristsโ to #CVPR2026 (Findings)! ๐บ๐ธ๐
Arxiv: arxiv.org/abs/2511.17004
We question whether current SOTA VLMs remain robust in simple cultural grounding QA when distracting contextual objects are present
For example, if you eat chicken schnitzel with Mt. Fuji in the background, will the model fail to recognize it as Japanese katsu?
ConfusedTourists introduces:
๐ 5k+ evaluation samples across 3 cultural item categories, comprising 243 unique cultural items from 57 countries and 11 sub-regions ๐
๐ Evaluation of 14 VLMs across 12 data features ๐ค
๐ Findings showing that simple concept mixing can cause up to a -40% drop in perform ๐
Special thanks to my co-authors @IkhlasulHanif0 , @emthehunt, @gentaiscool, @FajriKoto, and my advisor @AlhamFikri for the valuable contributions along the way!
#multimodal #vlm #multicultural #robustness #evaluation #NLProc #ComputerVision
Happy to have my first Nature paper. Thank you @CAIS for the collaboration nature.com/articles/s41586-0โฆ
Genta Winata retweeted
Last week, Humanityโs Last Exam was published in @Nature.
In just over a year, model scores on HLE have risen from under 5% to nearly 40%.
Thank you to @scale_AI and the 1000+ HLE co-authors for helping policymakers and the public track these rapid advances in AI capabilities.
Genta Winata retweeted
๐ข Introducing Routing with Generated Data (RGD), a new setting for annotation-free LLM routing. We study how routers can be trained without any ground-truth labels. We also introduce CASCAL, a novel label-free LLM router that identifies niche skills using consensus-voting and hierarchical clustering.
โก๏ธ Most LLM routers assume access to labeled, in-domain data to estimate model skills (query-answer routers). However, user distributions are unknown and labels are expensive or unavailable, highlighting the need for routers that work without labels.
โก๏ธ We introduce Routing with Generated Data (RGD): routers are trained only on Q&A data generated from task descriptions, without human annotation. We experiment with various LLM generators of different strengths (Gemini-2.5-Flash, Qwen-3-32B, Exaone-3.5-7.8B).
โก๏ธ CASCAL outperforms other query-answer and query-only routers across diverse datasets (MMLU-Pro, SuperGPQA, MedMCQA, BigBench Extra Hard), and is more robust to weaker generators.
๐กHave you ever wondered whether visionโlanguage models can be easily tricked by adding landmarks or flags to an image?
In the spirit of the holidays๐, we show that VLMs can indeed be easily confused like "Confused Tourists" โ๏ธ: their performance drops significantly when such image perturbations are applied.
๐ Check out "VLMs are Confused Tourists" โ๏ธ here arxiv.org/pdf/2511.17004
#vision #nlproc #robustness
Craving holiday-themed paper? Say less๐
Turns out, Vision Language Models are Confused Tourists โ๏ธ๐ตโ๐ซ
We show that adversarially induced cultural scenes significantly impair VLM cultural comprehension and trigger potential bias
#NLProc #multimodal #robustness
/thread ๐งต(1/8)
Genta Winata retweeted
Craving holiday-themed paper? Say less๐
Turns out, Vision Language Models are Confused Tourists โ๏ธ๐ตโ๐ซ
We show that adversarially induced cultural scenes significantly impair VLM cultural comprehension and trigger potential bias
#NLProc #multimodal #robustness
/thread ๐งต(1/8)
Genta Winata retweeted
One last call for interested applicants! Applications for the SEACrowd Apprentice Program 2026 will end on December 17, 23:59 (UTC-12). See the post below for more details.
๐ Applications are now open for the SEACrowd Apprentice Program 2026!
Join a 3โ4 month guided research journey where youโll collaborate with mentors.
๐๏ธ Apply between Nov 17 โ Dec 17, 2025 (UTC-12)
๐
Program runs Feb โ Jun 2026
seacrowd.org/apprenticeship
Genta Winata retweeted
Weโve just shared a full retrospective of the first batch our Apprentice Program from research wins to real lessons learned along the way. Itโs inspiring to see how our SEA mentees transformed ideas into papers and datasets through mentorship.๐๐
seacrowd.org/posts/apprenticโฆ
Genta Winata retweeted
Excited to be at #NeurIPS 2025 in San Diego this week! ๐ด๐ค
Iโm presenting our UniReps workshop paper on Dec 6: Leveraging Parameter Space Symmetries for Reasoning Skill Transfer in LLMs.
Come say hi! Would love to connect and exchange ideas
Paper: arxiv.org/abs/2511.10850
Genta Winata retweeted
๐ Applications are now open for the SEACrowd Apprentice Program 2026!
Join a 3โ4 month guided research journey where youโll collaborate with mentors.
๐๏ธ Apply between Nov 17 โ Dec 17, 2025 (UTC-12)
๐
Program runs Feb โ Jun 2026
seacrowd.org/apprenticeship
Genta Winata retweeted
๐ Following R3, we are thrilled to introduce mR3: Multilingual Rubric-Agnostic Reward Reasoning Models ๐
TL;DR: mR3 is a multilingual version of R3, a rubric-agnostic reward reasoning model trained on 72 languages, achieving the broadest language coverage in reward modeling to date. mR3 is able to surpass much larger models, such as GPT-OSS-120B, while being up to 9x smallerโก!
๐ Paper: arxiv.org/abs/2510.01146
๐ป Code: github.com/rubricreward/mr3
๐ค mR3 Models: huggingface.co/collections/rโฆ
๐ค mR3 Datasets: huggingface.co/collections/rโฆ
Huge thanks to all of my collaborators: @HLeiTR,
@Zilu_Tang_Peter, @ Annie Lee, @DerryTWijaya1, @llamagrp, and @gentaiscool for all of their efforts into creating this work!
Check this thread out! #mR3 #RewardModel #NLP #LLM #Metrics #Alignment #Preference (1/4)
Extremely proud to share that our paper: ๐๐: ๐ ๐๐จ๐จ๐ฅ-๐๐ซ๐ข๐๐ง๐ญ๐๐ ๐๐จ๐ง๐ฏ๐๐ซ๐ฌ๐๐ญ๐ข๐จ๐ง๐๐ฅ ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ฎ๐ฅ๐ญ๐ข-๐๐ฎ๐ซ๐ง ๐๐ ๐๐ง๐ญ๐ข๐ ๐๐ฅ๐๐ง๐ง๐ข๐ง๐ has been accepted into @NeurIPSConf 2025!
We introduce T1, a tool-augmented, multi-domain, multi-turn conversational dataset specifically designed to capture and manage inter-tool dependencies across diverse domains with a focus on the travel space.
I would like to shoutout all of my amazing co-authors Amartya Chakraborty, Nadia Bathaee, @pareshdashore19, Anmol Jain, @zzeet, Shi-Xiong ZHANG, Sambit Sahu, and Milind Naphade who all worked incredibly hard to make this paper possible!
Paper Link: arxiv.org/pdf/2505.16986
#nlproc #ml #agents
Key Highlights:
- ๐๐, a comprehensive multi-turn dataset consisting of 13.5k dialogues designed to evaluate tool using, LLM-based agents across nine key domains. The dataset covers a wide range of interaction scenarios, including single-domain, mixed-domain, and fully multi-domain conversations. It incorporates 14 distinct tools, enabling realistic and fine-grained assessment of agent capabilities in complex, tool-driven dialogue tasks.
- Cross-domain tasks and interdependent tool calls within the dataset to increase complexity and realism of the dataset, requiring agents to reason about tool selection and execution order within context.
- ๐๐-๐๐๐๐๐, An LLM based code-generation system, built on open-source language models and equipped with a caching mechanism for improved performance. This architecture enables scalable evaluation and provides a robust, efficient framework for tool-using agents.