PostDoc @amsterdamumc / NLP4Health. Prev. @UvA_Amsterdam/@illc_amsterdam. PhD @UniTrento_DISI - @AmazonScience. MSc @cimec_unitrento, BSc @Unibo.
Amsterdam
Joined January 2012
- Tweets70
- Following529
- Followers240
- Likes284
Alberto Testoni retweeted
Today, our team will share “From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions” at @aclmeeting! 🎉
Led by:
@carraznathanael @mhamdy_res @jaa_campos Lucas Weber @alberto_testoni @mziizm
@sandropezzelle @marcodeltredici
📜arxiv.org/abs/2502.13791
Alberto Testoni retweeted
Can LLMs collaborate effectively over long-term interactions, like a human teammate, especially in coding tasks? 🤔
We introduce MEMORYCODE, a new way to evaluate LLMs' ability to track & execute coding instructions across multiple sessions, even with irrelevant info thrown in.
Alberto Testoni retweeted
Excited to share insights from our new paper on evaluating LLMs in multi-session coding interactions! 📚📚📚
We introduce MEMORYCODE, a novel dataset to assess how well LLMs track & execute coding instructions across multiple sessions, mimicking real-world collaboration.
3/4 We find significant limitations of all models in responding to these questions. But what can go wrong when ambiguity is not recognized? In RAcQUEt-Bias, we analyze a critical yet underexplored problem: failing to address ambiguity can lead to stereotypical responses.
4/4 Our results reveal significant limitations and problems of overconfidence of state-of-the-art large V&L models. For more analyses on the role of the saliency features that guide the model selection and on CoT prompting, check out our paper! 🏓arxiv.org/abs/2412.13835
1/4 Excited to share our latest paper “🏓 RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs”. arxiv.org/abs/2412.13835 Joint work with @barbara_plank and @raquel_dmg. #NLProc 🧵
Alberto Testoni retweeted
🔍 Papers being presented:
1️⃣ Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain
@dmazzaccara, @alberto_testoni, @raffagbernardi
📰 arxiv.org/abs/2406.17453
Alberto Testoni retweeted
Flying to Miami! I will present “Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain” at #EMNLP2024 🌴
If you are interested in questions’ generation come to say hi!
⏱️ Wednesday 13th 16-17
📃 aclanthology.org/2024.findin…
ALT Our poster (poster session 09, 16-17:30)
Alberto Testoni retweeted
PhD opportunities in Munich 🥳 - consider applying to MCML and reach out if you are interested in @MaiNLPlab research themes: human-centric NLP, variation, uncertainty, dialects, multimodality & much more -- mainlp.github.io/
The #MCML offers fully funded #PhD positions in any field of research within the domains of machine learning!
The Online Application System is open from 15 October 2024 until 14 November 2024!
The position starts on 1 July 2025
Further information: mcml.ai/opportunities/fundin…
I am attending #ACL2024 in Bangkok with 2 papers on multimodal #NLProc 🇹🇭 🧵
1) "Naming, Describing, and Quantifying Visual Objects in Humans and LLMs" with @sandropezzelle and J. Sprott (poster 12/8 at 14:00 - with a fun game for attendees) arxiv.org/abs/2403.06935
2) "Don't Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models" work led by @anna_bavaresco_ with @raquel_dmg (poster 12/8 at 14:00 + oral 13/8 11:45) arxiv.org/abs/2405.20846
1/5 📣 Excited to share “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks”! arxiv.org/abs/2406.18403 🚀 We introduce JUDGE-BENCH, a benchmark to investigate to what extent LLM-generated judgements align with human evaluations. #NLProc
4/5 📊 The gap between open and closed models is narrowing, indicating promising prospects for reproducibility. When evaluating different linguistic dimensions, GPT-4o and Gemini-1.5 perform best in acceptability and verbosity, while Mixtral leads in coherence and consistency.
5/5 ⚠️ We conclude that LLMs are not yet ready to systematically replace human judges in NLP, and caution against using LLMs for this purpose. JUDGE-BENCH is intended as a living benchmark, and you are welcome to contribute: github.com/dmg-illc/JUDGE-BE…
Alberto Testoni retweeted
I'm looking for a last-minute emergency reviewer for a COLM submission related to generation with LLMs. Reviews need to be in by end of Thursday, May 23. Please DM if you can help out! 🙏 (PhD students with some reviewing experience welcome)
Alberto Testoni retweeted
22 researchers from 12 European institutions discussed future directions in open #LLMs and multimodal language technologies at the Oberwolfach Research Institute for Mathematics. The workshop was organised by @raquel_dmg & @sandropezzelle on behalf of the ELLIS #NLP program! #ML
Last week we had a very inspiring @ELLISforEurope Natural Language Processing workshop at the Mathematisches Forschungsinstitut Overwolfach (MFO). Thanks @raquel_dmg and @sandropezzelle for organizing it!
