@Penn Natural Language Processing group
Philadelphia
Joined December 2022
- Tweets92
- Following31
- Followers1.1K
- Likes109
UPenn NLP retweeted
In a globalized world, every event has ripple effects. When LLMs turn to the open web, how do they reflect state media?
We release the WorldStateMedia dataset: 118M articles from 136 countries in 114 languages, and study their prevalence, narratives, and impact on LLMs.
1/6🧵
If you are at @COLM_conf today, check out our poster on verification failures in compositional (scientific) claims! #COLM2026
Tomorrow at #COLM26, we are sharing some troubling findings regarding modern LLMs, especially when used for verifying scientific/medical claims. These LLMs score very well on benchmarks, but are they verifying claims or taking a shortcut? We show it's mostly the shortcut. 🧶
UPenn NLP retweeted
You already know Jev is fast and cheap. Our new arXiv draft shows, it makes the correlated mistakes as other flash-tier LLMs. The consequence: you don't gain much by cascading Jev's "low confidence" outcomes to an LLM! Explanation in thread. arxiv.org/abs/2609.29769
UPenn NLP retweeted
New paper out in @ScienceAdvances!
News organizations shape public understanding through the choices they make about what to cover and how to cover it. How can we measure these differences across publishers systematically and at scale?
We present the Media Bias Detector!
UPenn NLP retweeted
How to evaluate deep research at scale by rubric?
Check out our #ACL2026 (TACL) paper!
Poster: Tue. Jul 7 9:00–10:30; Poster Session F
realliyifei.github.io/assets…
How well can LLMs & deep research systems synthesize long-form answers to *thousands of research queries across diverse domains*?
Excited to announce 🎓📖 ResearchQA: a large-scale benchmark to evaluate long-form scholarly question answering at scale across 75 fields, using queries 💬and rubrics📋that are mined from survey articles 📚!
Website: cylumn.com/ResearchQA
Paper: arxiv.org/abs/2509.00496
Dataset: huggingface.co/datasets/real…
Code: github.com/realliyifei/Resea…
UPenn NLP retweeted
Last year, we built a scientific feasibility estimation agent for DARPA, which produced results with ZERO hallucinated references. In the paper, we unpack that and publish an open source CLI tool and agent skill (link in reply) that reveals how we did it.
Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
@deliprao et al. find 3–13% of citation URLs from commercial LLMs & agents are hallucinated & release an open-source tool that reduces non-resolving citations
📝arxiv.org/abs/2604.03173
UPenn NLP retweeted
Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
@deliprao et al. find 3–13% of citation URLs from commercial LLMs & agents are hallucinated & release an open-source tool that reduces non-resolving citations
📝arxiv.org/abs/2604.03173
UPenn NLP retweeted
What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis
Delip Rao, Chris Callison-Burch
arxiv.org/abs/2604.01657 [𝚌𝚜.𝙲𝙻]
UPenn NLP retweeted
📷 New #EMNLP2025 Findings survey paper!
“Conflicts in Texts: Data, Implications, and Challenges” Paper: aclanthology.org/2025.findin…
Conflicts are everywhere in NLP — news articles reflecting different perspectives or opposing views, annotators who disagree, LLMs that hallucinate or contradict themselves, and personal/enterprise document collections that grow apart and are conflicting. Most research tackles these in isolation, and our survey provides the first unified view of conflicting information in NLP. We chart the path toward conflict-aware, reliable NLP systems.
Builds on our earlier work on:
- Multi-perspective dataset aclanthology.org/2021.naacl-… and search aclanthology.org/2022.findin…
- Hallucination detection aclanthology.org/2025.findin…
- Open-domain QA with conflicting contexts aclanthology.org/2025.findin…
UPenn NLP retweeted
If you are at EMNLP 2025, check out this cool work led by @WeiqiuYou and chat with her about LLM reasoning soundness guarantees. Weiqiu is wrapping up her PhD and just entering the job market; she’s smart, tenacious, and overall amazing. I highly recommend her for your team.
I'll be presenting our work "Probabilistic Soundness Guarantees in LLM Reasoning Chains" at EMNLP 2025
Today (Nov 5) Hall C 14:30-16:00 802-Main
Blog: debugml.github.io/ares
Paper: arxiv.org/abs/2507.12948
Code: github.com/fallcat/ares
UPenn NLP retweeted
📆The full agenda for LM4Sci Workshop (10 Oct 2025, Montreal 🇨🇦): keynotes, panels, poster sessions & more!
lm4sci.github.io/docs/schedu…
UPenn NLP retweeted
How well can LLMs & deep research systems synthesize long-form answers to *thousands of research queries across diverse domains*?
Excited to announce 🎓📖 ResearchQA: a large-scale benchmark to evaluate long-form scholarly question answering at scale across 75 fields, using queries 💬and rubrics📋that are mined from survey articles 📚!
Website: cylumn.com/ResearchQA
Paper: arxiv.org/abs/2509.00496
Dataset: huggingface.co/datasets/real…
Code: github.com/realliyifei/Resea…
UPenn NLP retweeted
Many factual QA benchmarks have become saturated, yet factuality still poses a very real issue!
✨We present MoNaCo, an Ai2 benchmark of human-written time-consuming questions that, on average, require 43.3 documents per question!✨
📣Blogpost: allenai.org/blog/monaco
🧵(1/5)
UPenn NLP retweeted
In a world of geopolitical conflicts, how can AI help us navigate? Our #ACL2025-F work studies RAG robustness across 49 languages.
TL;DR: 📈 boost robustness w/ multilingual RAG, 🤔 take care w/ low-resource citations
📜arxiv.org/abs/2410.01171
🤗huggingface.co/datasets/bord…
1/4 🧵
UPenn NLP retweeted
Ever wondered what makes language models generate overly verbose, vague, or sycophantic responses?
Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓
UPenn NLP retweeted
🤖💬 Herding instincts… in AIs? Yes, even LLMs can follow the crowd!
• 📉 Conformity ↑ when agents lack confidence but trust peers
• 🧠 Presentation format shapes peer influence
• 🎯 Controlled herding can boost collaboration outcomes
👉 Read more: arxiv.org/abs/2505.21588
UPenn NLP retweeted
#NAACL2025
How to compare cultural differences with social media data in scale? Our work uses lexica to annotate X 🇺🇸 & Weibo 🇨🇳 posts with valence (😄☹️) & arousal (🔥❄️) scores, revealing cross-cultural differences in emotional expression.
aclanthology.org/2025.findin…
