Second Year MRes student @UTSEngage. Prev @XJTLU, @LivUni. AI for Social Good. #NLProc. INTJ 5w4
Sydney
Joined October 2021
- Tweets95
- Following641
- Followers54
- Likes277
Hongbin Na | 那洪彬 retweeted
📢 Just wrapped up some cool agent evaluation work for ICLR? Or are you simply interested in learning more about the evals landscape? Come to our Agent Evaluation Science Fall Symposium in New York City on November 20, 2026 where we bring together researchers & practitioners!
The day will focus on how agents should be scientifically evaluated, with topics spanning measurement and benchmark design, reliability, safety, and real-world performance. If you have some exciting work in this space, please consider submitting it for poster presentation!
Abstract registration deadline: October 20
Submission deadline: October 25
OpenReview: openreview.net/group?id=eval…
We are also excited to welcome a strong lineup of speakers, with more to be announced. Confirmed so far: @yoavartzi (Cornell Tech), @jaseweston (Meta AI), @nouhadziri (Cohere Labs), @PeterHndrsn (Princeton University), @walesalaudeen96 (Microsoft Research), @criticalneuro (Microsoft Research), and @saurabhjha2010 (IBM Research).
Important: You do not need to submit work in order to join! Just register through the Luma link below. If space becomes limited however, we will prioritize authors with accepted submissions.
Register: luma.com/cxmx1g4n
Details: evalscience.org/2026fall-nyc…
Huge thanks to the entire organizing team, including @yininghua, @HongbinNLP, @manuelcherep, @dubrovnik07, @HuaYilun, @ishapuri101, @liuzhuang1234, Lori Chibnik, @vbingliu, @CsabaSzepesvari, @PeterStone_TX, @sirrice, @ang3linawang, and @nikhilsinghmus for putting this together!
Feel free to reach out with any questions and we hope to see many of you there!
Hongbin Na | 那洪彬 retweeted
Can LLMs learn a new language using only a grammar book and a dictionary like how human adult L-2 learners do?
Check our in-progress paper! The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang arxiv.org/pdf/2509.00425
Hongbin Na | 那洪彬 retweeted
BREAKING NEWS! Most people aren’t prompting models with IMO problems :)
They’re prompting with tasks that need more context, like “plz make talk slides.”
In an ACL oral, I’ll cover challenges in human-LM grounding (in 60K+ real interactions) & introduce a benchmark: RIFTS.
🧵
Hongbin Na | 那洪彬 retweeted
While we’re building amazing new human-AI systems, how do we actually know if they work well for people?
In our #ACL2025 Findings Paper, we introduce SPHERE, a framework for making evaluations of human-AI systems more transparent and replicable.
✨aclanthology.org/2025.findin…
Hongbin Na | 那洪彬 retweeted
Today (w/ @UniofOxford @Stanford @MIT @LSEnews) we’re sharing the results of the largest AI persuasion experiments to date: 76k participants, 19 LLMs, 707 political issues.
We examine “levers” of AI persuasion: model scale, post-training, prompting, personalization, & more
🧵
Excited to present our survey on LLMs in Psychotherapy at #ACL2025!
Let's discuss a coherent path forward! Find me at my poster:
📍Location: Hall 4/5
🗓️Session: IP-Poster Session 5, Mon, July 28, 18:00-19:30
#LLM #AI #MentalHealth #Psychotherapy
Hongbin Na | 那洪彬 retweeted
Kimi K2 的一大亮点，是将文本任务里基于 token 的处理思路，成功迁移到 Agentic 场景中的 tool-call 级别：在 Agentic 任务中，tool call 就相当于“行动 token”。
什么意思呢？解释如下：
在文本任务中:
CoT 是一串 token
而在Agentic 场景中：
CoT 是一段 tool-call 序列，即planning
在文本任务中：
用BLEU 或ROUGE等衡量生成文本与黄金答案在 token或字符层面的相似度，如具体的machine translation, summarization 等任务。
在Agentic 场景中
Process Accuracy，用来衡量实际 tool-call 序列与理想动作轨迹的匹配度，颗粒度（这个词挺合适的😅）是tool call 级别的。
如此一来，Kimi K2 在Agentic 场景中实现了与传统 NLP 的联系，使模型的规划与执行得以像文本生成那样可度量和优化。
Hongbin Na | 那洪彬 retweeted
Most complex tasks feature ill- and underdefined goals, at least initially.
Check out @vysrini's paper demonstrating that agents need to do a lot more than follow instructions, incl. refining goals, and balance competing objectives.
Current frontier models fail at these tasks
Excited to share our paper, Substance over Style: Evaluating Proactive Conversational Coaching Agents, accepted to ACL 2025 Main Track!
LLMs do well when given clear context up front.
But what if that context doesn't exist yet?
(1/5)
🔗Read more: arxiv.org/abs/2503.19328
Hongbin Na | 那洪彬 retweeted
Agentic 的文章，因为采用的post training recipe类似，方法同质化非常严重，容易带来审美疲劳，往往读完abstract 就不想往下读了。
但依然有一些文章，在RL天然task-specific的限制下，可以让task本身超越benchmark，颇具美感。
这几个月我个人最喜欢的几篇：
Absolute Zero：
高度地将代码类推理任务为三类： Deduction 演绎， Abduction 溯因， Induction 归纳。
因而，模型学习到的是这三类高度的任务，超越coding。
Test-Time Interaction：
关注的是Agent 决策中的两个基本核心策略：Exploration (探索) 与 Exploitation (利用)。
CURE：
利用不同Agent之间的关系，生成-验证，对抗-互测，完成自动监督，获得更加多样和动态的信号。
WebThinker：
不是让LLM 仅仅学会几个动作，而是让 LLM 真正掌握一套专业的 workflow。
RL的task-specific属性，让很多工作局限在数据集中，但恰恰是这个task，可以很具体，也可以实现高度的抽象。
就像Pretraining task，无论是masked language modeling还是next token prediction，非常简单直接，却能够让LLM发展出解决复杂问题的能力，大道至简。
Hongbin Na | 那洪彬 retweeted
How do language models track mental states of each character in a story, often referred to as Theory of Mind?
Our recent work takes a step in demystifing it by reverse engineering how Llama-3-70B-Instruct solves a simple belief tracking task, and surprisingly found that it relies heavily on concepts similar to pointer variables in C programming!
Hongbin Na | 那洪彬 retweeted
Amazing thread. So much to unpack but for starters: “After writing, only 17 % of ChatGPT users could quote their own sentences, versus 89 % in the brain-only group.”
It’s a hefty 206-page research paper, and the findings are concerning.
"LLM users consistently underperformed at neural, linguistic, and behavioral levels"
This study finds LLM dependence weakens the writer’s own neural and linguistic fingerprints. 🤔🤔
Relying only on EEG, text mining, and a cross-over session, the authors show that keeping some AI-free practice time protects memory circuits and encourages richer language even when a tool is later reintroduced.
Hongbin Na | 那洪彬 retweeted
Check out our ACL paper! We use shapley interactions to see which words (and phones) interact non-linearly -- what we lose when we assume linear relationships between features. Chat to Diganta in Vienna!
Hongbin Na | 那洪彬 retweeted
"Fiction-brained". This needs to be the next big cultural analytics project.
If you consume 100x as much fiction as you do history, you will subconsciously derive your beliefs from fiction. This is a well-studied finding in psychology called “source confusion”: you remember things you “learned” long after you’ve forgotten where you learned them. This applies to moral “lessons” derived from fiction as well. In fact, this is the entire reason we read morality tales with talking animals to our children: we know they will internalize the “lessons” in those stories, even though the stories are fake. After a lifetime of consuming such made up stories, you end up in a state I call “fiction-brained”: your beliefs about the world are subconsciously based primarily on movies, novels, etc. Propagandists have always relied on this effect: Nazi Germany released a dozen fictional movies that everyone knew were fiction but whose plots involved Jews doing villainous things. But the average American today hasn’t watched a dozen hours of fictional content, they’ve watched tens of thousands, making them the most propagandized people to ever live. The only way to combat this on a personal level is to consume much less fiction and to interact much more with the real world and consume history, biography, etc to unlearn your false beliefs.
Hongbin Na | 那洪彬 retweeted
What if LLMs could learn your habits and preferences well enough (across any context!) to anticipate your needs?
In a new paper, we present the General User Model (GUM): a model of you built from just your everyday computer use.
🧵
Hongbin Na | 那洪彬 retweeted
When it comes to text prediction, where does one LM outperform another? If you've ever worked on LM evals, you know this question is a lot more complex than it seems. In our new #acl2025 paper, we developed a method to find fine-grained differences between LMs:
🧵1/9
Hongbin Na | 那洪彬 retweeted
Happy to share our preprint "Words Like Knives: Backstory-Personalized Modeling and Detection of Violent Communication" 🧵(1/9)
📄Paper: arxiv.org/pdf/2505.21451
Hongbin Na | 那洪彬 retweeted
Do people actually like human-like LLMs? In our #ACL2025 paper HumT DumT, we find a kind of uncanny valley effect: users dislike LLM outputs that are *too human-like*. We thus develop methods to reduce human-likeness without sacrificing performance.
Hongbin Na | 那洪彬 retweeted
In case you're curious whether you can use LLMs (including small local ones) for narrative topic modeling. The answer is yes! LLMs perform on par or better than humans at the task.
Hongbin Na | 那洪彬 retweeted
📢 New Paper!
Tired 😴 of reasoning benchmarks full of math & code? In our work we consider the problem of reasoning for plot holes in stories -- inconsistencies in a storyline that break the internal logic or rules of a story’s world 🌎
W/ @melaniesclar, and @tsvetshop
1/n
ALT A screenshot of the first page of the paper, containing the paper title: Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection and the names of the authors: Kabir Ahuja, Melanie Sclar, and Yulia Tsvetkov. All the three authors are from CSE department in the University of Washington in Seattle, USA. They can be reached at {kahuja,msclar,yuliats}@cs.washington.edu