Yepeng retweeted
Do LLMs know when they have enough information to answer, and what to ask a human when they don’t? @_yepeng
We spend a lot of time developing agents that can answer the questions we ask them. But the reverse direction gets much less attention: when the prompt does not contain enough information, can a model recognize what is missing and ask for it?
Paper: arxiv.org/abs/2608.14808
Project: zitniklab.hms.harvard.edu/MT…
For agents that interact with humans, biological systems, or clinical workflows, we need to evaluate not only whether they can answer questions, but also whether they know what they need to know before answering.
(1) Consider an agent evaluating a patient with pelvic pain. The duration of the pain determines which diagnostic pathway applies, so until the agent asks about it, no downstream symptom or test is clearly relevant.
(2) Now consider an agent reasoning about a gene regulatory network, where the outcome depends on the unknown states of several genes. Acquiring one of them can leave several outcomes possible, so the agent must identify the full set that determines the outcome.
Both cases raise the same three questions. Does the model recognize that information is missing? Does it identify which information it needs? Does it know when it has enough to answer?
New preprint! Do LLMs know what to ask, and when?
Website: zitniklab.hms.harvard.edu/MT…
Paper: arxiv.org/abs/2608.14808
(1/n)
We evaluate closed and open models on MT-InfoSeek, 5,251 problems spanning math, logic, biology, medicine, and general knowledge. Models usually notice that something is missing, but underestimate how much, ask for less informative things, and stop too early.
We wanted to know whether LLMs can do this. We formalize multi-turn information seeking as solving a k-underspecified constraint satisfaction problem, which gives verifiable ground truth for what a model should ask and when it has enough.
Almost every real decision is made under incomplete evidence. A doctor asks, examines, and orders the right test. A scientist designs the next experiment based on the last one. The seeking itself is the hard part: what to ask first, what next, and when you finally have enough.
Yepeng retweeted
(1/4) Excited to introduce ProCyon: a multimodal foundation model to model, generate, and predict protein phenotypes, led by stellar @oq_35 @YepHuang Robert Calef @valegiunca
👉 biorxiv.org/content/10.1101/…
ProCyon is an 11B parameter multimodal model that integrates protein sequences, structures, and natural language
ProCyon models, predicts, and generates phenotypes with zero-shot task transfer, breaking past the limits of fixed vocabularies or ontologies
🔬 The Challenge: Predicting protein structure is advanced, but predicting protein phenotypes—observable traits linking molecular function to biology—is still an open problem. 40% of human proteins lack context-specific insights, while 20% remain almost entirely uncharacterized
