Research Scientist @Google | PhD from @unccs advised by @mohitban47 | @Google PhD Fellow | Prev intern: @Google @AmazonScience, @MetaAI, @SFResearch
Joined February 2019
- Tweets326
- Following489
- Followers711
- Likes265
David Wan retweeted
Looking forward to giving a keynote at the Midwest Machine Learning Symposium (MMLS) 2026 (being held at Purdue University this year) & meeting folks from all the strong universities in the midwest, with their inspiring, long tradition of these exciting symposiums! 🙂
👇👇
The Midwest Machine Learning Symposium (MMLS) 2026 will happen at Purdue University!
📍 West Lafayette, IN
📅 June 24–25, 2026
🔗 midwest-ml.org/2026/
📌 Poster submission deadline: May 24
We have an amazing lineup of plenary speakers: Tong Zhang, Jennifer Neville @ProfJenNeville, Mohit Bansal @mohitban47, Joyce Chai.
Looking forward to seeing you there!
@PurdueCS @PurdueECE @PurdueStats
🥳 Excited to share that MuRGAt is accepted to #ICML2026!
Even strong MLLMs hallucinate citations to multimodal sources (video, audio, charts). Our new Fact-Level Multimodal Attribution benchmark tackles this by:
🕐 Requiring fine-grained temporal & per-modality citations (vs. just source-level)
🔍 Distinguishing verifiable claims from reasoning steps to evaluate multi-step responses
We also introduce MuRGAt-SCORE, a reference-free, decomposed metric aligned with human judgment, and show that Programmatic Grounding substantially boosts attribution!
👇
🚀Announcing MuRGAt!
MLLMs are improving at reasoning over complex multimodal inputs, but does that translate to faithful grounding to multimodal sources (video, audio, charts, etc.)? We find that even strong MLLMs often hallucinate citations despite getting the answer correct!🤯
We introduce a benchmark for Fact-Level Multimodal Attribution featuring:
✅ High-quality Human Annotations for validation.
✅ MuRGAt-SCORE: A decomposed metric that highly correlates with human judgment.
✅ Methods to improve citations, showing that Programmatic Grounding boosts attribution.
🧵👇
📄 Updated Paper: arxiv.org/abs/2602.11509
🔗 Code: github.com/meetdavidwan/murg…
🤗 HF Page: huggingface.co/papers/2602.1…
Great collaboration with @HanWang98 @ZiyangW00 @EliasEskin @hyunji_amy_lee and @mohitban47! @unccs @unc_ai_group
David Wan retweeted
🎉 Excited to share that our work on intrinsic dimensionality of reasoning has been accepted to #ICML2026 as a ✨spotlight✨ (top 2.2%)!
We analyze the effectiveness of teaching a model how to reason via the lens of intrinsic dimensionality (the minimum effective capacity a model needs to solve the task) and find that effective reasoning chains are inherently compressive!
Across Gemma-3 1B and 4B, lower intrinsic dimensionality strongly predicts not only in-distribution accuracy (GSM8K), but also robustness on OOD benchmarks (GSM-Hard, GSM-Symbolic, GSM-IC) -- outperforming reasoning length, token perplexity, and KL divergence.
Stay tuned for more results and exciting updates in the camera-ready! 🚀
🚨Excited to share our new work viewing reasoning strategies as teaching tools: for fixed target model, which CoT strategies best support learning and generalization?
✨Our answer is intrinsic dimensionality (minimum effective capacity a model needs to solve the task).
Somewhat counterintuitively, adding CoT – which requires generating longer and more structured outputs – can reduce learning complexity. Good reasoning compresses the task, i.e., it reduces the degrees of freedom the model needs to map inputs to correct solutions.
🧵⬇️ (1/5)
David Wan retweeted
🎉 Happy to share our paper introducing GCMs for confidence estimation based on historical predictions has been accepted to #ICML2026!
We find that models are no better at learning to predict their own correctness than others', i.e., they don't have privileged self-access (given training).
Training smaller models to predict the correctness of many models generalizes and leads to better calibration than self-reported confidence from much larger models!
Check out 🧵 for more
🚨 Announcing Generalized Correctness Models (GCMs) 🚨Finding that LLMs have little self knowledge about their own correctness, we train an 8B GCM to predict correctness of many models, which is more accurate than training model-specific CMs, and outperforms a larger Llama-3-70B’s self-emitted confidences in downstream selective prediction tasks.
We motivate GCMs and analyze them by answering 2 questions:
❓ RQ1: Are LLMs better than other LLMs at predicting their own correctness? We find that they are not, instead historical information (past LLM outputs and their correctness) drives performance, motivating cross-model transfer and training of GCMs!
❓ RQ2: How can we use historical information from multiple models for correctness prediction? Within RQ2, we explore 3 further subquestions, informing the design of GCMs:
1⃣ How does confidence prediction generalize across models? GCMs transfers strategies across models and datasets, even beating models trained directly on OOD datasets.
2⃣ What information should GCMs condition on? The exact way an LLM phrases an answer is a strong predictor for correctness + strategies leveraging world-knowledge seem to drive generalization.
3⃣ How do alternative methods for encoding history (e.g. post hoc calibration, ICL) compare? Including historical information ICL can aid larger models to predict correctness but underperforms GCMs, and post hoc calibration can complement GCMs to reduce calibration error.
🧵👇
David Wan retweeted
Glad that GCMs for analyzing confidence estimation from historical predictions was accepted to #ICML2026!
We examine whether models have an advantage when predicting their own correctness and confidence and find that little usable privileged information exist for confidence prediction. This leads us to train Generalized Correctness Models to predict the calibrated confidence and correctness of models, outperforming the logit and verbalized confidences of much larger models! Thanks to @vaidehi_patil_, @hyunji_amy_lee, @EliasEskin, and @mohitban47!
See more in 🧵below!
🚨 Announcing Generalized Correctness Models (GCMs) 🚨Finding that LLMs have little self knowledge about their own correctness, we train an 8B GCM to predict correctness of many models, which is more accurate than training model-specific CMs, and outperforms a larger Llama-3-70B’s self-emitted confidences in downstream selective prediction tasks.
We motivate GCMs and analyze them by answering 2 questions:
❓ RQ1: Are LLMs better than other LLMs at predicting their own correctness? We find that they are not, instead historical information (past LLM outputs and their correctness) drives performance, motivating cross-model transfer and training of GCMs!
❓ RQ2: How can we use historical information from multiple models for correctness prediction? Within RQ2, we explore 3 further subquestions, informing the design of GCMs:
1⃣ How does confidence prediction generalize across models? GCMs transfers strategies across models and datasets, even beating models trained directly on OOD datasets.
2⃣ What information should GCMs condition on? The exact way an LLM phrases an answer is a strong predictor for correctness + strategies leveraging world-knowledge seem to drive generalization.
3⃣ How do alternative methods for encoding history (e.g. post hoc calibration, ICL) compare? Including historical information ICL can aid larger models to predict correctness but underperforms GCMs, and post hoc calibration can complement GCMs to reduce calibration error.
🧵👇
David Wan retweeted
🎉 Excited to share EPiC is accepted to #ICML2026!
We show that learning precise camera control for video diffusion doesn't need expensive 3D supervision or large-scale data. No camera or point cloud processing — just mask source videos based on visibility to construct precise training anchor videos, and learn a SoTA camera controller with only 30M params, trained >100× faster on >100× less data than prior work, while generalizing across both I2V and V2V camera control tasks.
🚨Thrilled to introduce EPiC🎥: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance
A generative model enables precise 3D camera trajectory control over user-provided videos or images. It achieves highly efficient training, completing within just 16 GPU-hours (2 hours on 8×H100 GPUs), significantly faster compared to baselines that typically require at least 200 GPU-hours, while achieving better performance.
Thread 🧵👇(1/8)
David Wan retweeted
Excited to share that Symbolic-MoE has been accepted to #ICML2026! 🎉
We find that adaptive instance-level "mixture-of-experts" yields +8.15% average gain over the best multi-agent baseline, while almost 2x faster + generalize to unseen tasks!
Existing multi-LLM/multi-agent setups can improve reasoning, but they use a fixed set of LLMs, making them sensitive to the choice of model + hard to scale due to the high cost of multi-round agent discussion.
Symbolic MoE instead adaptively recruits the most relevant expert for each instance based on the skills needed and the strengths of each model. In addition to beating baselines in performance, Symbolic-MoE is also more efficient: it skips the expensive multi-round discussion, and our novel batching method allows us to integrate 16 experts on a single GPU!
🧵👇
🚨 We introduce ✨ Symbolic-MoE ✨ which uses skill-based instance-level recruiting to dynamically combine LLMs, allowing three 7-8B LLMs to beat GPT4o-mini and Llama3.3 70B across challenging + diverse reasoning tasks (MMLU-Pro, AIME, GPQA, MedMCQA) while running on 1 GPU!
Key highlights:
1️⃣ Instance-level adaptive recruiting: experts chosen for each question based on skills.
2️⃣ +8.15% average gain over the best multi-agent baseline.
3️⃣ Runs almost 2x faster than SOTA multi-agent discussion.
Existing multi-LLM/multi-agent setups can improve reasoning, but current methods are 1️⃣ too coarse-grained, using task performance to select LLMs 2️⃣ too rigid, using a fixed set of LLMs, making them sensitive to the choice of model 3️⃣ hard to scale due to high cost of multi-round agent discussion.
Symbolic MoE instead adaptively recruits the most relevant expert for each instance based on the skills needed and the strengths of each model. In addition to beating baselines in performance, Symbolic-MoE is more efficient: our novel batching method allows us to integrate 16 experts on a single GPU.
🧵👇
David Wan retweeted
🚨 Excited to announce SAS: Stabilizing Efficient Reasoning with Step-Level Advantage Selection is accepted to #ACL2026 Findings.
LLMs can compress reasoning during post-training with short context alone, but silently destabilize training. We propose SAS to fix it by zeroing advantages at the step level: suppressing low-confidence steps in correct rollouts + shielding high-confidence ones in failed rollouts.
✨ Result: >30% shorter reasoning traces, higher accuracy, and stable training – all from a simple zero-advantage operation at the step level.
David Wan retweeted
Our #ICLR2026 paper on inferring an executable, symbolic world model offline from one life in a hostile environment (One Life to Learn) will be presented by @EliasEskin (I couldn't make it to #ICLR2026)!
If you're interested in world modeling, open-ended exploration, or neurosymbolic methods, DM / email me or stop by and chat with Elias!
📅 Sat, Apr 25, 10:30AM — 1PM BRT
📍Pavilion 4 P4-#4916
onelife-worldmodel.github.io…
How can an agent reverse engineer the underlying laws of an unknown, hostile & stochastic environment in “one life”, without millions of steps + human-provided goals / rewards?
In our work, we:
1️⃣ infer an executable symbolic world model (a probabilistic program capturing environment dynamics) offline from one life (1 episode), with dynamic graph routing for credit assignment.
2️⃣ develop Crafter-OO, our reimplementation of the Crafter environment that exposes a structured, object-oriented symbolic state and a pure transition function that operates on that state.
3️⃣ implement 20+ executable scenarios to test knowledge of core mechanics + mutators that generate illegal distractor states to probe world model understanding.
4️⃣ introduce an evaluation protocol that measures the ability to distinguish plausible future states from implausible ones + the ability to generate future states that closely resemble reality.
5️⃣ show that the inferred world model can be used as a simulator for planning.
Thread 🧵👇
David Wan retweeted
Honored to receive the @NSF Graduate Research Fellowship! 🎉 A huge thank you to my mentors @mohitban47 and @EliasEskin for guiding me on my research journey, and to the brilliant MURGe Lab graduate students and postdocs who have supported me along the way during my time in @unccs.
My prior work has studied enhancing multi-agent reasoning through more advanced tool recruitment (DART) and also increasing model faithfulness (REMuL). I’ve been fortunate to apply these multi-agent ideas into impactful applications across education and healthcare. I hope to continue working on improving multimodal understanding and collaborative AI agent systems in the future!
Today, NSF announced it has made offers & will award 2,500 Graduate Research Fellowships for the 2026-2027 academic year to outstanding graduate students across the U.S. pursuing research-based degrees in science, technology, engineering and mathematics. bit.ly/4cpAHBO
David Wan retweeted
I won’t be attending #ICLR2026, but @LucasPCaccia and @EliasEskin will be presenting our work, Gistify!
We study whether coding agents can truly understand a repository by extracting its gist: generating a single, self-contained, executable file that reproduces the behavior of a target command (e.g., a test or entrypoint). It is lightweight, broadly applicable evaluation of codebase-level reasoning!
I’d also love to connect online. Feel free to reach out!
📅 3:15 PM-5:45 PM Thu, Apr 23, 2026
📍 Pavilion 3, Poster 1020
🚨 Excited to announce Gistify!, where a coding agent must extract the gist of a repository: generate a single, executable, and self-contained file that faithfully reproduces the behavior of a given command (e.g., a test or entrypoint).
✅ It is a lightweight, broadly applicable evaluation that tests whether models can reason at the codebase level
😯 Even strong LLMs/frameworks struggle, especially on long, multi-file traces!
David Wan retweeted
Not able to attend #ICLR2026 in person, but would love to discuss NuRL (Nudging the Boundaries of LLM Reasoning) online!
If you're interested in "nudging" the upper bound of LLM reasoning using self-generated hints, feel free to ask any questions via the ICLR online chat / send me emails!
📅 Thu, Apr 23, 3:15 PM – 5:45 PM
📍 Pavilion 4 P4-#4604
📰 arxiv.org/abs/2509.25666
🚨 NuRL: Nudging the Boundaries of LLM Reasoning
GRPO improves LLM reasoning, but often within the model's "comfort zone": hard samples (w/ 0% pass rate) remain unsolvable and contribute zero learning signals. In NuRL, we show that "nudging" the LLM with self-generated hints effectively expands the model's learning zone 👉consistent gains in pass@1 on 6 benchmarks w/ 3 models & raises pass@1024 on challenging tasks!
Key takeaways:
1⃣GRPO can't learn from problems the model never solves correctly, but NuRL uses self-generated "hints" to make hard problems learnable
2⃣Abstract, high-level hints work best—revealing too much about the answer can actually hurt performance!
3⃣NuRL improves performance across 6 benchmarks and 3 models (+0.8-1.8% over GRPO), while using fewer rollouts during training
4⃣NuRL works with self-generated hints (no external model needed) and shows larger gains when combined with test-time scaling
5⃣NuRL raises the upper limit: it boosts pass@1024 up to +7.6% on challenging datasets (e.g., GPQA, Date Understanding)
🧵
David Wan retweeted
Honored to receive the @NSF Graduate Research Fellowship (GRFP) 🎉. I’m deeply grateful to have conducted research @unccs in the MURGe-Lab under the guidance of @mohitban47 and @EliasEskin, and for the support of the many PhD students and Postdocs who shaped my experience at UNC and helped me grow through my projects.
My previous work leveraged a scientific understanding / interpretability perspective to compress LLMs via quantization (Task-Circuit Quantization) and also to predict their correctness (General Correctness Models). More recently, I used RL in long horizon tasks to study ToM (AI Double Agents). These works have inspired me to focus on developing a better scientific understanding of- and improved memory for- foundation models in the future!
Today, NSF announced it has made offers & will award 2,500 Graduate Research Fellowships for the 2026-2027 academic year to outstanding graduate students across the U.S. pursuing research-based degrees in science, technology, engineering and mathematics. bit.ly/4cpAHBO
David Wan retweeted
Had a great time visiting the University of Virginia last week for their CS Distinguished Speaker Series -- talked about skill discovery + improvement as well as some newer work in mixture-of-agents' skill profiling + matching + disagreement resolution, in context of building trustworthy (calibrated, controllable, collaborative) multimodal AI agents (incl. education and healthcare applications)!
Thanks again for the kind invitation and exciting discussions Yangfeng, Haiying, Aidong, Tom, Yen-Ling, Jundong, Rohan, et al. (and for the nice photos) 🙏
PS. Also found many similarities between UVA and UNC campuses w.r.t. the beautiful hilly terrain and drives, the big central quad/lawn, the traditional brick architecture (and the amazing farmers markets in both places) 🙂
David Wan retweeted
Thanks @_akhaliq for sharing our work!
If you are interested in multimodal agents on open-web search, please see our thread for more details: xitter.cf/HanWang98/status/20448…
MERRIN
A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments
paper: huggingface.co/papers/2604.1…
David Wan retweeted
🚨 Check out MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments --> diagnoses that even strong search agents e.g. Gemini 3/3.1, GPT-5.4 struggle (avg 22%, best 40%) -- and compared to humans, they check 50x more irrelevant pages, over-explore with repeated searches, and heavily favor text modality (87%)!
✅ Covers text, image, video, and audio
✅ Reflects real web noise and conflicts (not synthetic)
✅ Queries have NO explicit references to specific modality sources
✅ Multi-hop reasoning required
✅ Human-annotated with open-web search
🧵👇
🚨 Excited to announce MERRIN: a human-annotated, expert-vetted benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments! 🌐🔍
The web is noisy, conflicting, and multimodal. Even strong search agents fail when questions require reasoning across text, images, video & audio — MERRIN is built to diagnose exactly where they break down, via multi-hop reasoning on open-web search without explicit references to specific modality sources.
David Wan retweeted
Real-world web search 🔎 requires handling messy, multimodal information. It turns out current models – even ones that do well on text-based search – struggle on the complex, expert-vetted, multi-hop queries we introduce in MERRIN!
Agent search behavior differs substantially from humans: even the strongest agents perform poorly and search inefficiently, visiting 50x more sites than human searchers. MERRIN represents a new front for both search agent accuracy and efficiency, with plenty of room for improvement!
🧵👇
🚨 Excited to announce MERRIN: a human-annotated, expert-vetted benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments! 🌐🔍
The web is noisy, conflicting, and multimodal. Even strong search agents fail when questions require reasoning across text, images, video & audio — MERRIN is built to diagnose exactly where they break down, via multi-hop reasoning on open-web search without explicit references to specific modality sources.
David Wan retweeted
The web is full of noisy, multimodal information, and many user questions require multi-hop reasoning across various modalities without explicit modality cues.
Can models choose the right modalities and follow the correct reasoning path? To study this, we introduce MERRIN.
We find that even strong search agents struggle (avg 22.3%, best 40.1%). Compared to humans, they check ~50× more irrelevant pages, over-explore with repeated searches, and heavily favor text modality (87%).
This highlights the need for search agents that can reason efficiently and use the right modalities in a noisy web environment and MERRIN is a strong benchmark to evaluate them!
More details in the thread below ⬇️
🚨 Excited to announce MERRIN: a human-annotated, expert-vetted benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments! 🌐🔍
The web is noisy, conflicting, and multimodal. Even strong search agents fail when questions require reasoning across text, images, video & audio — MERRIN is built to diagnose exactly where they break down, via multi-hop reasoning on open-web search without explicit references to specific modality sources.
David Wan retweeted
MERRIN
A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments
paper: huggingface.co/papers/2604.1…