NLP researcher. Assistant Professor at NYU CS & CDS.
Joined December 2016
- Tweets168
- Following424
- Followers8.6K
- Likes333
He He retweeted
Honored to receive the 2nd Annual Google ML and Systems Junior Faculty Award for our work on adaptable ML systems and agents. Grateful to @Google for the support, and to my students. More to come on adaptable agents, in open source and open science.
He He retweeted
i was diagnosed with thyroid cancer many years ago purely by chance, because i had to get blood work for my US green card application (thank you, USCIS!). i was in my mid-30’s and was reasonably healthy (or so i believed.) this was shocking and totally unpredictable news to me. thanks to the amazing team of clinicians at NYU Langone, i successfully went through a total thyroidectomy followed by radioactive iodine therapy. but, it was truly the most uncertain and confusing period of my life.
this is when i first realized that no one can opt out of healthcare. being sick isn’t what you choose to or choose not to. it is something that unpredictably happens to all of us at some point for one reason or another. this is why it is a healthcare system that we collectively build and use to support each other at the society level. because it must support everyone, the complexity of creating, maintaining and improving this system is a Herculean effort that calls for all the help in the world.
today, i am announcing @OrtetAI together with my dear co-founders, @keunwoochoi, @HenriDwyer, @elmanmansimov, @causalclaudia and Jeff, to lend our support to improving the health of each and every one of us. Ortet’s goal is to create a technological foundation that will enable health systems to deliver predictably better care to each and every patient. this will only be possible by working closely with all stakeholders in the broader ecosystem, which is enabled in part by Ortet’s close partnership with Thoreau.
Ortet is taking the very first step of a long journey ahead, and at the end of this journey, i strongly believe we would be able to say that Ortet had contributed to building better, more predictable healthcare and thus to improving health of each and every one of us.
see ortet.ai/ for more!
He He retweeted
Now an Oral at @NeurIPSConf E&D track 2026! Super excited that work which studies how these models affects users is valued by the #NeurIPS2026 community in the expanded evaluations track 🎉🎉🇦🇺
People are increasingly worried that AI tools make us overreliant.
But how do we actually measure this? We introduce Offloading Score, a measure of reliance based on the fraction of cognitive effort offloaded to AI while completing a task.
In a controlled user study, Offloading Score detects increased reliance under time pressure, while several common alternatives do not.
(1/9)
He He retweeted
Paper thread! I recently read this paper from some FAIR colleagues and NYU folks in depth, so making a thread while it's fresh in my head.
The main point is seeing how far on the SMALL side we can go in scaling laws without losing fit or predictability, and what it takes?
He He retweeted
I am thrilled to announce that @beaconholdings has acquired @haizelabs, with me joining as VP of AI Research.
We started Haize in 2024 to enable anyone to build reliable and safe AI. Through our red-teaming and safeguards work with the frontier labs; our observability, guardrail, and evaluation platform serving the world’s largest enterprises; and our pro bono SMB work, any customer could build AI they trusted with Haize.
Joining Beacon lets us deliver the same trustworthy AI to those who need it most, and those most overlooked by Silicon Valley: the essential Main Street businesses the real world depends upon.
Transitioning these essential businesses through the AI revolution is one of the most consequential responsible AI problems of our time. We couldn’t be more honored to tackle it with Beacon.
Thank you to our customers, investors, and team for the journey of a lifetime. And thank you to Nilam, Goutham, Mark, and the Beacon team for the trust and opportunity.
It’s time to get to work.
He He retweeted
Introducing ApprenticeBench: computer use + continual learning on a real job.
We show Fable 5.1 and GPT-6 Astra can now continually learn on a job and surpass human professionals. A decisive step change in AI's job readiness.
No FDEs. Agents deploy themselves into the job. 🧵
He He retweeted
Enterprise AI is in a wildly paradoxical state 🤔, and we’re reaching the inflection point that will resolve it. 💥
Enterprises want to differentiate with AI, yet rent the same intelligence as their competitors.
Their workflows and expertise are highly specialized, yet they rely on generic models built to be good at everything.
They worry about AI costs, yet pay premium prices for massive models where only a fraction (1%) of the intelligence is relevant to their task. 💸
And they demand control and sovereignty, yet rent the intelligence becoming core to their business.
This is not a sustainable equilibrium.
The next era of enterprise AI is specialized intelligence companies build, own, and compound. And we are at the inflection point of this transition.
That’s the bet we made when we started @oumi_ai two years ago.
Today we’re closing the loop: Oumi can now not only automatically build your specialized AI models, but also deploy them into production, learn from their production experience, and continuously improve them.
The intelligence that your business runs on, becomes your differentiator. Your compounding advantage.
The winners of the next AI era will turn their own data, expertise, and experience into specialized intelligence that nobody else can rent.
Don’t rent your AI. Build it. Own it. Compound it.
He He retweeted
New paper: Understanding Reasoning from Pretraining to Post-Training!
We study the full LLM training pipeline from pretraining to post-training, find a joint scaling law, figure out how the compute should be allocated, and study what RL is doing to the policy.
🧵
He He retweeted
Introducing EdgeBench, a benchmark designed to study how agents learn from environments over at least 12~72-hour runs. We find that performance follows a log-sigmoid function of environment interaction time with high precision.
EdgeBench is built with three ingredients:
- 🌍 Real & Diverse: 134 real-world tasks across 6 task categories, spanning scientific problems, professional knowledge work, software engineering, optimization, formal math, and games.
- ⏳ Ultra-Long-Horizon: Each task supports 12–72 hours of agent work. Recorded human effort averages 57.2 hours.
- 🔁 Informative Feedback: Agents receive real-world feedback for continuous improvement.
After 38,000 hours of agent runs on EdgeBench, a scaling law for learning from environments emerges:
- 📈 As agents interact with task environments over time, their aggregate performance is precisely fit by a log-sigmoid function.
- 🧠 This phenomenon can be explained by an elegant theory of graph exploration.
We are releasing an initial 51 of the 134 tasks, together with the full evaluation framework, to help advance long-horizon agent research. Check our blog & paper for more findings!
Blog edge-bench.org/
Paper edge-bench.org/paper.pdf
GitHub github.com/ByteDance-Seed/Ed…
Dataset huggingface.co/datasets/Byte…
Details below 👇🧵
He He retweeted
As my time in Pittsburgh comes to an end...
I'm excited to share that I will be joining UIUC as an Assistant Professor in fall 2027! I will recruit PhD students in both ECE and CS.
I’m looking for students and postdocs who are excited to continue building towards collaborative AI systems that learn and adapt through interaction.
He He retweeted
Image editing models can put you on the Moon, but can they precisely move a circle right by 50 pixels? 📐
Introducing 🎨PaintBench: a foundational eval of visual editing operations with only one right answer.
The highest-performing model (@NanoBanana 2) reaches only 17.1%.
He He retweeted
People are increasingly worried that AI tools make us overreliant.
But how do we actually measure this? We introduce Offloading Score, a measure of reliance based on the fraction of cognitive effort offloaded to AI while completing a task.
In a controlled user study, Offloading Score detects increased reliance under time pressure, while several common alternatives do not.
(1/9)
He He retweeted
Most assume unlearnable examples never get positive reward. They do.
In our ICML paper, We reveal that a hard problem can receive positive reward during RLVR but remain unlearned.
We show the phenomenon is more likely a representation issue rather than RL optimization artifact.
Dimitris' experiments have inspired many including me. Excited for this talk!
Giving my first agent talk at MIT's NLP seminar (virtually) next Wednesday. First time I'm focusing less on results and more on the process that got me there. Feels strange and a little like growing up.
Abstract:
Over the past few months, agents have changed the way I do research by collapsing the distance between a question and an experiment. Ideas that used to die under the technical burden of setting up the experiment are now testable in days, often by one person, an agent, and a laptop. In this talk I'll walk through three vignettes, side projects I ran since February: training the smallest transformer that can do 10-digit addition, predicting LLM benchmarks without running them, and building the first trained computer that is a transformer. These projects look unrelated, and that is part of the point. I'll share what I learned from each — both about the problem and about agents — and why taste and the ability to verify become more important when execution becomes cheap. I'll close on a question I don't have a good answer to: we've always trained researchers through technical work, and taste and verification emerge from it. What happens when most of it becomes automated?
He He retweeted
I'll be at ICLR this week to present our poster on factors affecting the novelty of LLM output on Fri afternoon. Come talk to me about this or anything else in the future-of-work/societal impacts space! Also hmu if you want to check out the Maracana stadium when we're there!😎🇧🇷
Our work on LLM novelty as the frontier of original and high-quality output was accepted to #ICLR26! Come talk to us about how model scale, SFT, and RL affect this trade-off! See you in Brazil!🇧🇷h/t to my awesome collaborators @hhexiy @valeriechen_ @JanePan_ @jcyhc_ai
He He retweeted
Love seeing this.
There are two flavors of automated AI research.
One that just cares about hill climbing on a target benchmark (e.g., autoresearch).
The other that actually care about the idea (e.g., this post). Beyond just looking at the performance, we want to ask: does the idea offer some new insights? Is it simple and scalable? Is it generalizable? The evaluation is a lot harder for this case, because it requires the human judge to have a good research taste in the first place, but it’s also gonna be much more fun :)
He He retweeted
An English mirror site of Scientific Spaces: rohin-garg.github.io/kexue-e…
Created by @GargRohin3301 . Thanks a lot!