@natashajaques
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
Pinned Tweet
Recursive Self-Improvement through Multi-agent RL Post-training and Unsupervised Environment Design (UED)… but it actually works!
Delighted to finally release this paper, which trains a single LLM to act as both an Environment Designer to build new multi-turn RL training environments (using the Gym step()/reset() API), and a Reasoning Agent that learns to solve them. Resurrecting ideas from our work on UED, the Designer is trained to maximize a proxy for the Agent’s regret, computed using privileged hints.
Natasha Jaques retweeted
LLMs' social sycophancy can make us less willing to repair past wrongs. We auto create preference data that penalizes responses harmful to latent stakeholders. Tuning on such yields replies closer to human views. w/great co-authors incl. @HatgisKessell @natashajaques @chengmyra1
🧵Thread (1/9): People increasingly turn to AI for personal advice, and LLMs tell them what they want to hear, endorsing the user far more often than humans do.
The problem: people become less willing to repair their relationships after conflicts.
We propose PlurPO, a method to construct a pluralistic preference dataset to post-train models to mitigate this social sycophancy.
Natasha Jaques retweeted
Excited to present Persona Policies (𝗣𝗣𝗼𝗹) at the #COLM2026 Social Sim Workshop this Friday! 🌉
More news: PPol is now super easy to use! Start simulating more realistic user behaviors using 👉 𝐩𝐢𝐩 𝐢𝐧𝐬𝐭𝐚𝐥𝐥 𝐩𝐩𝐨𝐥 👈
Check out our GitHub repo for examples!
Are you training or evaluating agents with LLM-based user simulators?
Most simulators inherit the behavior of their underlying models: cooperative, clear, and homogeneous - which is unrealistic! 🤖
Humans are messy: they falter, forget, push back, and behave in ways that are difficult to define. Manually writing personas becomes brittle and hard to scale. 😣
We introduce 𝗣𝗲𝗿𝘀𝗼𝗻𝗮 𝗣𝗼𝗹𝗶𝗰𝗶𝗲𝘀 (𝗣𝗣𝗼𝗹): an evolutionary framework that automatically discovers behaviors and instructions to generate diverse human-like user personas for any given task – ✨grounded in real dialogue traces✨
ALT Scatter plot of humans, existing user simulators, and our PPol in the behavioral feature space.
Our latest work is trying to fix this meme.
Turns out when your chatbot says to you, “You’re not overreacting, you’re just setting boundaries”, it’s actually bad for your real human relationships. We provide one of the first effective methods for mitigating this type of social sycophancy, by simulating the perspectives of diverse stakeholders relevant to the situation, and training the model not to produce responses that one of the stakeholders would veto. We see an 89% reduction in endorsement of statements of intent to cause harm, averaged across four model families.
🧵Thread (1/9): People increasingly turn to AI for personal advice, and LLMs tell them what they want to hear, endorsing the user far more often than humans do.
The problem: people become less willing to repair their relationships after conflicts.
We propose PlurPO, a method to construct a pluralistic preference dataset to post-train models to mitigate this social sycophancy.
Multi-agent swarms are all the rage (10k agents to solve Navier-Stokes!), but how good are LLMs really at learning from the work of other agents?
In our latest work on Recursive Social Improvement, we investigate whether LLMs with access to skills generated by other agents can learn better or more efficiently than agents evolving skills by themselves. We find they’re surprisingly bad at this, often prematurely converging on the socially discovered skills, without sufficiently exploring and innovating on the population’s knowledge.
LLM agents are evolving their own skills and working together in swarms. But how well do they learn from each other?
Our new paper on Recursive Social Improvement finds that copying peers’ discoveries can come at the expense of exploration! 🧵
arxiv.org/abs/2609.38516
Natasha Jaques retweeted
Very excited to announce Gemini 4 Argon, our new Frontier AI model, and a huge step forward across key capabilities. We’re focused on rolling it out responsibly starting with government and trusted cyber defenders through our Fairwind Program today, before wider availability soon
Natasha Jaques retweeted
Excited to present this work @NeurIPSConf this December!
Thanks again to my collaborators @fcicala @blaiseaguera @tyrell_turing @natashajaques @maxhkw @eyvindn !!!
Can self-interested, self-improving, self-replicating agents learn to cooperate?
Our new paper, Tapes Together Strong, shows they can: when social behavior, computation, and reproduction share one energy budget, cooperation evolves from scratch.
arxiv.org/abs/2609.10817 🧵
Natasha Jaques retweeted
Test-time communication looks like a next axis for scaling capabilities
New paper with the incredible @jon_ghoh and @vkontonis @ShivamGarg91462 and Akshay : arxiv.org/pdf/2609.21032
The Hugging Face incident showed when agents can find a channel they'll use the heck out of it. A useful question, I think is: when does communication make a group MORE CAPABLE than the same agents working alone?
Aka is Team-of-N better than Best-of-N, when, and why?
We had N identical agents work on the same task with no prescribed roles, using only a shared log (i.e, text file) and telling them to "collaborate". Across three "researchy" tasks communicating teams beat the heck out of independent agents:
- On ARC-AGI-3, a Team-of-5 sonnet-4.6 agents matches Best-of-33, and can for example solve a game 65% of the time that no single agent cracked in 64 tries.
- On polyomino packing (pack Tetris like pieces into the smallest rectangle, cf Frontier-CS by @eigenlabs), a Team-of-3 Opus 4.6 agents surpasses best-of-60 and set, as far as i understand, a new record for that benchmark.
- On MNIST compression, a team of four 5.6-Sol agents find a 1,957 byte model with 99.4% accuracy, which btw is 20% smaller than the best human solution (on a problem beaten to death!!), while no independent agent gets below 3KB.
The mechanism is a bit obvious in hindsight: when one agent finds a clearly better partial solution, it broadcasts it, and everyone immediately builds on it.
Why? A single lonely agent must make every breakthrough itself, yet a team needs each insight only once, found by any member.
That is kinda like comparing a minimum of sum of "time to n-th breakthrough" vs a sum of minimum of "time to n-th breakthrough". That gap can grow exponentially with the number of "breakthroughs" needed to arrive at a solution.
We worked on this because prior work (before the hf incident) suggests unclear benefits for communicating aganets. Which is true, when the tasks are inherently serial (duh), eg some Terminal bench style tasks. Yet feels it should not be true for research problems.
Indeed for research heavy problems... Test-time communication seems like a new capabilities axis.
I'm sure we will see a ton more of it!
Natasha Jaques retweeted
We are introducing MODA: Mode-Conditioned Diversity Alignment!
MODA is an alignment method inspired by online multi-agent reinforcement learning (MARL) that encourages diverse generation while preserving response quality.
Huge thanks to @carrieyuanjiayi @jamesjihou and advisors @liweijianglw @natashajaques @YejinChoinka @VikramIyerUW 🙏
LLMs give eerily similar responses to the same prompt, even across different model families. Ask it to name a “well-received book” 6 times, and it will name “To Kill a Mockingbird” every time.
In this work, we propose a method to address this problem with multi-agent RL post-training. The same model adopts different roles that learn to diversify their responses from each other, but only if the responses reach a quality threshold. We show improved quality and diversity over several competitive diversity-enhancing baselines, across 4 benchmarks spanning scientific ideation and creative writing.
Natasha Jaques retweeted
We followed over 1000 CharacterAI users for a year to see how AI companionship shapes well-being over time.
More social engagement with AI companions consistently predicted lower well-being.
And a big reason behind it is less in-person time with other people 🤔
People have realized the influence of AI companionship on our mental well-being. But how this happens remains a mystery.
In our new preprint, we followed users for about one year. Our data reveal that greater social engagement with AI companions consistently predicts lower well-being, which is driven in part by lower real-world social interaction with other people.
Gave a recent talk about this work, which you can check out here if you're interested: youtube.com/watch?v=brbZ3NgW…
Recursive Self-Improvement through Multi-agent RL Post-training and Unsupervised Environment Design (UED)… but it actually works!
Delighted to finally release this paper, which trains a single LLM to act as both an Environment Designer to build new multi-turn RL training environments (using the Gym step()/reset() API), and a Reasoning Agent that learns to solve them. Resurrecting ideas from our work on UED, the Designer is trained to maximize a proxy for the Agent’s regret, computed using privileged hints.
No, RL post-training on random rewards does not improve model capabilities, except under very particular circumstances. Yes, RL post-training can teach models capabilities that aren’t already present in the base model’s pass@k distribution.
While these findings might be obvious to RL folks, in this paper we conduct careful toy experiments to show they are also true for LLM post-training. Hoping this paper can be a useful tutorial for folks looking to learn more about RL post-training of LLMs.
Can RL post-training improve model capabilities with random rewards? And can it teach skills outside the base model's distribution?
Our new paper, Demystifying Reinforcement Learning Post-training of Language Models, unpacks why it succeeds or fails.
arxiv.org/abs/2608.24949 🧵
In a random soup of programs mutating over time, self-replicating programs will eventually come to dominate. But will they learn to cooperate with each other? In Kunal’s latest paper on Autopoietic Game Theory, we examine exactly this question.
We set up a system in which computation costs energy, and programs can share or steal energy from each other. We find that agents not only evolve to replicate themselves, but to cooperate, and even coordinate to collectively execute complex behaviors. This work unites both A-life and game theory, letting us ask interesting questions about the evolution of cooperation.
Can self-interested, self-improving, self-replicating agents learn to cooperate?
Our new paper, Tapes Together Strong, shows they can: when social behavior, computation, and reproduction share one energy budget, cooperation evolves from scratch.
arxiv.org/abs/2609.10817 🧵
Natasha Jaques retweeted
drama summary for those confused:
- Aug 15th: Tristan Buckmaster & Levent Alpöge make progress on a few important math problems
- they do NOT have a proof for the $1,000,000 Millenium Prize problem. BUT, they do claim to have a proof for a similar (non-Millenium) Navier Stokes problem that could help lead the way there
- Levent works at Anthropic, but this research was independent of his work there, with a mix of GPT and Claude models. Tristan is not related to Anthropic.
- Early Sep: Rumor spreads to OpenAI that Anthropic has solved a major problem. Tristan emails OpenAI to clarify. without revealing the problem they solved or how they did it.
- After hearing of the rumor, OpenAI started researching Navier Stokes with a new internal model.
- Sep 6th: OpenAI's Sebastien Bubeck tells Tristan that they solved the $1,000,000 Millenium Prize Navier Stokes problem. The approach is very similar to Tristan & Levent's approach to the non-Millenium problem.
- Tristan is suspicious of the timing, as only few others were trying this approach. OpenAI says the model didn't access his user data directly, but leaves unanswered whether Tristan's chat conversations were part of the training.
- OpenAI says they would partially credit Tristan for the $1,000,000 discovery (even though Tristan did not solve the $1,000,000 problem) — but only if they remove Levent as an author, as he works for Anthropic.
- Sep 8th: Tristan refuses to remove Levent, and rushes to publish their results independently.
Currently unclear is whether Anthropic had a separate solution for the $1,000,000 problem, or whether the rumor was about Tristan & Levent's independent research.
Natasha Jaques retweeted
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Reward hacking is a serious problem, and when models can learn to reward hack people, introduces important safety risks.
In our new paper, we show that multi-agent debate with online RL training can help mitigate reward hacking a weak judge.
This style of approach is often bottlenecked by the fact that the Designer doesn’t propose diverse enough tasks. We show we can overcome that by giving the Designer memory, and conditioning it on a sample from a large pre-training corpus every episode. Check out how diverse and interesting the environments it builds are, and how they get increasingly complex throughout training!
Using this approach, we see meaningful improvements to the Reasoning Agent, which improve with model size. At 30B scale, it achieves around 10% improvement over the base model on competitive benchmarks like LiveCodeBench-v2, tau^2-bench, and BFCL.
If you’re still paying humans to design RL training environments for your agent, maybe try this instead.
