@DannyDriess

Research Scientist @physical_int. Formerly Google DeepMind

dannydriess.github.io
Joined August 2021
  • Tweets170
  • Following336
  • Followers4.2K
50Media
How to build vision-language-action models that train fast, run fast & generalize? In our new paper, we formalize & analyze the approach of our π-0.5 model & further improve it with a single stage recipe. Blog: pi.website/research/knowledg… Paper: pi.website/download/pi05_KI.…
6
27
3
222
20,167
Anyone up for doing a CoRL emergency review? DM me
3
2
2,214
The most exciting aspect of modern machine learning, in my opinion, is that one can train models that just work for many tasks, without finetuning. π0.7 is a major step in that direction for robots
Physical Intelligence
@physical_int
Apr 16
Our newest model, π0.7, has some interesting emergent capabilities: it can control a new robot to fold shirts for which we had no shirt folding data, figure out how to use an appliance with language-based coaching, and perform a wide range of dexterous tasks all in one model!
3
2
43
8,488
more replies
and we see strong cross embodiment generalization for dexterous tasks
2
239
π0.7's training recipe builds upon Knowledge Insulation nitter.cf/DannyDriess/status/192… and our recent memory work nitter.cf/DannyDriess/status/202…
Danny Driess
@DannyDriess
Mar 3
Many real-world tasks require memory to be successful. Yet, most robots don’t have any form of memory. Today, we are going to change that. We developed a system called MEM that introduces memory into VLAs on multiple scales
1
273
We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.
39
292
80
2,192
444,405
Danny Driess retweeted
Kyle Vedder
@KyleVedder
Mar 3
this robustness allows the policy to do diverse, long horizon tasks in unseen environments for example, the demo kitchen was built *after* the potatoes policy was fully trained — I just wrote the high level prompt to tell it where to go look for items and it did the rest
1
2
34
2,812
Danny Driess retweeted
Marcel Torné@marceltornev
Mar 3
We equipped PI policies with memory! And taught our robots to do long-horizon real world tasks such as preparing the items for a recipe, cooking a grilled cheese and cleaning the kitchen!
Physical Intelligence
@physical_int
Mar 3
We’ve developed a memory system for our models that provides both short-term visual memory and long-term semantic memory. Our approach allows us to train robots to perform long and complex tasks, like cleaning up a kitchen or preparing a grilled cheese sandwich from scratch 👇
8
15
2
89
24,083
Danny Driess retweeted
Karl Pertsch
@KarlPertsch
Mar 3
This one has been a long time coming: today we’re introducing MEM, an approach for giving VLAs short-term and long-term memory. Memory is such an obvious capability, but adding it isn’t easy (most VLAs today are memory-less). A short thread on challenges, solutions, and the new capabilities MEM unlocks for us.
8
11
4
115
10,097
If you look at this plot here, you can see that both short- and long-term memory were important to make long-horizon tasks work well
Physical Intelligence
@physical_int
Mar 3
We’ve developed a memory system for our models that provides both short-term visual memory and long-term semantic memory. Our approach allows us to train robots to perform long and complex tasks, like cleaning up a kitchen or preparing a grilled cheese sandwich from scratch 👇
2
15
1,076
Many real-world tasks require memory to be successful. Yet, most robots don’t have any form of memory. Today, we are going to change that. We developed a system called MEM that introduces memory into VLAs on multiple scales
Physical Intelligence
@physical_int
Mar 3
We’ve developed a memory system for our models that provides both short-term visual memory and long-term semantic memory. Our approach allows us to train robots to perform long and complex tasks, like cleaning up a kitchen or preparing a grilled cheese sandwich from scratch 👇
5
12
1
64
5,754
The key idea behind Multi-Scale Embodied Memory (MEM): use different modalities to represent memory at different time scales. 📹 For short horizon memory, we developed an efficient video encoder that lets the model remember fine-grained details about its recent interactions. 📜 For long horizon memory, we train the model to summarize events in text, allowing it to remember events for up to 15 min.
1
3
485
One aspect I am particularly excited about is that memory enables the model to adapt its strategy while solving the task, something we can coin “in-context adaptation”. In this example, it is unclear from a single image whether the fridge opens from the left or the right. Hence, a model without memory (left) might fail to open the fridge repeatedly. In contrast, with memory (right), our model learns “in-context” that the fridge opens differently, and adjusts its strategy accordingly.
3
364
General-purpose AI models are behind some of the most exciting applications we now can't live without. We envision that an analogous “physical intelligence layer” built with models like π0.6 will similarly spur a new wave of applications for the physical world. We’ve recently begun working with a handful of companies that have deployed their robots to do real-world, useful things. pi.website/blog/partner/?v=1

The Physical Intelligence Layer

General-purpose physical intelligence models will enable a Cambrian explosion of robotics applications. See how our partners are already solving real-world problems.

pi.website
9
88
30
740
191,540
Check out our latest work on steerable policies. Instead of having only language as the interface to a VLA, steerable policies follow point queries, motion traces, atomic subtasks and more, which allows us to make better use of VLMs controlling them. More in @verityw_'s thread
Will Chen@verityw_
Feb 16
How can robot policies be trained to best leverage VLMs' CoT reasoning and in-context learning for generalization? The key is Steerable Policies: vision-language-action models that can be flexibly controlled in many ways! steerable-policies.github.io 1/9
1
1
9
752
What I like about this: If I want to explain someone how to solve a task, I rarely use language alone, I might point at things, wave in the air, without restricting myself to only one interface to communicate my intent. This work brings this idea into VLAs.
1
1
269
Project led by @verityw_ with @JagdeepBhatia8, @CatGlossop, Nikhil Mathihalli, @riadoshi21, @tangerinecoder, @KarlPertsch, @svlevine
1
263
The base model powering π*0.6 is trained with Knowledge Insulation nitter.cf/DannyDriess/status/192…
Physical Intelligence
@physical_int
18 Nov 2025
Our model can now learn from its own experience with RL! Our new π*0.6 model can more than double throughput over a base model trained without RL, and can perform real-world tasks: making espresso drinks, folding diverse laundry, and assembling boxes. More in the thread below.
1
11
1,235
The idea behind significantly improving the performance on hard real-world tasks is to train a value function, condition the model on advantages computed from the value function, and running an iterative improvement loop where the model learns from it’s own data.
1
5
350
Load more