PhD @Stanford @KnightHennessy. AI, robotics, and conservation
Los Angeles, CA
Joined December 2019
- Tweets75
- Following413
- Followers411
- Likes263
Pinned Tweet
Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?
Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)
With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T
We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇
Jaden Clark retweeted
Standard diffusion policies struggle when real world conditions change sensor rates or introduce latency delays.
Changhao Wang, Jaden Clark, and Shuran Song just released Chronos Policy to solve this by embedding physical timestamps directly into policy learning.
Ground breaking highlights:
Unifies sensor sampling, inference latency, and execution delays into a single physical time formulation
Encodes physical timestamps as positional encodings in attention layers using time aware RoPE
Outperforms standard Diffusion Policy when observation timing or hardware inference latency shifts unexpectedly
Uses temporal randomization during training to extract far more training signal from high rate data recordings
Introducing Chronos Policy ⏳ giving policies explicit timestep-conditioning makes them more robust and improves performance on dynamic tasks!
While other works (such as latency matching, asynchronous chunking, etc) assume action continuity or fixed delays, Chronos Policy unifies observation and action rates into a single temporal formulation.
A super simple change that everyone should add to their policy!
Most robot policies treat time implicitly, using sequence position as a proxy for physical time. But real robot systems rarely run on a fixed clock. Sensors run at different rates, observations arrive with delays, and actions take time to compute and execute. Successful control depends not only on what the robot observes but also on the timing of observations and actions.
We introduce 🕒 Chronos Policy, a diffusion policy that explicitly conditions on observation capture times and requested action execution times. We use time-aware positional encodings to make attention depend on elapsed physical time. Because timestamps let the policy interpret each sequence's timing, we can randomize sensing and action timing during training without making those variations ambiguous. This exposes the policy to a wider range of timing conditions, improving robustness to timing changes at deployment.
Jaden Clark retweeted
Most robot policies treat time implicitly, using sequence position as a proxy for physical time. But real robot systems rarely run on a fixed clock. Sensors run at different rates, observations arrive with delays, and actions take time to compute and execute. Successful control depends not only on what the robot observes but also on the timing of observations and actions.
We introduce 🕒 Chronos Policy, a diffusion policy that explicitly conditions on observation capture times and requested action execution times. We use time-aware positional encodings to make attention depend on elapsed physical time. Because timestamps let the policy interpret each sequence's timing, we can randomize sensing and action timing during training without making those variations ambiguous. This exposes the policy to a wider range of timing conditions, improving robustness to timing changes at deployment.
Jaden Clark retweeted
Every robot policy should be a 🕒 Chronos Policy
— one that explicitly models physical time across the entire system stack, from observation to inference to action execution.
Because the physical world runs on real time, not sequence indices.
Most robot policies treat time implicitly, using sequence position as a proxy for physical time. But real robot systems rarely run on a fixed clock. Sensors run at different rates, observations arrive with delays, and actions take time to compute and execute. Successful control depends not only on what the robot observes but also on the timing of observations and actions.
We introduce 🕒 Chronos Policy, a diffusion policy that explicitly conditions on observation capture times and requested action execution times. We use time-aware positional encodings to make attention depend on elapsed physical time. Because timestamps let the policy interpret each sequence's timing, we can randomize sensing and action timing during training without making those variations ambiguous. This exposes the policy to a wider range of timing conditions, improving robustness to timing changes at deployment.
Jaden Clark retweeted
I've been thinking a lot about making robot learning scalable. To me, that means learning on the job, and using the human help still needed efficiently.
In Mulligan ⛳️, we focus each data-collection round on the initial states where the robot fails. This happens in deployment, with the robot doing the task and a person stepping in when needed.
On 3 real tasks and 2,550 blind evals, it improves on uniform collection at the same budget.
Jaden Clark retweeted
Per-joint Gaussian noise is a terrible way to explore a 20+ DoF hand
We introduce EigenDEXplore, which explores along coordinated patterns from human hand motion, while keeping full control of every joint
Better learning, from tool use 🔨 to reorientation 🔄 to bimanual 👐
Jaden Clark retweeted
How can we train better behavior from imperfect robot demos?
In NEEDLEWORK, we study how we can get stitching to work on robot data, and more importantly, why these connections help.
Website: needle-work.github.io
Paper: arxiv.org/abs/2610.02339
Jaden Clark retweeted
The recent breakthrough in Navier-Stokes has garnered a lot of attention to the new possibilities that emerge when agents work together.
We have been interested in this question for a while. How can a team of agents achieve more than agents working alone?
I'm excited to finally share our new work, "Self-Organizing Agent Teams Learn to Reason Together" 🧵
Continual learning gets a lot of hype lately, but few works systematically analyze what makes it challenging in robotics. Check out Max's paper for some really interesting insights - the findings were crucial to getting our latest paper MuSe to work
When a robot learns a new behavior, what keeps it from forgetting the ones it already knows?
Prior work, especially on VLAs, shows that rehearsing past experiences can enable strong continual learning. But why does rehearsal work so well, and when does it fail?
We find that a small but ✨special subset✨ of rehearsed data largely determines whether old behaviors are retained or forgotten. We call these examples Memory Anchors. ⚓
Withholding just the top 10% of Memory Anchors from rehearsal increases forgetting by up to 4.5x. Increasing their presence, meanwhile, reduces task forgetting and enables a real robot to learn challenging task sequences.
Website: robot-adaptation.github.io/M…
Paper: arxiv.org/abs/2608.26545
Curious? Read on! 🧵👇 (1/9)
Want to understand how to get ICL like GEN-1.5 and S1 to work? Check out Behavior Prompting
With GEN-1.5 from @GeneralistAI and S1 from @SkildAI, we are seeing exciting progress with in-context demonstration prompting! But can we study this in academia?
In our recent work Behavior Prompting Policy (behavior-prompting.github.io…), we released open-source sim benchmarks and real-world experiments to study in-context physical/behavior prompting without requiring industrial-scale data collection/compute.
We also released iPhUMI, an UMI gripper for collecting behavior prompts to instantly in-context condition the robot at test time.
Important lessons: (1) prompting improves with task diversity instead of more demos per task, (2) behavior prompts act as dense sub-goal conditioning, like a step-by-step instruction book, (3) adding robot actions in the prompt improves prompt following.
If you want to study this too, everything is open-source!
Jaden Clark retweeted
Today we’re launching Enact: post-training infrastructure that makes robotics models work in the real world.
Robotics models fail when execution reaches states their training data never covered. A slip, an off-angle grasp, or an external disturbance can leave the policy without a learned recovery.
Enact finds those failures and generates the targeted recovery data teams use to make their models reliable in production.
@govindchada @enact_labs
Jaden Clark retweeted
Teleop systems are usually designed for a single embodiment, but do they have to be?
Introducing ModPack 🎒: a modular teleoperation interface for bimanual mobile robots.
A wearable backpack provides shared infrastructure, robot-specific leader arms adapt to different embodiments, and plug-and-play modules add capabilities like haptic feedback, active perception, and mobile manipulation. 🧵(1/n)
Jaden Clark retweeted
Success rate has long been the primary metric for evaluating robot manipulation. What about speed?
Today, we introduce ⚡️B-spline Policy (BSP). Instead of predicting discrete fixed-rate action chunks, we parameterize actions as continuous B-spline curves. Together with our system design, BSP enables fast manipulation on low-cost robot arms.
This project is co-led by @xshenhan, check out his following threads for more details. 🧵
PS: one of my favorite parts of this project was the first time we saw the robots move significantly faster and smoother than the baselines. The videos below are all real time. 👇
Jaden Clark retweeted
Long horizon bimanual mobile manipulation requires reasoning in many coordinate frames: base, L/R hands, etc. In which frame would policy work best?
It really depends, so don’t pick. Mixture of Frames Policy: denoise in multiple frame in parallel.
🌐mofpo.github.io (1/9)
We've been using iPhumi for MuSE/UMI-FT - iPhone is a big upgrade from original UMI design with better slam and easier UI!
We are also excited to release iPhUMI ("eye-foo-me”)! It solves the localization challenges with the GoPro UMI, enabling rapid data collection across diverse environments and tasks. During deployment, iPhUMI lets you command your robot via demonstration.
github.com/real-stanford/iPh… (7/8)
Jaden Clark retweeted
Tactile/force data are critical but rare. They can never reach the scale of pretraining tasks, so we got to find intelligence in other ways.
In MuSe, we show that finetuning with a small amount of force data can even improve pretrain tasks, given the model the ability to do force prediction on tasks with no force data before.
The ability is enabled by three key modeling designs. Checkout Jaden's post for details!
Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?
Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)
With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T
We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇
Jaden Clark retweeted
I'm increasingly interested in the problem of *multisensory continual learning*, since it feels inevitable for robotics.
Unlike vision, many robot sensors (e.g., force/torque, tactile, audio) are highly task- and system- specific. It's unrealistic to expect a single pretraining dataset to contain every future sensor. And as robotics evolves, we'll keep building new sensors.
So the question is: Can we plug a new sensor into a pretrained vision-only foundation model without forgetting everything it already knows?
Better yet, can the new sensor actually improve the model's existing vision-based skills?
That's exactly the question that motivated MuSe 👇
Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?
Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)
With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T
We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇
Jaden Clark retweeted
MuSe is our first attempt to enable robots to learn touch while preserving the generalization gained from large-scale vision pretraining. More surprisingly, touch learned later can in turn improve the skills acquired earlier during pretraining.
Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?
Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)
With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T
We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇
Jaden Clark retweeted
WAMs + force data improves on both pre-training and fine-tuning tasks:
Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?
Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)
With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T
We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇