Chief Scientist of Rhoda AI prev. PhD Student, Stanford University
Joined December 2020
- Tweets79
- Following110
- Followers913
- Likes98
Pinned Tweet
Today, we announce our team’s progress in pursuing a different type of foundation model for robotics: the Direct Video Action Model (DVA), which does our best to take robotics and turn it into a generative modeling problem we can scale.
Technical blog: rhoda.ai/research/direct-vid…
The result is a new way of learning representations that is stable, informed by the theory, and most importantly, scalable. And it's a general recipe, so maybe we'll start to see some of these techniques applied to new domains and modalities!
One key insight is a connection to contrastive learning. Contrastive methods learn to distinguish pairs of observations that occur together from randomly paired observations. The optimal score for this task is pointwise mutual information (PMI): a measure of how much more likely two observations are to occur together than by chance.
Rather than learning PMI indirectly through contrastive learning, our method uses likelihoods from a generative model to estimate PMI and directly supervise the representation!
Excited to share @BaadeAlan's latest work: a new approach to representation learning designed specifically for generative modeling!
Semantic representations like DINO have become remarkably effective tokenizers for image and video generation. But DINO was never designed for this task. Its success in generative modeling is almost accidental. What if we learned representations specifically to make generation easier?
1/ The tokenizer is a new scaling axis!
We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse.
We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion
w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI
Thread, Paper, and Blogpost below
Eric Chan retweeted
Very excited about this one: a completely new representation learning objective, with a proper theory of what it converges to: the representation of an image converges to features such that their dot product with another image's representation is their pointwise mutual information! @BaadeAlan knocked it out of the park :) (1/n)
1/ The tokenizer is a new scaling axis!
We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse.
We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion
w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI
Thread, Paper, and Blogpost below
Eric Chan retweeted
(1/n) Watch our robot search a box and find textured balls, entirely without vision!
Over the past few months, it’s been incredibly fun to work on bringing touch into our Direct Video-Action (DVA) model. Here’s how our video model is “seeing” through touch 🧵
Eric Chan retweeted
Touch is essential to human manipulation. Yet, we have so far not had a tactile sensor that can do what human skin can: sense texture, firmness, vibration at a viable form factor. At Rhoda, we have built a new sensor in-house that can do all of these things, while being much much smaller than commercially available ones!
I often get asked how world models will integrate tactile data, when they’re trained primary from video.
Here’s a simple solution that works surprisingly well: embed the tactile information into the video stream, as pixels. At Rhoda, we’ve developed inexpensive and sensitive tactile sensors, and have demonstrated their capabilities, including allowing a blind robot to sense. It’s an exciting move towards greater dexterity!
Eric Chan retweeted
More industries. More real customer work.
Introducing RhodaAtWork, a series on the customer workflows our robots are learning to handle across manufacturing and logistics.
First up: unpacking, scanning and stacking component reels for electronics manufacturing.
Eric Chan retweeted
Rhoda AI checked whether scaling web video pretraining actually helps real robots, using a real customer task. Unpacking 10kg boxes of bearings and sorting the packaging.
Turns out yes. Bigger models do better, more pretraining compute does better, and the compute gap is widest when you don't have many robot demos.
Refreshing to see a company display this amount of rigour. Rare these days.
We recently came out with our blog post, Does Scaling Web-Video Pre-training Help Real Robots Do Real Work?
This is a pretty exciting moment for robotics, because as far as I am aware, it is the first evidence that robot foundation models can scale on internet video data; not merely on teleoperation, UMI data, or egocentric data, but on the data you can truly find anywhere.
It’s the necessary requirement for reaching massively powerful models.
An insightful study from @tongzhou_mu and the @RhodaAI team that examines how web video pre-training translates to robot performance.
The gist: pre-training doesn’t just help; it scales, i.e. you can continue to reap greater benefits as you put more effort into pre-training. A better pre-trained model will lead to a more robust policy.
But how you achieve that pre-training quality is up to you! Whether by scaling the model, or scaling your training tokens, it’s a design choice informed by your compute constraints, training budget, and how much you care about compute optimality.
Does scaling pre-training on general web video improve a complex manipulation task in real deployment?
We scale model size and pre-training compute, and test on one industrial task.
Yes. The better a pre-trained model predicts web video, the better its post-trained policy. 🧵
ALT Seven pre-trained video-model checkpoints on one chart. Task performance on a real robot (at-speed completion rate: every requirement met within 100 s, 0 to 100%) rises with pre-training quality (DINO FD on held-out web video, log scale, axis reversed so better is to the right). XS, S, M, L are model sizes. The four M points are one model at 0.08×, 0.18×, 0.37× and 1× of its pre-training compute.
Eric Chan retweeted
I was a guest on the @MIT_CSAIL podcast to discuss robots, AI, and what the near future might look like. @klgiven did a great job steering the conversation, and I think much of our chat is accessible to non-experts - hope you find it interesting!
bit.ly/4zruZdt
Also on Spotify:
open.spotify.com/episode/5QX…
Eric Chan retweeted
I'll be at ICML to present Latent Forcing!
Come stop by the poster on Tuesday at 2:00 PM KST, Hall A #3507
Reach out if you want to chat about the future of generative models!
What's the right space to diffuse in: Raw Data or Latents?
Why not both!
In Latent Forcing, we order a joint diffusion trajectory to reveal Latents before Pixels, leading to improved convergence while being lossless at encoding and end-to-end at inference.
w/ @drfeifei+...
1/n
Eric Chan retweeted
Robot learning is moving beyond policies built for one robot, one scene, one task.
At MIT, we’re exploring a different path: turning video world models into embodiment-agnostic robot policies.
Introducing VERA: a 14B video-to-action system that controls robots across embodiments, skills, and environments.
From zero-shot pick-and-place on a real Panda arm to contact-rich cube reorientation with a 16-DoF robotic hand.
Different robots. Different environments. Different tasks.
Same video planner. Same weights.
We’re open-sourcing everything so you can fine-tune VERA for your own robot setup too. Deep dive in the thread:
🔗 vera.csail.mit.edu
🧵 (1/7)
Delighted to have you on the team! Here’s to many new insights together :)
I am thrilled to join rhoda.ai @RhodaAI as an advisor, where I am helping harness the abilities of large-scale pre-training and video models for robotics, putting many of my lab's research learnings of the past few years into practice! I will be in-person at the Mountain View office for part of the summer - reach out if you want to chat :)
Eric Chan retweeted
Introducing MilliVid, our new method for long-context video generation! MilliVid creates videos that are consistent over long time spans, without using retrieval heuristics or 3D maps! (1/n)
davidcharatan.com/millivid/#
Eric Chan retweeted
The shell game is a fun challenge that cannot be solved by looking at a single frame. The model has to track every move, from the moment the object is hidden. Excited to share this!
Eric Chan retweeted
Here’s something we’ve never seen done before.
Real-world tasks are long and ambiguous. Solving them requires visual memory and state tracking. Most robot policies only see the last few frames. Ours doesn't.
We put our DVA, FutureVision, to the perfect testbed: the shell game 🐚. The DVA nails it.
Eric Chan retweeted
1/ We are speed running industrial robotics.
It took us just 19 days from the first day of data collection to filming a 2.5-hour continuous run of our model autonomously breaking down industrial containers — zero human intervention.
The data efficiency of our DVA model is fundamentally changing how fast we bring robots out of the lab and into the factory.
Autonomous operation with 3 hours of data collection at a customer factory.