Trying to understand the emergence of generally intelligent robotic behavior at @berkeley_ai. Previously @CILVRatNYU @MIT & @Apple AI/ML fellow.
New York, NY
Joined October 2010
- Tweets994
- Following724
- Followers3.4K
- Likes204K
Pinned Tweet
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data?
Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵
Mahi Shafiullah 🏠🤖 retweeted
How do we scale dexterous manipulation data for real robots? Do as I Do is a retargeting algorithm for turning internet or generated videos into high-quality, executable robot data for high-dof hands. Learn more on @RoboPapers ->
Achieving human-like dexterity is the next frontier for robotics, and yet dexterity data is often subtly hard to scale. Real-world dexterity data, including things like finger-pose estimates, is often slightly off, making it physically invalid and hard to execute on real hardware and hard to learn from.
DO AS I DO is an algorithm for reconstructing and retargeting monocular RGB videos to robot hands, outperforming the state of the art and even working from generated videos. @bhawna_paliwal_ @HarithejaE @willjhliang and @notmahi join us to tell us more.
Watch Episode 107 of RoboPapers, with @chris_j_paxton and @DJiafei, now!
Mahi Shafiullah 🏠🤖 retweeted
Incredibly excited to return to ETH Zurich as Assistant Professor of Robotics and Artificial Intelligence next summer. I'll be recruiting PhD students to build a new lab on general-purpose embodied intelligence for humanoid robots and dexterous manipulation. Details coming soon!
Are we experiencing a GPT moment in robotics? Our MolmoSpaces benchmark seems to suggest so. Check out @omarrayyann's thread to see the details 👇
GPT-Astra outperformed all open-source VLA/WAM baselines on a subset of our zero-shot MolmoSpaces-v1 benchmark! traces: orayyan.com/ms-results
Mahi Shafiullah 🏠🤖 retweeted
AgentSTAR is an agentic method for both reconstruction and tracking of articulated objects from monocular videos.
We can now track through rapid motion, severe occlusion and thin objects!
There’s no spoon but we can still track it 🥄
Learning mobile manipulation is hard because collecting data for mobile manipulation is hard. But seems like @omarrayyann has found a promising way to crack this problem!
Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.
Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵
Presenting the poster at #RSS2026 in 25 minutes, please stop by!
We will be in the #RSS2026 poster session at 6:30 PM – stop by with all of your questions!
I'm at #RSS2026 – presenting MolmoSpaces on Tuesday & CAP (Contact Anchored Policies) on Wednesday.
I've been thinking a lot about what robotics 2-5 years from now looks like: beyond teleop and position control. If you're interested about anything from robot free/human data or sim evals to force/torque controlled dexterous hands, let's chat! 🧵
CAP tries to drill down onto what we need to condition robots – must language (and the associated bulk) inevitable? We find that contacts may be a much better representation. Presenting on Wednesday in Imitation Learning 2 (3:30 PM). 3/4
xitter.cf/notmahi/status/2021272…
Finally happy to chat about Do as I Do, even though we're not presenting it here at RSS: mostly about the future of human data in robotics, and why I think it's inexorably tied with the rise of dexterous hands. 4/4
xitter.cf/notmahi/status/2067640…
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data?
Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵
I think it's a good investment of time for anyone working on robot policy learning to understand where our algorithms came to be; even when you can just use off-the-shelf packages to get a robot going in no time. Also, we have many practical tips that should be relevant for imitation-learning based manipulation research:
supervised-robot-learning.gi…
Thank you for the kind words, @zuwang95!
supervised-robot-learning.gi… One of the best intros I always recommend ppl new to robot learning to watch. @notmahi is my robot learning 101 professor.
Mahi Shafiullah 🏠🤖 retweeted
If the end goal of robot hands is to perform human motion, then we should optimize the hardware design with human motion - and on a large scale!
We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.
One of the underappreciated factor in learning dexterity from humans is the hardware – imitating humans the way we did would be much more difficult if we didn't have a roughly human sized hand.
Thank you @SharpaRobotics for your support!
We can use videos from the internet to teach robots!
Do as I Do, from
@bhawna_paliwal_ , @HarithejaE , and Willian Liang at UC Berkeley — advised by @pabbeel, @notmahi, and @JitendraMalikCV, used an algorithm that reconstructs hand-object interactions from monocular RGB video and retargets them into real, executable trajectories for multi-fingered dexterous hands. Just using "low quality" video footage of humans doing tasks. No sensors.
The Sharpa Wave robot hand being anthropomorphic, it matches human kinematics. Not only that works, but at fast speed, too!
Congrats to the team, that's super exciting!
Project: do-as-i-do.com/
#Robotics #SharpaWave #Sharpa #EmbodiedAI #DexterousManipulation #RobotLearning
Mahi Shafiullah 🏠🤖 retweeted
Excited to release Do As I Do: a pipeline that turns everyday RGB human videos into dexterous robot manipulation trajectories!
Most prior work has been narrow, consisting of just lab recorded demos, egocentric-only, or assuming a closed set of objects. We develop a modular pipeline that can handle Internet, egocentric, exocentric, AND generated videos with virtually any rigid object. Also check out Mahi's post below!
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data?
Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵
Mahi Shafiullah 🏠🤖 retweeted
We can convert human videos to robot hand-object interaction trajectories in 4D. Enjoy!
Paper: arxiv.org/abs/2606.19333
Website: do-as-i-do.com
Code: github.com/malik-group/do-as…
Authors:@bhawna_paliwal_,@HarithejaE,@willjhliang, @pabbeel , @notmahi , @JitendraMalikCV
Everything is better when grounded in physics – check out @willjhliang's thread on how to properly ground your human data in a robot embodiment, even when the recording is casual and the reconstruction is noisy.
Introducing Do as I Do 👀, a framework to transform everyday human videos into 100s of dexterous robot demos. Co-led with @bhawna_paliwal_ and @HarithejaE, and check out @notmahi's thread!
Here’s a little preview of our dexterous manipulation results. More about how we produce them from human reconstructions in this mini-thread! 🧵
xitter.cf/notmahi/status/2067640…
Mahi Shafiullah 🏠🤖 retweeted
Enabling learning motion directly from videos rather than using them for action supervision is a superior method and likely more scalable.
While it is early this line of work suggests replicating the playbook that made robots walk.
--> Real videos provide state supervision (not action)
--> retargeting provides reference trajectories.
--> RL tracks these trajecotries.
This is a very good example of the separation of the "What" and the "How"
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data?
Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵