Learning, Robotics, Controls & Open Source Enthusiast
San Francisco, CA
Joined October 2012
- Tweets104
- Following473
- Followers286
- Likes586
Pinned Tweet
Over the last few years, end-to-end robot learning has moved incredibly fast. Even working at the frontier, I’m regularly surprised by how much it can already do. And yet we still haven’t seen it deployed at scale.
At @enact_intel, we think that’s because no one has fully committed to this new paradigm yet. Most approaches fall into one of two camps:
Humanoid companies start with the form factor. The body comes first, built to look like us, and the AI gets adapted to hardware designed before the model existed.
Pure model companies run on off-the-shelf hardware. Fast and convenient, but it limits the modalities and the quality of the data.
We see it differently: the body shapes the data, and the data shapes the model.
So we’re rebuilding the whole stack: model architectures for multi-frequency, multi-scale intelligence, the data pipelines, and all the way down to the actuators and sensors feeding the model. Our robots sense force and touch through their own actuators, so they feel what’s happening in real time. That makes learning more data-efficient while pushing reliability and performance. It works across form factors and is built for real-world deployment from day one.
Huge thanks to everyone who’s helped along the way: early supporters, everyone who took time to give advice, our angels, and the fantastic team I get to tinker on robots with late into the night.
@redalpine @firstminutecap @Notion_Cap @ylecun @AndrewLonsberry @arthurmensch @Thom_Wolf @mapo1 @HaraldNieder @spencer_crawley @ascribblenau7 and many more...
Manuel Yves Galliker retweeted
1/ General-purpose robotics is the rare technological frontier where the US / China started at roughly the same time and there's no clear winner yet.
To better understand the landscape, @zoeytang_1007, @intelchentwo, @vishnuman0 and I spent the last ~8 weeks creating a deep dive on humanoid robotics hardware and flew to China to see the supply chain firsthand.
Here's everything we've created + our takeaways about the components, humanoid comparisons, supply chains, and geopolitics👇
Manuel Yves Galliker retweeted
Introducing MARS, the first Personal AI Robot.
In technology, getting to the future we want starts with building it together.
This is why we designed Mars. Inspired by early PCs, Mars is a powerful, complete, extendable robot.
It runs BASIC, an open embodied AI agent we created for you to program with code, demonstrations, and prompts. This allows Mars to perform complex long-horizon tasks involving spatial memory, reasoning, manipulation and navigation.
Pre-orders open now
⬇️ Demos & details in the thread
Manuel Yves Galliker retweeted
this year alone, I've met hundreds of the world's elite AI researchers + engineers steering the future of intelligence, and they ultimately want to do it here, in the US 🇺🇸
If a visa or green card are holding you back, join us on 7/31 in SF and hear real stories from @MatternJustus, @manumerous, @theaievangelist, and me @lighthousehq_ on getting unblocked to pursue their ambitions
RSVP in thread
Glad to share the Real-Time Action Chunking Algorithm @kvablack, @svlevine and I developed at @physical_int! By inferring the VLA in a receding horizon manner we enable parallel planning and execution improving speed, performance, and motion smoothness.
Manuel Yves Galliker retweeted
We got a robot to clean up homes that were never seen in its training data! Our new model, π-0.5, aims to tackle open-world generalization.
We took our robot into homes that were not in the training data and asked it to clean kitchens and bedrooms. More below⤵️
The last 2 month since I joined @physical_int have been quite the ride bringing robots out of the lab and into unseen environments. With our new model, π-0.5, our robot could clean up kitchens and bedrooms in homes that were never seen in its training data! More below⤵️
More information can be found in our blog: physicalintelligence.company…
Manuel Yves Galliker retweeted
We are excited to share new experiments with AgiBot @AgiBot_zhiyuan on multi-task, multi-embodiment VLAs! With one model that can perform many tasks with both two-finger grippers and multi-fingered hands, we take another step toward one model for all robots and tasks.
Manuel Yves Galliker retweeted
It's All in the Hips
Can we learn anything from the evolution of hip kinematics in humanoid bots?
Is there a growing consensus of the optimal design?
Yes. And there is data to prove it.
First, a primer on Hip Kinematics
🧵
The latest MuJoCo release now enables Massively Parallel GPU-accelerated physics simulation! Gone be the days of debugging proprietary physics simulators! 😃
playground.mujoco.org/
Manuel Yves Galliker retweeted
“Pretraining as we know it will end”
Robotics: Hold my 🍺 , new data coming
Source: Agibot
Wow, with the recent announcements from @OpenAI and @AnthropicAI Zürich starts to look surprisingly similar to Silicon Valley these days. Is someone trying to send me a sign that it's time to pack up my hiking boots and move back home? 🏔️ 🚠🚶🇨🇭 picture by @GreaterZurich
Manuel Yves Galliker retweeted
Can generative video models accurately model real world futures and contact physics?
At @1x_tech, we're working on world models as a scalable architecture to teach robots to solve tasks and evaluate themselves. We're announcing our World Model Sampling Challenge with $10,000 awarded to to the model which best predicts the next robot state from a history sequence.
In collaboration with @NVIDIARobotics, we're releasing a new 100 hour EVE robot dataset of navigation and manipulation encoded with the Cosmos Tokenizer. Cosmos compresses video sequences by 512x into a discrete, temporal representation optimized for world model learning.
Manuel Yves Galliker retweeted
The original RL algorithms, inspired by natural learning, were online and incremental—they were streaming in the sense that they learned from each increment of experience as it happened, then discarded it, never to be processed again. The streaming algorithms were simple and elegant, but the first big successes of RL in deep learning were not with streaming algorithms. Instead, methods such as DQN chopped the stream of experience into individual transitions, then stored and sampled them in arbitrary batches. Subsequent work followed, extended, and refined the batch approach into asynchronous and offline RL, while the streaming approach languished, unable to produce good results in popular deep learning domains.
Until now. Now researchers at the University of Alberta have shown that streaming RL algorithms can work just as well as DQN on Atari and Mujoco tasks (arxiv.org/pdf/2410.14606). How did they do it? Mostly just by getting signal normalization and step-size bounding right for the streaming case—otherwise they use standard streaming algorithms like TD(lambda) and Q(lambda). To me it looks like they were simply the first researchers knowledgeable of streaming RL algorithms to seriously address deep RL without being over-influenced by batch-oriented software and batch-oriented supervised-learning ways of thinking.
Would you believe that deep RL can work without replay buffers, target networks, or batch updates? Our recent work gets deep RL agents to learn from a continuous stream of data one sample at a time without storing any sample. Joint work with @Gautham529 and @rupammahmood.
Manuel Yves Galliker retweeted
You can now watch my early career keynote at #CoRL2024 on the @corl_conf YouTube channel: youtu.be/n3xvEoq2tXg?t=464&s…
Slides are available here: pearl-lab.com/2024/11/20/ear…
Hope you find it interesting!🤖
Honored to give an Early Career Keynote at #CoRL2024! My talk, "On the Quest for Robotic Embodied Intelligence," will explore how structure in robot learning enables faster, safer, and adaptive robot behaviors. Excited to share this vision! 🚀
corl.org/program/speakers #robotics
It was such an pleasure to been able to contribute to this unique workshop! It was great to see so amy different perspectives from both industry amd academia come together! Big thanks to the organizers! 😀 – at Santa Clara, CA
A lot of energy at our workshop to close the week at @corl_conf! Thanks to all speakers, panelists, and the huge number of participants for making it so lively and insightful 🤖
And huge thanks to the other organizers @YoungwoonLee @Xingyu2017 @pabbeel
