rl & robots @UW @NVIDIA, prev @berkeley_ai
Seattle, WA
Joined June 2023
- Tweets105
- Following108
- Followers812
- Likes631
Pinned Tweet
This is the most impressive result I’ve seen in my PhD.
One policy, trained from scratch, solves all 20 NIST assembly tasks. It can even complete 10 tasks in a row, with a single binary reward only after all 10 are done.
@octi_zhang combined diverse resets from OmniReset, task sampling from SGS, and multi-task training to unlock finer dexterity and longer horizons. No extra GPUs needed: one run on the same hardware replaces 20 specialist runs.
Videos and details: alpha-nist.github.io
Proud to be part of this incredible work! More in his thread 👇
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
Patrick Yin retweeted
Exploration is hard for RL when training high-DoF dex hands ✋
X-Reset makes generalist object-centric RL easier by resetting to human hand-object states 🙋♀️
We mitigate the exploration problem, and train a single policy to manipulate a variety of objects to any goal pose 🧵
Patrick Yin retweeted
Three AlphaNIST findings I'm particularly excited about:
1. Task diversity can make difficult tasks easier to learn.
2. Dexterity isn't just about more fingers.
3. Long-horizon coordination emerges without explicit planning.
All with binary task reward!
See @octi_zhang 's thread
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
Patrick Yin retweeted
Amazing to see this level of dexterity emerging from a binary success / fail reward. Great work @octi_zhang !
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
Patrick Yin retweeted
this is super cool, multi-task RL for dexterous assembly!
I'm surprised that multi-task training shows positive transfer between tasks with only 20 tasks, perhaps it's because the skills required are similar? Either way, makes the case for sim rl stronger 😁
(4/6) We started this project with one goal: solve all 20 NIST assembly tasks with RL. Our earlier work on Success-Guided Sampling (SGS) had already solved one of them, so we reused its recipe: a binary task reward with a simple control cost, a broad reset distribution from OmniReset, and the SGS task sampler. Training a specialist policy for each task got us 17 of the 20. The last three, the 4 mm peg, 4 mm rod, and M4 nut, wouldn’t budge. Their policies couldn’t even pick the parts off the table: a slight miss with the gripper pushed these tiny parts away instead of pinching them.
The obvious fix was to design easier resets or denser rewards for those three until they learned. Instead, we wondered: could skills learned on the other 17 tasks help with the hard grasps the last three needed? So we tried what seemed like the harder path: one policy trained on all 20 tasks at once, with each episode starting from any of the 20 parts.
Then the magic happened. The generalist learned to pick up and assemble all three tiny parts, alongside the other 17. Multi-task training, the seemingly harder path, made the hardest tasks easier to learn.
Patrick Yin retweeted
@octi_zhang and team with some pretty cool new results :)
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
Patrick Yin retweeted
Simulation will be a bitter lesson for human data folks 😁
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
Patrick Yin retweeted
@octi_zhang is a legend. the multitask policy learning super impressive
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
Patrick Yin retweeted
At last… torturous, terrifyingly tiny tasks have been solved by RL.
The M4 nut, 4mm peg, and 4mm rod were too small for any single-task policy to learn. Training on all 20 NIST tasks at once enabled success on all of them. And the same policy can do 10 tasks back-to-back, with reward only once at the very end.
Videos and details: alpha-nist.github.io
More in @octi_zhang’s thread
(3/6) We also never told it what order to work in. Over long horizons, AlphaNIST began organizing its own work: bringing parts within reach, moving obstructions aside, and returning to pieces it had left unfinished. It received a single binary reward only when the entire multi-step assembly was complete, so the task order and strategy emerged purely from RL.
Patrick Yin retweeted
What does RL learn from 10 assembly tasks in a row, with sparse reward only at the very end of all 10 tasks? Traditionally, absolutely nothing!
That’s 1,125 actions over 45 seconds. Even if a random policy somehow nails each task 1% of the time, chaining all 10 happens with probability 10⁻²⁰. Worse odds than winning Powerball twice in a row.
Excited to share AlphaNIST, a single policy which achieves 98.2% success rate across all tasks, and is able to solve 10 assemblies in a row under sparse reward! Check out our website: alpha-nist.github.io.
More details in @octi_zhang ‘s thread, and some extra comments of my own below! 🧵
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
Beyond the collaborators @octi_zhang thanks in his thread, special thanks to Karl Van Wyk, who invented the @NIST board, and to Joe Falco and Kenneth Kimble for their work on it since. None of this would exist without them.
Patrick Yin retweeted
Failures are inevitable for robots that are deployed in the *big* world. We introduce ReSYNC, a system that leverages large-scale RL sim2real to recover from local failures and concept discovery to explain/avoid new failures. Check it out here: jaraxxus-me.github.io/ReSYNC…
Patrick Yin retweeted
since people seemed to enjoy our locomotion results, chef 🧑🍳@octi_zhang cooked up a special treat à la SGS on a different embodiment (a wheeled @UnitreeRobotics robot) to celebrate the release! The robot is placed in this big world, and its job is to reach the yellow dots.
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈
Patrick Yin retweeted
The value of scaled RL training in simulation can't be understated. Huge congratulations to @octi_zhang and everyone else who worked on the SGS paper. What comes next is even cooler - stay tuned!
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈
Patrick Yin retweeted
Super impressive work and recovery behaviors from friends at UW! There's a night and day difference between RL trained models vs behavior cloned ones but It's still challenging to scale this up and apply it to different usecases. It needs a couple of ingredients: Good assets that look good and simulate well. The parts are visually simple (no texture, simple material) so that's a win for sim because it makes the rendering realistic. The only tricky object is the reflective board itself which needed to be taped to make sim renderings close enough to real. Also another part is what would happen if the board is not bolted to the table, the recovery behaviors are too wild for a loose board.
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈
Patrick Yin retweeted
so much more to unlock with simulation :)
very cool work! you know this is good when you see others already reproducing it
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈
We kept throwing new tasks at SGS, and the same recipe kept working. New assembly tasks, new terrains, new embodiments. I really believe this is a step toward one RL recipe for every embodiment and task!
Patrick Yin retweeted
A little curriculum learning can make a world of a difference for contact-rich robotic RL! @octi_zhang @mateoguaman @patrickhyin @rosario_scaling showed that choosing sampling strategies carefully while using resets to drive RL exploration in simulation can really shine on high-precision problems in our new work SGS!
The most intuitive way I think about the methodology is by ensuring that the learning signal in any batch of reset-driven RL is maximized by adaptively concentrating reset sampling around states that are "moderate" empirical success rate. This ensures the batch is not wasted on states that are too successful or unsuccessful, enabling RL to solve high-precision problems such as contact-rich assembly and dynamic quadruped locomotion.
This results in some of the most fun robot behaviors we've seen yet on a real robot! I'd recommend reading @octi_zhang's thread and our paper for a more thorough description. Hope y'all have fun using it, it's dead simple!
Website: sgs-rl.github.io
Paper: arxiv.org/abs/2610.12465
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈
Patrick Yin retweeted
@iggydagnino came from the trenches of highschool @FIRSTweets FRC2976 frc-events.firstinspires.org… straight to work with us and got straight to work!! We are so happy to have him working with us at UW and look forward to great research to come!
Did that robot just do a kickflip? Did I just witness emergent tool use?? AGI???
Never thought I’d ever see zero-shot sim2real policies trained only with RL be capable of all this, but here we are…
It wasn't large scale teleop data or a new algorithm that enabled these new emergent capabilities. It's where you reset when training RL. Check out @octi_zhang's thread and the website for more!
sgs-rl.github.io/
🔊
Patrick Yin retweeted
Banger work! It's so cool how far we can go with the right and simple design choices with scaling!
Maybe I should grab some popcorn because it'll take a while to see all these cool results :)
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈