Cool work!

Congratulations @canondetortugas!

Real2Sim2Real, finally done right with “neuro-symbolic representations” and a new technique we call “controlled diversification”! Introducing Dex-One2Many: from one human video in one setting to a robot that handles many configurations beyond the demonstration—different object positions, goal poses, and grasps. How did we achieve such remarkable generalization? The key is controlled diversification—generating diverse training experiences in simulation while preserving the underlying task constraints. The symbolic structure specifies what must hold at each stage; the neural policy discovers how to achieve it. Procedurally: (1) **Extract the task structure**. A vision-language model converts the demonstration into a sequence of scene graphs that capture the task’s relational structure: which object the hand should hold, what a tool should contact, and where an object needs to end up. These relationships preserve the task while leaving room for different configurations and motions. (2) **Diversify under constraints**. We use these scene graphs to generate diverse, physically valid training states across different object positions and grasps, while preserving the task-defining relationships. The graphs also provide dense rewards that guide RL at each stage. (3) **Learn through reinforcement learning**. We train the policy in these diversified simulation environments. Rather than imitating the demonstrated trajectory, the robot discovers different ways to accomplish the same task. This is how one demonstration becomes a starting point for learning many ways to accomplish the same task. Across five manipulation and tool-use tasks, the learned policies transfer zero-shot to a real dexterous hand, achieving 75% mean success on configurations not shown in the demonstration, compared with just 4% for the next-best baseline—with no real-world fine-tuning. A wonderful collaboration with @jusukle, @JayLEE_0301, @jbhuang0604, and the entire team! Project: dex-one2many.github.io/ Paper: arxiv.org/abs/2610.12470 #Robotics #NeuroSymbolicAI #ReinforcementLearning #DexterousManipulation

Real2Sim2Real, finally done right with neuro-symbolic representations! Introducing Dex-One2Many: from one human video in one setting to a robot that generalizes to configurations beyond the demonstration—different object positions, goal poses, and grasps. The key to the remarkable generalization is the abstraction. Dex-One2Many extracts the task’s relational structure into “scene graphs”, then uses them to generate diverse training states and rewards for RL in simulation. The graphs specify what must hold at each stage (constraints, our inductive bias); RL discovers how to achieve it across different configurations. One demonstration opens up many ways to accomplish the same task—and the learned policy transfers zero-shot to a real dexterous hand. Project: dex-one2many.github.io⁠