Robotics Tinkerer. RS @Amazon FAR Prev: @META (FAIR), @DFKI, @TUDarmstadt https://xitter.cf/t.co/RQpq7Prbln X https://xitter.cf/t.co/umZQeDjJv4

Joined November 2017
A first glimpse of the wildest robotics project of my PhD. 🛹 The Blind Skateboarder 🛹 No vision. No markers. Paper, code, weights — coming soon. And we have more. 😉
250
506
125
4,952
513,763
Julen Urain retweeted
Presenting PTLD: an approach to learning tactile dexterous policies without ever simulating the tactile sensor. Tactile is essential for performing highly dexterous manipulation. However, collecting tactile observations reliably has been a fundamental bottleneck: (1) teleoperating a multi fingered hand for dynamic tasks is challenging, making sim-to-real imperative, (2) one can’t realistically simulate tactile today
6
46
10
188
38,755
Julen Urain retweeted
We are excited to announce the 2026 cohort of RSS Pioneers! This year’s cohort brings together an outstanding group of early-career researchers whose work spans the breadth of robotics. A heartfelt thank you to all the organizers who made this year’s program possible.
16
8
76
10,939
Julen Urain retweeted
1/ World models are getting popular in robotics 🤖✨ But there’s a big problem: most are slow and break physical consistency over long horizons. 2/ Today we’re releasing Interactive World Simulator: An action-conditioned world model that supports stable long-horizon interaction. 3/ Key result: ✅ 10+ minutes of interactive prediction ✅ 15 FPS ✅ on a single RTX 4090🔥 4/ Why this matters: it unlocks two critical robotics applications: 🚀 Scalable data generation for policy training 🧪 Faithful policy evaluation 5/ You can play with our world model NOW at yixuanwang.me/interactive_wo…. NO git clone, NO pip install, NO python. Just click and play! NOTE ⚠️ ALL videos here are generated purely by our model in pixel space! They are **NOT** from a real camera More details coming 👇 (1/9) #Robotics #AI #MachineLearning #WorldModels #RobotLearning #ImitationLearning
26
85
28
504
130,742
Pretraining with dynamics models of motor behavior (aka world models) from video will be much more central to robotics than VLMs. There are multiple choices of representations (e.g. 3D? JEPA?) but we will figure this out by and by. Exciting times!
Replying to @DrJimFan
- Project website: dreamdojo-world.github.io/ - Paper: arxiv.org/abs/2602.06949 - Code repo and model ckpts: github.com/NVIDIA/DreamDojo This is a huge team work at NVIDIA. All credits go to the wonderful teams who poured their hearts into it!
12
28
2
363
83,444
Julen Urain retweeted
Fully agreed with the sentiment that much of computer vision research (concretely, those not for “human consumption”) should be grounded in robotics. But as a robotics researcher, I think the more nuanced question is: how can we *rethink* these intermediate representations for embodied intelligence rather than discarding them? Why? The challenge, as also pointed out in Vincent’s article, is precisely the lack of perception-action data at scale. This is why intermediate representations IMO are *preferable rather than obsolete* because they open up training from scalable data sources. This can include even the vision/language encoders people love and use in robot learning — it’s hard to imagine training low-level visual representation or high-level language understanding purely from limited robot data. The same goes for intermediate representations at the structure level — world modeling, learning from Internet videos, learning from humans, and simulation — many of which still rely on 3D representations too.
In my recent blog post, I argue that "vision" is only well-defined as part of perception-action loops, and that the conventional view of computer vision - mapping imagery to intermediate representations (3D, flow, segmentation...) is about to go away. vincentsitzmann.com/blog/bit…
1
5
2
81
11,355
At the RI seminar at CMU yesterday, I presented a 3 level analysis of robot skills & discussed the pros and cons of teleoperation, simulation, and learning from videos, before presenting our research. Enjoy! youtube.com/watch?v=ry8itipz…
9
38
3
356
103,081
Julen Urain retweeted
As video world models become increasingly powerful, do we still need explicit 3D? A commonly misunderstood point is this: video world models are not “just 2D.” Their ability to maintain multi-view consistency, temporal stability, and realistic interaction necessarily implies that their latent knowledge encodes 3D world structure. Without some notion of 3D, consistency itself would not be possible. The real distinction, therefore, is not whether a model has 3D but whether that 3D exists implicitly or explicitly. Implicit 3D lives inside latent spaces and network weights. It supports generation, but it is difficult to localize, edit, constrain, or reason about. It allows the world to exist, but not to be used. Explicit 3D, in contrast, exists as structure and state: it is addressable, editable, composable, and transferable. Its purpose is not better visual fidelity, but operability to allow the world to be manipulated, controlled, and executed. From this perspective, video and 3D are not competing paradigms but a layered system: 2D/video is the interface to human perception; 3D is the interface to the physical world. They can reinforce each other, but neither forms a closed loop on its own. In practice, data not model architecture sets the upper bound of world models. Explicit 3D may not be the final user-facing representation, but it is likely the most effective pathway toward scalable, high-quality, and controllable data. Through explicit 3D/4D representations, worlds can be constructed systematically: interactions can be programmatically sampled, states and actions can be composed, rendered into images and videos, and fed back to train video world models. Seen this way, 3D is not the destination it is the starting point for scaling. What truly drives progress forward is never the model itself. Whether we capture the world or imagine new ones, whether data comes from observation or intent, whether we model what is or what should be the direction of the world is ultimately determined by human choice and purpose. Models may extend the world, but humans decide where it goes. #Genie3 #worldmodel
10
24
7
272
54,758
We just released AINA, a framework for learning robot policies from Aria 2 demos, and are now open-sourcing the code: github.com/facebookresearch/…. It includes: ✅ Aria 2 data processing into 3D observations like shown ✅Training of point-based policies ✅Calibration Give it a try!
4
31
7
141
28,522
Julen Urain retweeted
After a year of team work, we're thrilled to introduce Depth Anything 3 (DA3)! 🚀 Aiming for human-like spatial perception, DA3 extends monocular depth estimation to any-view scenarios, including single images, multi-view images, and video. In pursuit of minimal modeling, DA3 reveals two key insights: 💎 A plain transformer (e.g., vanilla DINO) is enough. No specialized architecture. ✨ A single depth-ray representation is enough. No complex 3D tasks. Three series of models have been released: the main DA3 series, a monocular metric estimation series, and a monocular depth estimation series. The core team members, aside from me: @HaotongLin, Sili Chen, Jun Hao Liew, @donydchen. 👇(1/n) #DepthAnything3
78
485
81
3,545
516,476
Julen Urain retweeted
Introducing DINOv3 🦕🦕🦕 A SotA-enabling vision foundation model, trained with pure self-supervised learning (SSL) at scale. High quality dense features, combining unprecedented semantic and geometric scene understanding. Three reasons why this matters…
12
137
17
1,011
136,266
Touch perception holds the key to unlock robot dexterity Our new @SciRobotics work shows how to fuse tactile & vision, track pose+shape of novel objects during dexterous manipulation science.org/doi/10.1126/scir… It's a culmination of our work over the last 4 years, see @Suddhus 🧵⬇
For robot dexterity, a missing piece is general, robust perception. Our new @SciRobotics work combines multimodal sensing with neural representations to perceive novel objects in-hand. 🎲 Featured on the cover of the November issue! #ScienceRoboticsResearch 🧵1/9
1
12
1
80
6,927
Perfect start to the #CoRL2024 week! Was a pleasure organizing the NextGen Robot Learning Symposium at @TUDarmstadt with @firasalhafez @GeorgiaChal Thanks to the speakers for the great talks! @YunzhuLiYZ @NimaFazeli7 @Dian_Wang_ @HaojieHuang13 @Vikashplus @ehsanik @Oliver_Kroemer – at Darmstadt, Germany
1
7
4
61
9,760
We extended the workshop paper deadline to 06/17 AOE. Feel free to submit if you didn't catch! ✅ We also welcome volunteers for reviewing the workshop papers, even if you are an author! 🔖
🤖 Excited for #RSS2024? Don’t miss our workshop on "Geometric and Algebraic Structure in Robot Learning"! Submit workshop papers (by 6/10 AOE) and dive into discussions on leveraging these structures for enhanced #robotics. 🚀🔍 Join us on 07/19 in Delft, Netherlands!
4
9
1,852
Julen Urain retweeted
Come check out our workshop today for some great discussions on roadmap for skill learning for manipulation🛣️ @ Monday 9am (JST), @ G6, Pacifico Yokohama North!
This quoted post is unavailable.
2
45
3,641
Julen Urain retweeted
A major challenge to constructing foundation models for decision making is data scarcity. We present a “compositional foundation model”, which addresses this by composing existing foundation models, each capturing a sub-part of decision making. hierarchical-planning-founda… (1/4)
4
37
4
216
93,300
🔥Theseus 0.2.0 release is out! github.com/facebookresearch/… 🚀Brings it to PyTorch 2.0 🚀Introduces torchlie and torchkin - efficient standalone libs for differentiable Lie groups and kinematics More updates and round up of few research projects in community enabled by Theseus 🧵👇
2
18
2
88
26,865
Replying to @robotgradient
@theCamusean, @n_w_funk, @Jan_R_Peters and me are very happy to receive the best paper award at the workshop on Geometric Representations: The Roles of Screw Theory, Lie algebra, & Geometric Algebra @ieee_ras_icra! Thank you to the organizing committee! – at Greenwich, London
4
9
62
4,862
Interest in hierarchical policy blending as inference for reactive robot control? Check out our ( @theCamusean @Jan_R_Peters @GeorgiaChal ) poster tomorrow ( June 1st ) @ieee_ras_icra from 09:00-10:40 at PODS 39-40 :D
Happy to share that Hierarchical Policy Blending as Inference for Reactive Robot Control (HiBPI) was accepted at #ICRA2023! Many thanks to @theCamusean,@Jan_R_Peters & @GeorgiaChal from @ias_tudarmstadt @TUDarmstadt ! Interested? Check out sites.google.com/view/hipbi 1/5
9
1
22
9,438
Julen Urain retweeted
[LG] Riemannian Flow Matching on General Geometries R T. Q. Chen, Y Lipman [Meta AI] (2023) arxiv.org/abs/2302.03660 #MachineLearning #ML #AI
1
16
2
100
14,026