Pinned Tweet
World-action models typically imagine the future in RGB – but are pixels really the right representation for robotics?
Our bet is no: RGB spends capacity on fine-grained details and variation that are often irrelevant for robot policies.
DINO features, point tracks, and depth capture more useful features like semantics, motion, and geometry.
But no single modality captures everything — how can we effectively combine them?
We introduce ✨ModAR✨, which predicts the future one modality at a time, with each prediction informing the next. We train from scratch and find that this formulation performs best.
Our 30.1M scratch-trained model even outperforms a 6B video-model-initialized model finetuned on the same data!
🧵 [1/8]
Adam Hung retweeted
Students sometimes ask me if it still makes sense, in this accelerating age, to pursue a PhD in AI.
Perhaps counterintuitively, I think it's a great time to do so.
I wrote up some thoughts on this here: web.mit.edu/phillipi/www/wri…
Adam Hung retweeted
Concept bottleneck models promise interpretable vision: predict named attributes first, classify from them. Where do the attributes come from? Usually an LLM prompted with the class name. No image is ever consulted. That turns out to be the weak link 🧵
ALT For "strawberry" on ImageNet-Sketch, an LLM lists red and ripe because it never sees an image. Scoring attributes against the sketch images returns what actually holds: colorless, hand-drawn.
Instead of training a world model to condition on robot actions, PointZero conditions on partial 3D point tracks. This allows it to learn rich 3D dynamics priors from actionless data!
We found that pretraining on this objective effectively transfers to a variety of robotics applications, outperforming strong application-specific baselines on imitation learning and robot-action-conditioned dynamics modeling.
We trained PointZero only on synthetic data, but the zero-shot real-world dynamics predictions were surprisingly impressive!
Super cool work led by @bardienus
Introducing PointZero—a 3D world model pre-trained without robots.
Dexterous manipulation requires understanding diverse 3D dynamics, but current approaches rely on expensive robot data. So how can we pre-train a 3D dynamics model without it?
PointZero introduces a simple idea: learning to complete 3D point tracks yields transferable 3D dynamics!
Our model achieves SOTA results on:
🔥 Zero-shot 3D dynamics
🔥 Action-conditioned 3D dynamics (post-training)
🔥 Imitation learning (post-training)
Details and links 👇 1/6
Adam Hung retweeted
Introducing PointZero—a 3D world model pre-trained without robots.
Dexterous manipulation requires understanding diverse 3D dynamics, but current approaches rely on expensive robot data. So how can we pre-train a 3D dynamics model without it?
PointZero introduces a simple idea: learning to complete 3D point tracks yields transferable 3D dynamics!
Our model achieves SOTA results on:
🔥 Zero-shot 3D dynamics
🔥 Action-conditioned 3D dynamics (post-training)
🔥 Imitation learning (post-training)
Details and links 👇 1/6
Adam Hung retweeted
Humans learn by interacting with the world: How can we teach robots to similarly adapt to ambiguous dynamics? Introducing Wiggle and Go! #CoRL2026. We find that a specialized model can learn rope properties from just one interaction, and complete complex dynamic tasks.
The robot wiggles a rope once, figures out how it behaves, then goes, using the estimated dynamics to execute the task. No real-world training data. No retries. #robotics #robotlearning
Thank you to collaborators @BDuisterhof, Deva Ramanan, and @jeff_ichnowski! This work was supported by the NSF GRFP, and used Bridges-2 at PSC through ACCESS.
Check out the paper for details and the project page for more visualizations and rollouts!
📑 arXiv: arxiv.org/abs/2609.17524
🌐 Project Page: adamhung60.github.io/ModAR/
@CMU_Robotics @SCSatCMU
We also tested ModAR on real-world bimanual manipulation tasks:
ModAR beats baselines and also shows that it can improve its performance by training on human videos,
using videos from both the same setting (ID) and a public dataset (OOD).
🧵 [6/8]
Next: how should future prediction and action prediction be coupled?
• Unified: denoise everything together
• Disjoint: denoise futures and actions independently; skip generating futures at test time
• ModAR: denoise modalities sequentially, then actions
ModAR wins at every data scale – and benefits most from actionless data.
🧵 [4/8]
We also fine-tune Flex-π, a recent multimodal WAM that initializes from a pretrained video model, on the same data. Despite ~200× fewer parameters (30.1M vs 6B) and ~20× fewer training FLOPs, ModAR achieves a slightly higher average success rate: 75% vs 72%.
We think this demonstrates a promising new direction:
scaling from-scratch WAMs with structured representations that transfer better to manipulation,
rather than relying on RGB-centric video models.
🧵 [5/8]
📑 arXiv: arxiv.org/abs/2609.17524
🌐 Project Page: adamhung60.github.io/ModAR/
Why generate modalities one at a time?
The idea is simple: predict the future in easier, more structured forms first, then use those predictions to help generate harder, more detailed modalities.
To test this, we conduct a controlled study across six RoboTwin tasks, varying the predicted modalities, WAM formulation, and amount of actionless training data.
We ask two main questions: which modalities to predict, and how to couple future prediction with action prediction.
🧵 [2/8]
Adam Hung retweeted
Today we are introducing Atlas! An autoregressive and multimodal DiT for generating image and depth frames. Atlas is a state-of-the-art 3D reconstruction model and outperforms all open-source models. @HaoZhang623 and I had lots of fun pushing the 3D capability of this model; can't wait to see you build with it!
Adam Hung retweeted
[1/6] Can one model fix rendering artifacts from any 3D representation? And what does it even mean to "fix" a rendering?
At #ECCV2026, we present FixAnything (fix-anything.github.io): a single generalist video model that refines 3DGS/mesh/sparse point-cloud renderings into photorealistic, 3D-consistent videos.
Adam Hung retweeted
Can robot learn manipulation skill with one object and perform it on a completely different one without any new demonstrations?
Excited to introduce SemAnCorr, a framework for dense correspondence and manipulation skill transfer across diverse objects.
semancorr.github.io
Adam Hung retweeted
How can we get robot hands to “hear” slip and contact through microphones, and react to them?
We’re excited to share VibeAct, an approach that uses piezoelectric microphones embedded in robot fingertips to estimate contact and slip, then learns reactive policies from this tactile feedback!
vibeact.github.io
Adam Hung retweeted
Excited to share the first paper of my PhD!
If you’ve ever tried to control a VLA via natural language, you know it rarely does what it is told. 🗣️ We introduce a multi-stage pipeline for training a Language Feedback Policy (LFP) to steer a VLA in-the-loop.
Adam Hung retweeted
Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation!
Text-to-image (T2I) models learn rich representations of the spatial world.
How do we build on this prior for high-quality depth generation?
modality-forcing.github.io
🧵 [1/6]
Adam Hung retweeted
Excited to share SoftAct, a framework for retargeting human manipulation demos to soft robot hands using explicit contact force reasoning! How do you transfer human skill to a hand that looks and moves nothing like yours🐙🖐️? It turns out VR environments can let us capture privileged force interaction demonstrations to help. 🧵1/7