I’m rethinking what we should optimize for in robot generalization. The central question is: what data and pretraining would make a model fast at learning almost any new task? The tasks it already performs, and the demonstrations it was trained on, tell only part of the story. Consider learning tennis. Coding knowledge is not directly needed to hit a ball. Yet humans bring a broadly developed efficient learning system that supports acquiring unfamiliar skills compared to monkeys. My intuition is that diverse experience improves learning efficiency beyond the specific skills it teaches. That makes labels like “spatial intelligence” or “cat-level intelligence” less central to me than understanding how to build a more capable learner. It also changes how I think about System 1/System 2. A sufficiently general world model might eventually handle manipulation efficiently on its own. A world model as System 1, paired with an LLM agent as System 2, could still offer practical advantages in latency, compute, and deployment. But today’s binding constraint in robotics is generalization. Efficiency becomes much more valuable once the system can reliably adapt. My current bet is that multimodal foundation models offer broader pretrained knowledge and visual understanding than the video backbones we currently use for WAMs. I suspect a key advantage lies upstream: their training pipelines can draw on a much larger pool of high-quality visual data and egocentric manipulation data, while highly diverse pretraining—whether in language or vision—equips them to learn brand-new tasks efficiently. Robotics agents offer a way to turn that foundation into learning through interaction, feedback, and experimentation. That is why prioritizing Agent for Robotics now could be a good idea. I expect rapid progress over the next 6–12 months. The question is: How broad pretraining can make robots substantially faster learners? Apprecite the insights shared by @RunhanH!

That’s exactly what we do in VTAM and DexTacWAM at @berkeley_ai: without any tactile pretraining, we transfer world-modeling capabilities learned from vision to the tactile domain—supporting not only grippers, but also distributed tactile sensing on @SharpaRobotics dexterous hands. Take a look! VTAM: plan-lab.github.io/projects/… DexTacWAM: dextacwam.github.io/

Thanks @SharpaRobotics for sharing our summer work about DexTacWAM @berkeley_ai ! Looking forward to further collaboration!

Really excited to see DexTacWAM featured by @berkeley_ai 🖐️🤖 I led this work as first author during my summer visit at BAIR, where we explored how tactile world modeling can scale from simple grippers to multi-finger dexterous manipulation. Recent work such as T-Rex has shown how powerful high-frequency reactive tactile control can be: tactile feedback can refine actions at a much faster timescale than the full vision-language stack, enabling fast correction without giving up the pretrained policy’s broader capabilities. DexTacWAM explores the complementary side: predictive tactile world modeling. A world model can anticipate how contact, slip, pressure, and grasp stability may evolve, while a high-frequency reactive tactile policy can handle sudden disturbances and self-correct when reality deviates from prediction. Predictive modeling for anticipation + reactive touch for correction. Our key finding is that touch should not simply be injected as another policy input. By making multi-finger contact evolution part of the predicted world state, tactile prediction becomes a self-supervised objective for learning richer contact representations. At inference, we do not need to fully generate future visual or tactile observations. A single world-model forward pass exposes predictive visuo-tactile features directly to the action expert. This also raises a broader question: If models like GPT-6 already show strong spatial understanding and manipulation reasoning, why do we still need specialized World-Action Models? I don’t think this kind of spatial capability is magic, or simply language intelligence generalizing zero-shot into the physical world. Some evidence suggests that it is enabled in large part by large-scale visual, video, and interaction data. OpenAI itself describes GPT-6 Astra as significantly stronger on visual/computer-use tasks, while its public materials do not disclose the exact training-data recipe. That distinction matters. If spatial intelligence ultimately comes from learning structure and dynamics from visual and interaction data, then the real question is: what is the most efficient model to run for physical control? A frontier multimodal model also carries enormous capacity for language, coding, knowledge, tool use, and general reasoning. Even with sparse MoE routing, that generality still comes with substantial memory, communication, and inference overhead. For high-frequency manipulation, repeatedly invoking such a model can impose latency and cost that are hard to justify. Low-level dexterous control does not need the full machinery of language at every timestep. Language is extremely useful for instruction, semantics, task decomposition, and human communication—but the inner loop of manipulation is fundamentally about physical dynamics. That is where we still need World-Action Models. A WAM can concentrate most of its capacity on: observation + action → future physical state → next action Instead of being a general-purpose model that also happens to understand physics, a WAM can be a physical-modelity-centric architecture optimized for visual dynamics, contact evolution, action-conditioned prediction, and low-latency control. That makes it a natural model to deploy at the edge. DexTacWAM also suggests that this edge world model does not need to remain vision-only. Human video is abundant, while tactile data is scarce and expensive. Continual vision-to-touch learning lets us reuse a pretrained video prior and extend it to distributed multi-finger tactile dynamics with relatively little tactile data. This points to a broader architecture I find increasingly compelling: ☁️ Cloud multimodal foundation model language, semantics, long-horizon reasoning, task decomposition, skill selection ↓ ⚡ Edge World-Action Model visual-tactile dynamics, predictive features, short-horizon control ↓ 🖐️ High-frequency reactive tactile loop fast correction under unexpected contact changes The cloud model decides what to do. The WAM predicts how the physical interaction should evolve. The tactile loop corrects when reality deviates from prediction. I think this division of labor could offer a better combination of intelligence, latency, energy efficiency, robustness, and deployment cost than asking one giant model to solve everything at every timestep. DexTacWAM is one step toward that picture: predictive video priors as compact physical models, continually extended to touch and paired with fast reactive feedback for robust dexterous manipulation. 🖐️ DexTacWAM: dextacwam.github.io/ 📄 Paper: arxiv.org/abs/2609.24976 Grateful to the entire team for making this project possible, and especially to everyone who supported the project throughout the summer at BAIR. 🙏

Join us if you’re interested in visuo-tactile world action models (WAMs) and contact-aware manipulation!