Towards Physics of Multimodal Pretraining Knowledge Flow, Modality Synergy, Early Unification, and Recipes paper: huggingface.co/papers/2608.0…

@__JohnNguyen__ and I are presenting the Beyond Language Modeling paper as an ICML spotlight in 30 minutes at the 10:30 AM poster session! I’ll also be at the AMI Mixer on Thursday and hanging around in Seoul until early next week :)

The video of my Stanford CS25 guest lecture, From Language Models to Native Multimodal Intelligence, is now online. I discussed how the core ideas behind LLMs has shaped multimodal AI, from architectures to training paradigms and scaling, and where the next challenges may lie. 🧠🌐 🎥: youtube.com/watch?v=NDdc39KY…

Part 2: Is 3D really necessary to understand the world? IMO, probably not. I always hear people say that since we live in a 3D world, we must use explicit 3D geometry. But I think we often confuse generating the world with understanding the world. Real understanding can actually be a lot less constrained. Of course, whether you use explicit 3D representations depends on the application and the resources you have. But for what robotics mainly cares about, models should probably have fewer rigid inductive biases. Internet-scale images and videos are already our most diverse sources of world concepts. Here is why I think world understanding shouldn’t bake in too much 3D: - Humans don’t render meshes: As humans, we don’t need explicit 3D information to understand an object. We learn by interacting with the environment through raw sensory streams like touch, sound, and vision. I don’t think there is a clean way to model all these rich interactions just by baking rigid 3D information into a model. Doing so can easily introduce a chain of compounding errors from imperfect inputs. - Scalability: High-quality 3D data is sparse and unlikely to scale. Because of this, these models often suffer from diversity issues. For example, if you post-train a video model by adding camera poses, it may struggle the moment it encounters an unseen/weird trajectory, and the scene can just collapse. - Where does 3D belong: This doesn’t mean 3D is useless. It is still great for maintaining consistent state in many applications, like gaming, but it shouldn’t be the bottleneck for the world model, which should act more like the brain. If we want robust world models for robotics, letting the model learn implicit physics and understand visual concepts and interactions feels much more natural. At the same time, figuring out how to efficiently learn from the real world feels more important now than ever. After all, you can always train a renderer to output pixels, as many other works have shown, if that’s what you care about. But true world modeling should aim for robust representation learning through interactive data with minimal assumptions. Of course, with fewer assumptions, it becomes significantly harder to learn. That’s exactly what I’ve seen from JEPA-style works and my early experience with energy-based models (EBMs) in 2020. But that’s also the challenge worth tackling. Just my 2 cents and not trying to fight anyone and would love to hear your thoughts on this :)

Introducing VGGT-Ω: scaling feed-forward reconstruction across static and dynamic scenes, and studying whether the learned geometric representations transfer beyond reconstruction.

i’m joining forces with @ylecun and an incredible group of people to start AMI Labs @amilabs. AMI isn’t a conventional lab. we don’t intend to become one. a lot to say about why this moment matters, but for now we’re heads down building. join us: amilabs.xyz