Pinned Tweet
Your policy doesn't need 7B params. It simply needs dense features.
Introducing Patch Policy: pretrained ViT + small transformer beats OpenVLA-OFT with 0.7% of its params, and trains on a 5090.
Here it inserts a cable (~2mm tol), and does it again as we unplug mid-rollout. 🧵
Jeff Cui retweeted
NAVA-WAM: Can we learn latent actions for World Action Models?
We revisit a classic idea: jointly learning inverse and forward dynamics models while avoiding the identity shortcut (e.g., @SeonghyeonYe, @garridoq_, @jeffacce, @pulkitology, and others).
The twist: Both models are implemented within a single WAM architecture, using attention masks!
To prevent the identity shortcut, the latent actions can only sees a noisy version of the next frame, forcing the latent action to capture useful dynamics rather than simply copying the future.
This simple pretraining strategy improves both performance and efficiency across robotics benchmarks!
🧵👇
🤖 Abundant videos capture how the world works, but robot policies still rely on costly action labels to learn.
🔥 Excited to share our work at @Meta:
📢 Native Action-Prior Learning from Videos for World Action Models 📢
📄arxiv.org/pdf/2610.03391
🌐zhaochongan.github.io/NAVA-W…
🧵
Jeff Cui retweeted
The most profound paper of my PhD so far. We truly did something special to make LMs play/explain chess, needing both architectural & algorithmic innovation (natural-language analogue of the Alphazero algorithm) + applicable to many other domains. Please read on and share!
1/10
Jeff Cui retweeted
Alphazero distills the numerical evaluation produced by NN + search into NN without search. We propose its natural language analog: explain leaves of a search tree w/ LM -> have an LM combine explanations upward -> use root explanations as SFT data for the next iteration.
6/10
Jeff Cui retweeted
me: how do i open pdf
claude: You're right to ask. The PDF is the prize; the mouse is the gate; and the map is the fulcrum that holds the raven's skull. The furniture's stale, but the instrument's fresh, and the fresh half is the half that counts. I deleted your home directory.
Jeff Cui retweeted
Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.
Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵
Jeff Cui retweeted
Scientific terms should have precision. If we use the terms VLM, VLA, WAM in an indiscriminate fashion, as is becoming common in robotics, we are not helping clarity in communication. Let's keep the historical origins of these terms in mind. VLMs arose as multimodal extensions of LLMs-the training was for tasks like VQA (VIsual Question Answering). These capture the static semantics of the scene behind an image. No dynamics. World Models (e.g. @ylecun , Ha & Schmidhuber 2018) on the other hand are primarily dynamics models, which go back to control theory -1960 (Bellman, Kalman etc.) This makes them natural for robotics planning / policies- I am in a state s, what action a should I perform to get to state s'. In classical control, these models were written down a priori by modeling the physics of the system; today we think of them as learned neural networks trained from temporal data e.g. video, robot trajectories. But the concept is the same. We shouldn't mix this concept with VLMs.
Jeff Cui retweeted
Strong multimodal models = strong robot models. Check out some early glimpses of Meta's Muse Spark 1.2 on robots here!
We've released code for Patch Policy! Try it out here: github.com/gaoyuezhou/patch_…
You can run this on a single GPU, as well as on 8xH200 nodes!
Jeff Cui retweeted
Grasping is only the beginning.
To work in the real world, robots must also reorient, align, and precisely manipulate objects after contact.
Simple grippers lack dexterity, while anthropomorphic hands are often complex and hard to control.
Introducing VTAP Gripper. (1/7)
Jeff Cui retweeted
Pretrained ViTs see the world in rich, dense detail. Most policies pool it to a single vector before acting, discarding most of it.
We introduce Patch Policy: a minimal architectural extension that enables transformer-based policies to consume dense tokens directly, no billion-param VLM required. It outperforms a fine-tuned 7B VLA by 18% with ~0.7% of its parameters, enabling robust, precise manipulation.
The recipe is simple: freeze a SOTA vision backbone, feed all patch tokens to a small transformer policy, add a block-causal mask. That's it. No VLM or backbone fine-tuning.
This works with any transformer-based policy (VQ-BeT, Diffusion Policy), and trains on a consumer GPU.
Work co-led with @GaoyueZhou, with @LangfordAd91020, @bowentan_, @ylecun, @LerrelPinto.
arXiv: arxiv.org/abs/2607.18236
Videos: patch-policy.github.io/
Code (coming soon): github.com/gaoyuezhou/patch_…
Robot learning has been either using global-pooled features from ResNet or ViT CLS, or training from scratch, or hauling around a 7B param VLM for policy learning, using its pretraining wholesale.
Turns out all you need is off-the-shelf dense pretrained features.
Jeff Cui retweeted
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data?
Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵