Postdoc @berkeley_ai. Researcher @IBMResearch. PhD @TelAvivUni. Working on Multimodal Robot Foundation Models and Structured Physical AI.
Berkeley, CA
Joined March 2017
- Tweets2.6K
- Following939
- Followers2.1K
- Likes8.9K
Pinned Tweet
If you want a vision encoder for dexterous manipulation, what should be the most important part to model? 🤔
Current standard models like CLIP, SigLIP, and DINOv2 have an incredible grasp of semantics and spatial details. But they lack the action-centric structure needed for downstream visuomotor control.
But collecting annotated robotic trajectories at scale is SUPER expensive and largely unrealistic. So, how do we bridge this gap?
We introduce CAIP (Contrastive Action-Image Pre-training) ⬇️
🔸 Action-centric upstream: we align visual observations with action chunks through a contrastive objective.
🔸 Human video as a proxy: we represent 3D human hand poses analogously to robotic end-effector actions, tapping into a massive source of human demonstrations.
🔸 Massive scale: pre-trained on over 32,000 hours of manipulation video, driving both sample efficiency and robust generalization.
🔸 Hardware proven: achieves a 76% average success rate on a real-world Dexmate Vega bimanual @DexmateAI manipulator with dual 22-DoF Sharpa Wave hands @SharpaRobotics .
🔸 State-of-the-art: significantly outperforms strong baselines like DINOv2, SigLIP, MVP, and Qwen3.5 ViT across complex tasks, even under unexpected lighting changes and visual distractors.
🌐 Project: caip-encoder.github.io/
📝 Blog: xitter.cf/roeiherzig/status/2063…
📄 Paper: arxiv.org/abs/2606.17256
💻 Model: huggingface.co/yuvansharma/c…
Playing for the win! Nice work 🥳
We taught a robot to play!
Introducing Dexterity from Supervised Play (DSP): turning minutes of human play into a world model and dexterous manipulation policy. No task demonstrations or sim2real!
play-policy.github.io
That’s not true. Capability should come with efficiency.
Have you seen a kid grasping all the objects in the world before he manipulates a new object?
Like compression in information theory, once you’re data-efficient, you’ve captured the structure that leads to generalization.
Welcome back!!
Announcing Gemini 4 Argon, our new frontier model.
Argon is built to sustain deep reasoning across complex, long-horizon workflows and delivers frontier performance in complex workflows across real-world software engineering, enterprise knowledge work like legal and finance, and cybersecurity defense.
We’re also expanding the model’s output token limit to an industry-leading 1M tokens.
Argon is currently rolling out to a set of trusted cyber defenders in the Fairwind Program, with broader availability as soon as possible.
Amazing opportunity to continue pushing the robotic harness frontier! Don't miss it. There are very few places and labs where you can do it.
If you're at #IROS26, completing your PhD soon, and interested in a Postdoc position to collaborate with me and my students on Agentic Robotics for bimanual manipulation in the AUTOLab @UCBerkeley, please come talk with us at the Bistro near the Westin Hotel Registration Desk between 7-8pm tonight! We're particularly interested in PhDs with skills in model-based methods, control theory, computer vision, and motion planning who are excited to work with frontier LLM and VLM models!
Maybe just Findings?
Maybe computer vision conferences need something like an "AI methods" track. This would not be a traditional paper -- it would describe how you leveraged a frontier model to solve an outstanding problem.
The publication would describe the problem, give the prompts, resulting code, demo, etc. This would allow the field to say "ok, that problem is solved sufficiently well" and move on.
This acknowledges something that's been bothering reviewers for a long time. Reviewers feel papers require technical novelty. If one uses a frontier model, there may not be anything technically interesting. But the fact that an open problem is solvable, and how you solve it, is interesting.
I would not have a full review process for such things. An abstract and a link to a demo might be enough. People would get a poster or demo space to show what they did.
I can imagine that practical sessions where people get insights about how to solve problems today could be interesting.
Just a thought.
Congrats to everyone 🎉
We are excited to announce that World Labs is joining @AMD.
The research and technical breakthroughs we have achieved since our founding in 2024 have given us a clear vision for AI’s potential to solve problems in the spatial and physical world.
Accelerating the future of spatial and physical intelligence requires scaling our efforts, scaling our reach, and getting closer to the hardware.
Best Sukkah ever!
Na I can’t do this anymore
Readers added context they thought people might want to know
The photo is AI-generated. Replies and quotes identify it as such; no records exist of this exact sponsored sukkah event.
x.com/_djpn/status/2…
x.com/MattHasTheShoe…
forward.com/culture/520812…
techcrunch.com/2026/08/04/ben…
If Codex is down, does #ICLR allow a 1-day grace period?
These days, it's worse than if OpenReview is down.
I've been playing with @Muse recently, but still haven't figured out the best way to use it.
What are the good use cases people are actually using it for?
For example, it refuses to control my WhatsApp and send automatic messages, but it sends on Instagram, which is really not useful.
Every few years the field agrees the review system is broken. Then submissions double. ICLR just passed 50,000 🤯
At some point we have to cap how many papers an author can submit ✋📄...Nobody has more than 5 conference-worthy ideas in a single cycle.
🚨 Call for participation: the Child–Robot Dexterity Challenge at the CoRL 2026 𝗦𝗣𝗜𝗡 𝗪𝗼𝗿𝗸𝘀𝗵𝗼𝗽! Bring your robot, or run on ours. Apply by Oct 5.
🧒🤖 Robots and a child will take on the same everyday tasks, side by side, live on stage. The demonstration offers a simple way to see what today’s robots can already do and where there’s still room to improve.
We make it easy to join:
- Bimanual YAM setup (MolmoAct 2 style) + a dexterous Sharpa hand setup, provided on stage.
- Bring your own hardware (subject to approval).
🌐 Website: spin-workshop.github.io/#con…
🗣️ Speakers & panel from Stanford, Columbia, TU Darmstadt, and AMI Labs.
📍 Nov 12 · Austin, TX
#CoRL2026
Possible tasks: grasping, opening bottles, insertion, folding, and pouring. Every task will be child-appropriate.
We’ll share the final list and completion guidelines in advance.
Academic and industry teams are welcome. Bring approved hardware, use a provided setup, or contribute hardware, data, or sponsorship.
Apply by Oct 5: [email protected], [email protected] @_amirbar , or [email protected]. @carlo_sferrazza
Fun fact: fewer than 0.8% of researchers are one coauthorship hop from Jitendra Malik 😀
alphaxiv.org/number/@jitendr…
