Where humans, robots, and data come together to advance intelligence. 🦾 Backed by @a16z. Host of @RoboCon_AI 🤖 🔗https://xitter.cf/t.co/YsgmtlbLTP
San Francisco, CA
Joined July 2024
- Tweets3.8K
- Following22
- Followers53.5K
- Likes2.5K
Pinned Tweet
Hard lighting cut our expert-only π0.5 checkpoint's success roughly in half. The same lights cost the full fine-tune nothing.
New from @castorhat: the fine-tuning ablations behind our IROS demo.
We nudged the cameras on a π0.5 policy a few degrees and a few centimeters at a time.
Lifting barely changed. Placing fell apart.
What @castorhat's fine-tuning ablations found about the most overlooked part of a robot rig:
The fix is boring: lock the mounts, match camera models and field of view, and get the rig CAD and camera specs from whoever collected your data.
Every rig in our fleet ships with an individually measured URDF and cameras you can buy anywhere.
Full ablations, plus the open-source simulator we ran them in:
blog.prismax.ai/article.html…
We built a path-traced YAM simulator to run thousands of π0.5 rollouts a day without resetting a scene.
It's open source now, so you can run the same ablations on your own rig: github.com/PrismaXAI/yamulat…
Hard lighting cut our expert-only π0.5 checkpoint's success roughly in half. The same lights cost the full fine-tune nothing.
New from @castorhat: the fine-tuning ablations behind our IROS demo.
If you buy data for post-training, ask for the rig specs: full CAD, the teleop control law, and camera SKUs you can reproduce.
Next in the series: the latency bug that was quietly costing us accuracy.
We also took the walls off the cell and put a photo of a conference hall behind it. The LoRA checkpoint went from 84% to 89%.
That's why we were comfortable facing the arms out into the IROS floor.
Cameras matter more than people think. Moving the overhead camera barely changes whether the robot lifts the target. It changes whether it places it.
A 1 cm lateral shift of the wrist camera made 11 of 30 paired rollouts worse.
Training scheme first. π0.5 from @physical_int has a big backbone and an expert 10x smaller, so training only the expert is tempting.
Expert-only: 51% pickup, 12% dropoff.
Full fine-tune: 89% and 53%.
r=128 LoRA: 84% and 59%, and it fits on a 32 GB card.
Under the hood: MuJoCo contacts, a port of the @I2RTRobotics control law, and a path-traced renderer with physically based lights. The sim scores a little higher than the real rig, but the trends hold.
Demo is not deployment.
From Lab to Deployment, the full RoboCon panel with @sharonal_lee of @moonlake, @vivekgr92 of @agilityrobotics, and @castorhat, is now live.
Sim vs. real, models that fit on the robot, and what "good" means to a customer.
ALT From Lab to Deployment
Physical Intelligence made fine-tuning a frontier robot model an API call.
The loop they describe: collect demonstrations, train, test on the robot, collect again. The compute is handled. The demonstrations are still the input.
We've been building for this loop. More soon.
