You should make it more clear that these are not the results of the base released policies, but a fine-tuned version of them
we released a dataset previously that is within-domain to our benchmark (huggingface.co/collections/a…). but others got such results with Astra in real-world evals so I wouldn't attribute the performance to this dataset existing
they can also just add them to the pool of rl envs they use for post-training. maybe they already do that
Omar Rayyan retweeted
Are we experiencing a GPT moment in robotics? Our MolmoSpaces benchmark seems to suggest so. Check out @omarrayyann's thread to see the details 👇
GPT-Astra outperformed all open-source VLA/WAM baselines on a subset of our zero-shot MolmoSpaces-v1 benchmark! traces: orayyan.com/ms-results
I think distilling the model’s spatial knowledge into a raw action space policy would be an interesting direction. its good vision understanding makes this verifiable enough to happen in an automated closed-loop way
It used fewer reasoning tokens than other VLM APIs (though the counts might not be directly comparable). I couldn’t run the full benchmark for these API models to include them on our leaderboard (molmospaces.allen.ai/leaderb…) because of the API cost :)
All models used the same closed-loop harness in @robocurve’s results. they get exocentric/egocentric rgb observations from the DROID setup and robot state, and output cartesian poses. history is cleared after each of the 200 episodes.
GPT-Astra outperformed all open-source VLA/WAM baselines on a subset of our zero-shot MolmoSpaces-v1 benchmark! traces: orayyan.com/ms-results
Are you learning the noise of size action_dim*chunk_size? If so, 40 episodes might be too little for that. For real-world experiments, they learn a noise of size action_dim and tile it across the chunk dimension. Should be more sample efficient (assuming you have no bugs) given the simple pick task you have.
Omar Rayyan retweeted
A solid step toward humanoid loco-manip, super cool!
Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.
Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵
Omar Rayyan retweeted
Learning mobile manipulation is hard because collecting data for mobile manipulation is hard. But seems like @omarrayyann has found a promising way to crack this problem!
Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.
Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵