A new piece of research today by @wangyenjen, @jiaman01, @pabbeel et al. Called VLK, for Vision-Language-Kinematics, it is a new model that shows it is possible to learn humanoid loco-manipulation from synthetic scenes. Here is what it means in practice. The team teaches a Unitree G1 humanoid to walk to, pick up, carry, and place objects from a first-person camera plus a spoken instruction All this trained entirely on synthetic data, with zero real-robot data collection (no teleoperation etc.). Here is what the pipeline looks like: - scan a room with an iPhone (Polycam) into a metric-scale 3D Gaussian-Splatting replica (overly simplified: a fast low compute 3D model) - synthesize whole-body G1 motions with conditional diffusion - replay this synthetic data them in Isaac Sim to render egocentric frames with randomized lighting/appearance (i.e. domain randomization in simulator) - fine-tune a VLA to predict a 1-second future kinematic trajectory at 30 fps A separate contact-aware whole-body tracker (SceneBot) then turns those trajectories into joint commands at 50 Hz on the real robot. Total data: 48,000 synthetic trajectories across 8 scanned rooms, evaluated on real hardware with no fine-tuning. Really impressive. What I find really interesting about this approach, is that it could replace teleoperation entirely. No human ever moves the robot or wears a mocap suit. If you compare it to China's playbook (Xiaomi's 100k UMI hours, JD paying collectors ~$3/hr), VLK's entire corpus is super cost effective. Now remains the question: can it scale efficiently and generalize to more complex tasks (dexterous, for example)? Also worth mentioning: this is a strong win for domain randomization. Randomizing lighting beats collecting more data, by far: - no randomization 41% walking success rate - full randomization 90% - lighting-only already gets 87% - camera-jitter-only gets just 48%. Randomized lighting in the renderer costs nothing and is unreasonably effective! Is most of the sim-to-real gap here is a rendering-appearance problem, and not a physics problem? A positive answer to this question would shift the entire robotics data acquisition race happening right now. Last learning from this piece of research: predicting a whole second at once with the VLA/VLK solves the real-time control problem that happens when a single action is predicted instead. Outputing a plan and letting the robot controller take care of the low-level dynamics seems like a much more efficient approach. I hope you enjoy this video as much as I did, where the robots adapts in real time to a changing environment: