PhD student @UTCompSci | Learn to understand ourselves and build intelligence.šŸ¤–šŸ§ šŸ‘ļø

Austin, TX
Joined July 2023
Last year I trained the advantage-conditioned VLA for Ļ€*0.6 during my internship @physical_int. Now, I have a much improved recipe. Introducing ✨Advantage Guidance✨, a performant RL method with stable advantage-conditioned supervised training and contrastive guidance sampling
5
35
5
333
25,357
Yu Lei retweeted
Discovering dexterity: At home with my infant, and at Origami @origami_robotic ... More tomorrow šŸ‘€
17
24
12
139
30,864
While GPT-6-Astra is making everyone anxious, we’ve been working on the System 1 side of robotics: timely responses, continuous motion, and efficient execution. I think a robot’s intelligence also shows in its ability to keep pace with a changing world. Perception, inference, and control operate at different rates and with different latencies. New observations keep arriving, inference takes time, and actions need to keep moving forward. Coordinating these asynchronous streams—so a robot can act smoothly despite latency and adjust as new information arrives—is a problem worth studying carefully. Real-time performance takes work across the whole system. Model capabilities, inference strategies, and execution mechanisms have to come together in a continuous feedback loop. I’d love to see more real-robot experiments and discussion around these questions. Proud of our team for putting over two months of work into ManiMux, now open source. I hope these tools make it easier for more people to deploy their robots and test their ideas. Give it a try—contributions welcome!
Robotic manipulation is not only about success rate. It is also about getting things done efficiently. The real world runs in real time, asynchronously. It does not pause for model inference. Turning model capabilities into smooth, continuous, efficient robot behavior requires the right inference and execution strategies—from handling latency to transitioning between action chunks. From early work on asynchronous inference in Diffusion Policy to π’s @physical_int later work on RTC, the robotics community has paid increasing attention to these challenges. As robot foundation models become available through cloud APIs, coordinating inference and execution becomes even more critical. The release of the Dream-1 API @dreamlabs_ai gave us a timely opportunity to explore this. With several official demos shown at 8Ɨ speed or faster, we wanted to see: How does Dream-1 perform in real time? And how much could running inference and execution asynchronously improve motion continuity and task efficiency? Our team quickly integrated Dream-1 into ManiMux, our in-house framework, and tested it on YAM. The results surprised us: even with seconds of cloud API latency, better asynchronous inference and execution scheduling improved fine-manipulation performance and efficiency. In our block-stacking tests, asynchronous inference doubled Dream-1’s task completion efficiency! Increasingly capable System 2 models are exciting. The System 1 side of robotics—timely responses, continuous motion, and efficient execution—deserves attention too. We want to make these capabilities easier for the community to use, compare, and reproduce. While GPT-6-Astra @OpenAI is making everyone anxious, we want to share a few insights beyond System 2—and make it easier for everyone to build a fast, responsive System 1 for real robots. That is why we are building ManiMux, an open-source inference harness and experiment management platform for real robots, designed to make deployment, inference, and evaluation easier and more reproducible. ManiMux currently integrates policy frameworks including XPolicyLab and StarVLA, inference algorithms including RTC, PAINT, and AutoHorizon, and robot platforms including Tianji, YAM @i2rt_robotics , and AgileX @AgilexRobotics . Support varies across model, algorithm, and robot combinations, and we are continuing to expand real-robot validation and improve usability. To make integration easier, the repo also includes comprehensive agent documentation and Skills. Use your coding agent to quickly integrate your own robots and policies, adapt deployment workflows, and build new features—so you can spend more time experimenting on real robots. We have been developing ManiMux for over two months. We are also collaborating with teams around the world on a systematic comparison of inference algorithms, with more observations and practical recommendations from real-robot experiments to come. Stay tuned. Try ManiMux, open an issue, contribute code, or share your real-robot experience. Let’s make robot experiments more reproducible—and robot execution smoother and faster. Let’s make something meaningful, together. Github: github.com/SII-LiuLab/manimu…
1
2
7
568
Yu Lei retweeted
We built a robotics data engine with Astra-level performance at 125x lower cost. And we are open-sourcing the weights today! Introducing Flora, a suite of open models to parse the physical world and turn raw footage into rich training signals. Flora already powers our in-house data engine, labeling 200K hours of videos a day on 500 GPUs. We created it to scale the frontier for robot learning. Now you own the weights too. Flora gives you: - High-quality, timestamped captions to curate and filter human egocentric and robot data; - Dense progress rewards for quality control and RL post-training; - Rubrics scoring for real-world benchmarking Try it today, links in thread:
128
153
50
1,314
81,794
Yu Lei retweeted
Common belief: Robot policies need continuous action heads; discrete actions are š„šžš¬š¬ š§ššš­š¢šÆšž ššš§š š„šžš¬š¬ š©š«šžšœš¢š¬šž. ā—We find policies w/ the simplest discrete representations š¦ššš­šœš” šØš« š›šžššš­ continuous ones—in sim & real, in single-step & diffusion/flow settings. Here's why 🧵
4
35
6
170
14,182
Replying to @EkaRobotics
@EkaRobotics has raised a $150M Series A. Generality v/s Performance is the fundamental tension at the frontier of robotics. Generality without performance is just a prototype. Performance without generality is just a single-purpose tool. Customers shouldn’t have to choose. Eka breaks the tradeoff. Powered by our Vision-Force-Action (VFA) model, we deliver the performance and versatility customers need to put robots to work every day. Eka is built on three convictions: āš”ļøForce is the language of manipulation. šŸš€ Reinforcement learning is how dexterous manipulation scales. šŸŽ‚ System 1—fast, reliable physical execution—is the bottleneck to solve. With this funding, we will scale our model and accelerate customer deployments. Grateful to be building alongside a dream team and investors who share our ambition.
69
91
60
842
88,398
Can we make dexterous robots compliant without sacrificing precise control? Excited to introduce FOMO-Dex: Force-Modulated Dexterity! šŸ–ļø FOMO-Dex brings force-aware compliance to dexterous teleoperation, enabling robots to pick up thin cards and chips šŸƒ, cut paper šŸ”Ŗ, turn nuts šŸ”©, and use screwdrivers šŸŖ›. These force-aware demonstrations also train better autonomous policies! šŸŽ„ Videos & paper: yingyuan0414.github.io/fomo-… šŸ§µšŸ‘‡ (1/8)
11
40
4
164
13,048
[8/n] By the way, for the simulation folks wondering how we simulated tight-clearance parts: the usual tools didn't work for us. SDF was so slow and hard to tune. Off-the-shelf convex decomposition like V-HACD or CoACD filled the clearance, so parts that should fit jam. So we built mjfit. It decomposes common assembly parts (pegs and holes of any outline, threads, and gears) into convex pieces that keep the clearance exactly as designed, and it runs an order of magnitude faster than SDF. It's open source: github.com/younghyopark/mjfi…
7
32
7
330
44,115
Super excited to share some of the work I’ve been doing at the GEAR lab @NVIDIAAI. After three months, we’re reintroducing ASPIRE: Agentic Skills Discovery for Robotics (v1.0). Enough of Astra ✨ doing pick-and-place—let it learn reusable strategies and solve a Rubik’s Cube online. Quick 🧵 for details.
10
15
6
121
15,242
Yu Lei retweeted
Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents? We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration. A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills. The results: • ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control. • +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro. • One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs. The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning. We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it! Project + sessions: robot-agent-exploration.com/ Paper: assets.robot-agent-explorati… GitHub: N Project fully led by @RunhanH and @xczhangxx who are absolutely awesome. How it works, experiments, and robot videos below 🧵 (1/N)
34
103
31
558
61,911
Yu Lei retweeted
What to remember in the past to act—spatial, temporal or motion details? We often assume what’s useful, but that’s hard to know in advance. Meet IMPRINT 🧠: retain rich memory first, then learn to use it for control. Compact latents keep inference fast as history grows. 🧵 1/10
4
29
11
182
20,891
Replicated the laundry task in mujoco web, and also self-trained the RL WBC. By Opus 5.5 in 3 hrs. Agentic real2sim is nearly solved.
Inspired by the discussion between @qiayuanliao and @wenlong_huang, @kun_h____'s explanation, and insights from Pengxiang and Junfeng. I admire @DynaRobotics' thoughtful choices and look forward to what's next.
31
75
8
732
49,804
DL theory very hard. Try DL alchemy instead!
Deep learning is often taught as a sanitized collection of facts and math. But work at the frontier of AI is messy and empirical. Our new class, Deep Learning Alchemy, aims to teach the art and science of deep learning in the hands-on spirit of Language Modeling From Scratch. Students build intuition by running experiments on small LMs, then test it through prediction quizzes: given an unseen code diff, can you predict what happens? deep-learning-alchemy.github… My co-instructor @kothasuhas and I think there's significant demand for classes that treat students as serious learners who want to be assessed for what they’ve learned, and so we're trying a new exam-heavy approach to the class (7 exams, 85% of the grade, with the remaining 15% in-person discussion assessments). We’ve designed the class so students can use AI agents to run their experiments and do their assignment, but they then have to demonstrate that mastery on exams without access to the assignment, and any partial credit on the exam has to come through citations to the assignment, incentivizing them to know what’s in their write-up. The same assignments and quizzes are public for anyone to try (lecture recordings will be released through @StanfordOnline with some delay). We have some very fun assignments and quizzes covering both classic topics (like NQM, run-to-run stochasticity, and scaling laws) and new and timely topics like muP or hyperball. Made by some real experts like @TristanThrush and @wen_kaiyue Try some quizzes below!
5
12
166
10,227
🤣 DL optimization / theory still very hard We need more complete theory about batch size and we show our humble parts here!
Which optimizer is best? It turns out the answer can change with batch size! We perform a comprehensive sweep and find the best BSZ scaling rule depends on settings and can't be derived from early theories like SDE or convex bound!
2
9
2
142
9,423
Yu Lei retweeted
What’s missing from robot foundation models? An explicit understanding of WHY motions work: why move in this direction, why place the object in this way, how hard to grasp the object. This motion-level reasoning bridges instructions to actions—and provides knowledge that can generalize across embodiments and scenes. Pure behavior cloning supervises WHAT actions to take, but leaves models to infer WHY those actions work. This becomes especially limiting when learning from large-scale heterogeneous data, including egocentric human data, where humans and robots can take very different actions to achieve the same physical effect. We need models to extract the intent behind those motions, rather than simply recite them. Why leave that knowledge buried in trajectories when we can teach it explicitly? Introducing EgoLAP: a shared language for human and robot motion. Language actions teach HOW to move; motion-level reasoning teaches WHY the motion works. Together, they unlock, to our knowledge, unprecedented gains in zero-shot performance for robot foundation models without additional robot demonstrations. arxiv.org/abs/2610.08726 ego-lap.github.io/
13
33
9
140
19,255
I finally baked a long-form audit of ā€œrobot scaling laws.ā€ AI agents did a lot of the digging; I did a lot of checking, rewriting, and arguing with the drafts. I reviewed 19 public results related to robot scaling. The short version: scaling curve āœ… scaling law āŒ Here is the evidence funnel: 19 public results reviewed → 16 report multi-scale curves → 13 keep the main variable and evaluation reasonably controlled → 1 tests a held-out loss forecast → 0 completes the full loop from fitting loss, to forecasting the next scale, to validating that forecast on real robots from the same model family. This distinction matters. In LLM scaling experiments, data volume D is usually measured in tokens. Token count does not tell us how much new information the data contains, but it remains a relatively stable training unit when the tokenizer, data pipeline, and distribution are held fixed. Robotics does not yet have an equivalent common unit. Different studies use recorded hours, trajectories, frames, action steps, or action tokens. Even the same ā€œone hourā€ can produce very different training examples under different sampling rates, action representations, chunking schemes, filtering rules, tasks, environments, and embodiments. The reported objectives and y-axes vary even further: held-out denoising or vector-field MSE, decoded action error, thresholded accuracy, simulated progress, task completion, cycle time, and real-robot success all answer different questions. A few examples: — Figure Helix 2.5 provides the only held-out loss forecast I found: the first three Index-data scales are used to predict offline action loss at 8Ɨ. But its 9%→56% real-robot result comes from a separate with-vs-without-pretraining experiment, not the endpoints of that loss curve. — Dyna-2 scales egocentric human video from 1k to 1M hours. Offline [email protected] moves from roughly 40% to 47%, while post-trained robot performance rises from 20% to 53% on its normalized metric. Scaling helps, but the public results cannot tell us whether the final 900k hours were worth the cost. — Xiaomi-Robotics-1 reports that, with a fixed 5B model, increasing UMI data from roughly 2.5k to 20k hours raises mean success from 53% to 75%. At fixed data, moving from 5B to 10B adds only four points, from 75% to 79%. Useful evidence, but the full model Ɨ data grid is still missing. — Sunday ACT-2 shows that a quality-selected 12.5% subset can outperform a uniformly sampled 50% subset. Nominal scale is not enough: data distribution, repetition, and coverage can matter more than another hour of collection. My current view: A general law spanning tasks, environments, and embodiments will probably arrive later, because robotics still lacks a shared training and validation distribution. But deployment may not need to wait for a predictive scaling law. If a system is reliable enough on a concrete task at an acceptable cost, products may arrive before a general law does. Perhaps robotics is not waiting for a GPT moment, but an iPhone moment: hardware, software, data, and intelligence finally coming together in a product that works reliably in the real world. Full visual audit, original figures, evidence boundaries, and sources: minghuanliu.com/robot_scalin…
5
6
1
37
1,870
I've long believed hierarchical representations are one of the most promising paths forward for vision problem. QuadTok is our first step: a tokenizer for adaptive image representation and generation. Paper: arxiv.org/abs/2610.10497 Code: github.com/myc634/QuadTok Details below.
5
26
4
197
8,128
Now math is doing math. What should humans do is highly puzzling
Math humbles many of us: there is always something deeper. Watching our internal model last month solved so many problems I’ve long stood in awe of has humbled me even further. It feels like the start of a Copernican shift in our view of intelligence… openai.com/index/sharing-ai-…
4
6
59
5,443
We’ve all seen the cool ā€œAstra for roboticsā€ demos. But behind the scenes? Slow rollouts, hand-engineered APIs, and/or simulated behaviors that struggle on real robots. How do we make coding agent actually work in the physical world? 🦾 Introducing SimEX: Simulation-Integrated Robotics AutoResearch 🧵 arxiv.org/abs/2609.38982
11
60
8
295
31,813
Doing proper science in robotics is hard, because human demonstrations are expensive and less controllable. OCBench is a new benchmark providing "human-like" scripted policies. This enables scientific studies of BC and RL with virtually unlimited data!
29
63
8
578
30,487