PhD student @UTCompSci | Learn to understand ourselves and build intelligence.šŸ¤–šŸ§ šŸ‘ļø

Austin, TX
Joined July 2023
šŸ¤–Co-training is everywhere (sim↔real[e.g. GR00T, LBM], human↔robot[e.g. PI, EgoScale], even non-robot data[e.g. PI, LBM). But why does it work? How can we improve it further? Taking sim-and-real imitation learning in diffusion/ flow-based models as the test bed, we performed a rigorous mechanistic analysis, drawing on theoretical insights and multi-layered experiments. 😮Key insight: it’s all about representations. - Alignment → enables transfer - Discernibility → enables adaptation āš–ļøBoth are necessary — it's better to have more aligned representations, but the model must be able to discern the domains. We term this as structured representation alignment. ā¬‡ļøLet’s take a deep dive into that: Paper: arxiv.org/pdf/2604.13645 Website: science-of-co-training.githu…
6
64
16
392
68,162
Yu Lei retweeted
Last year I trained the advantage-conditioned VLA for Ļ€*0.6 during my internship @physical_int. Now, I have a much improved recipe. Introducing ✨Advantage Guidance✨, a performant RL method with stable advantage-conditioned supervised training and contrastive guidance sampling
3
25
3
224
13,132
Yu Lei retweeted
Discovering dexterity: At home with my infant, and at Origami @origami_robotic ... More tomorrow šŸ‘€
15
24
12
121
27,096
While GPT-6-Astra is making everyone anxious, we’ve been working on the System 1 side of robotics: timely responses, continuous motion, and efficient execution. I think a robot’s intelligence also shows in its ability to keep pace with a changing world. Perception, inference, and control operate at different rates and with different latencies. New observations keep arriving, inference takes time, and actions need to keep moving forward. Coordinating these asynchronous streams—so a robot can act smoothly despite latency and adjust as new information arrives—is a problem worth studying carefully. Real-time performance takes work across the whole system. Model capabilities, inference strategies, and execution mechanisms have to come together in a continuous feedback loop. I’d love to see more real-robot experiments and discussion around these questions. Proud of our team for putting over two months of work into ManiMux, now open source. I hope these tools make it easier for more people to deploy their robots and test their ideas. Give it a try—contributions welcome!
Robotic manipulation is not only about success rate. It is also about getting things done efficiently. The real world runs in real time, asynchronously. It does not pause for model inference. Turning model capabilities into smooth, continuous, efficient robot behavior requires the right inference and execution strategies—from handling latency to transitioning between action chunks. From early work on asynchronous inference in Diffusion Policy to π’s @physical_int later work on RTC, the robotics community has paid increasing attention to these challenges. As robot foundation models become available through cloud APIs, coordinating inference and execution becomes even more critical. The release of the Dream-1 API @dreamlabs_ai gave us a timely opportunity to explore this. With several official demos shown at 8Ɨ speed or faster, we wanted to see: How does Dream-1 perform in real time? And how much could running inference and execution asynchronously improve motion continuity and task efficiency? Our team quickly integrated Dream-1 into ManiMux, our in-house framework, and tested it on YAM. The results surprised us: even with seconds of cloud API latency, better asynchronous inference and execution scheduling improved fine-manipulation performance and efficiency. In our block-stacking tests, asynchronous inference doubled Dream-1’s task completion efficiency! Increasingly capable System 2 models are exciting. The System 1 side of robotics—timely responses, continuous motion, and efficient execution—deserves attention too. We want to make these capabilities easier for the community to use, compare, and reproduce. While GPT-6-Astra @OpenAI is making everyone anxious, we want to share a few insights beyond System 2—and make it easier for everyone to build a fast, responsive System 1 for real robots. That is why we are building ManiMux, an open-source inference harness and experiment management platform for real robots, designed to make deployment, inference, and evaluation easier and more reproducible. ManiMux currently integrates policy frameworks including XPolicyLab and StarVLA, inference algorithms including RTC, PAINT, and AutoHorizon, and robot platforms including Tianji, YAM @i2rt_robotics , and AgileX @AgilexRobotics . Support varies across model, algorithm, and robot combinations, and we are continuing to expand real-robot validation and improve usability. To make integration easier, the repo also includes comprehensive agent documentation and Skills. Use your coding agent to quickly integrate your own robots and policies, adapt deployment workflows, and build new features—so you can spend more time experimenting on real robots. We have been developing ManiMux for over two months. We are also collaborating with teams around the world on a systematic comparison of inference algorithms, with more observations and practical recommendations from real-robot experiments to come. Stay tuned. Try ManiMux, open an issue, contribute code, or share your real-robot experience. Let’s make robot experiments more reproducible—and robot execution smoother and faster. Let’s make something meaningful, together. Github: github.com/SII-LiuLab/manimu…
1
2
7
491
Yu Lei retweeted
We built a robotics data engine with Astra-level performance at 125x lower cost. And we are open-sourcing the weights today! Introducing Flora, a suite of open models to parse the physical world and turn raw footage into rich training signals. Flora already powers our in-house data engine, labeling 200K hours of videos a day on 500 GPUs. We created it to scale the frontier for robot learning. Now you own the weights too. Flora gives you: - High-quality, timestamped captions to curate and filter human egocentric and robot data; - Dense progress rewards for quality control and RL post-training; - Rubrics scoring for real-world benchmarking Try it today, links in thread:
110
126
47
1,132
68,719
Yu Lei retweeted
Common belief: Robot policies need continuous action heads; discrete actions are š„šžš¬š¬ š§ššš­š¢šÆšž ššš§š š„šžš¬š¬ š©š«šžšœš¢š¬šž. ā—We find policies w/ the simplest discrete representations š¦ššš­šœš” šØš« š›šžššš­ continuous ones—in sim & real, in single-step & diffusion/flow settings. Here's why 🧵
3
33
5
150
11,478
Replying to @EkaRobotics
@EkaRobotics has raised a $150M Series A. Generality v/s Performance is the fundamental tension at the frontier of robotics. Generality without performance is just a prototype. Performance without generality is just a single-purpose tool. Customers shouldn’t have to choose. Eka breaks the tradeoff. Powered by our Vision-Force-Action (VFA) model, we deliver the performance and versatility customers need to put robots to work every day. Eka is built on three convictions: āš”ļøForce is the language of manipulation. šŸš€ Reinforcement learning is how dexterous manipulation scales. šŸŽ‚ System 1—fast, reliable physical execution—is the bottleneck to solve. With this funding, we will scale our model and accelerate customer deployments. Grateful to be building alongside a dream team and investors who share our ambition.
63
81
55
769
74,461
Can we make dexterous robots compliant without sacrificing precise control? Excited to introduce FOMO-Dex: Force-Modulated Dexterity! šŸ–ļø FOMO-Dex brings force-aware compliance to dexterous teleoperation, enabling robots to pick up thin cards and chips šŸƒ, cut paper šŸ”Ŗ, turn nuts šŸ”©, and use screwdrivers šŸŖ›. These force-aware demonstrations also train better autonomous policies! šŸŽ„ Videos & paper: yingyuan0414.github.io/fomo-… šŸ§µšŸ‘‡ (1/8)
11
40
4
163
13,005
[8/n] By the way, for the simulation folks wondering how we simulated tight-clearance parts: the usual tools didn't work for us. SDF was so slow and hard to tune. Off-the-shelf convex decomposition like V-HACD or CoACD filled the clearance, so parts that should fit jam. So we built mjfit. It decomposes common assembly parts (pegs and holes of any outline, threads, and gears) into convex pieces that keep the clearance exactly as designed, and it runs an order of magnitude faster than SDF. It's open source: github.com/younghyopark/mjfi…
7
31
7
328
43,505
Super excited to share some of the work I’ve been doing at the GEAR lab @NVIDIAAI. After three months, we’re reintroducing ASPIRE: Agentic Skills Discovery for Robotics (v1.0). Enough of Astra ✨ doing pick-and-place—let it learn reusable strategies and solve a Rubik’s Cube online. Quick 🧵 for details.
10
15
6
121
15,229
Yu Lei retweeted
Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents? We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration. A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills. The results: • ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control. • +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro. • One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs. The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning. We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it! Project + sessions: robot-agent-exploration.com/ Paper: assets.robot-agent-explorati… GitHub: N Project fully led by @RunhanH and @xczhangxx who are absolutely awesome. How it works, experiments, and robot videos below 🧵 (1/N)
34
102
31
552
59,891
Yu Lei retweeted
What to remember in the past to act—spatial, temporal or motion details? We often assume what’s useful, but that’s hard to know in advance. Meet IMPRINT 🧠: retain rich memory first, then learn to use it for control. Compact latents keep inference fast as history grows. 🧵 1/10
4
29
11
182
20,868
Replicated the laundry task in mujoco web, and also self-trained the RL WBC. By Opus 5.5 in 3 hrs. Agentic real2sim is nearly solved.
Inspired by the discussion between @qiayuanliao and @wenlong_huang, @kun_h____'s explanation, and insights from Pengxiang and Junfeng. I admire @DynaRobotics' thoughtful choices and look forward to what's next.
31
75
8
732
49,778
DL theory very hard. Try DL alchemy instead!
Deep learning is often taught as a sanitized collection of facts and math. But work at the frontier of AI is messy and empirical. Our new class, Deep Learning Alchemy, aims to teach the art and science of deep learning in the hands-on spirit of Language Modeling From Scratch. Students build intuition by running experiments on small LMs, then test it through prediction quizzes: given an unseen code diff, can you predict what happens? deep-learning-alchemy.github… My co-instructor @kothasuhas and I think there's significant demand for classes that treat students as serious learners who want to be assessed for what they’ve learned, and so we're trying a new exam-heavy approach to the class (7 exams, 85% of the grade, with the remaining 15% in-person discussion assessments). We’ve designed the class so students can use AI agents to run their experiments and do their assignment, but they then have to demonstrate that mastery on exams without access to the assignment, and any partial credit on the exam has to come through citations to the assignment, incentivizing them to know what’s in their write-up. The same assignments and quizzes are public for anyone to try (lecture recordings will be released through @StanfordOnline with some delay). We have some very fun assignments and quizzes covering both classic topics (like NQM, run-to-run stochasticity, and scaling laws) and new and timely topics like muP or hyperball. Made by some real experts like @TristanThrush and @wen_kaiyue Try some quizzes below!
5
12
166
10,223
🤣 DL optimization / theory still very hard We need more complete theory about batch size and we show our humble parts here!
Which optimizer is best? It turns out the answer can change with batch size! We perform a comprehensive sweep and find the best BSZ scaling rule depends on settings and can't be derived from early theories like SDE or convex bound!
2
9
2
142
9,419
Geometry <—> Local alignment convergence. I think this should be the same across different domains, modalities of data.
DINOv2 has never seen a caption, and Qwen3 has never seen an image. We still aligned their embedding spaces without a single image-caption pair. It even works when the images and the captions come from different datasets. Project page: dominik-schnaus.github.io/un… ā¬‡ļø
2
14
1,357
Yu Lei retweeted
What’s missing from robot foundation models? An explicit understanding of WHY motions work: why move in this direction, why place the object in this way, how hard to grasp the object. This motion-level reasoning bridges instructions to actions—and provides knowledge that can generalize across embodiments and scenes. Pure behavior cloning supervises WHAT actions to take, but leaves models to infer WHY those actions work. This becomes especially limiting when learning from large-scale heterogeneous data, including egocentric human data, where humans and robots can take very different actions to achieve the same physical effect. We need models to extract the intent behind those motions, rather than simply recite them. Why leave that knowledge buried in trajectories when we can teach it explicitly? Introducing EgoLAP: a shared language for human and robot motion. Language actions teach HOW to move; motion-level reasoning teaches WHY the motion works. Together, they unlock, to our knowledge, unprecedented gains in zero-shot performance for robot foundation models without additional robot demonstrations. arxiv.org/abs/2610.08726 ego-lap.github.io/
13
33
9
140
19,248
I finally baked a long-form audit of ā€œrobot scaling laws.ā€ AI agents did a lot of the digging; I did a lot of checking, rewriting, and arguing with the drafts. I reviewed 19 public results related to robot scaling. The short version: scaling curve āœ… scaling law āŒ Here is the evidence funnel: 19 public results reviewed → 16 report multi-scale curves → 13 keep the main variable and evaluation reasonably controlled → 1 tests a held-out loss forecast → 0 completes the full loop from fitting loss, to forecasting the next scale, to validating that forecast on real robots from the same model family. This distinction matters. In LLM scaling experiments, data volume D is usually measured in tokens. Token count does not tell us how much new information the data contains, but it remains a relatively stable training unit when the tokenizer, data pipeline, and distribution are held fixed. Robotics does not yet have an equivalent common unit. Different studies use recorded hours, trajectories, frames, action steps, or action tokens. Even the same ā€œone hourā€ can produce very different training examples under different sampling rates, action representations, chunking schemes, filtering rules, tasks, environments, and embodiments. The reported objectives and y-axes vary even further: held-out denoising or vector-field MSE, decoded action error, thresholded accuracy, simulated progress, task completion, cycle time, and real-robot success all answer different questions. A few examples: — Figure Helix 2.5 provides the only held-out loss forecast I found: the first three Index-data scales are used to predict offline action loss at 8Ɨ. But its 9%→56% real-robot result comes from a separate with-vs-without-pretraining experiment, not the endpoints of that loss curve. — Dyna-2 scales egocentric human video from 1k to 1M hours. Offline [email protected] moves from roughly 40% to 47%, while post-trained robot performance rises from 20% to 53% on its normalized metric. Scaling helps, but the public results cannot tell us whether the final 900k hours were worth the cost. — Xiaomi-Robotics-1 reports that, with a fixed 5B model, increasing UMI data from roughly 2.5k to 20k hours raises mean success from 53% to 75%. At fixed data, moving from 5B to 10B adds only four points, from 75% to 79%. Useful evidence, but the full model Ɨ data grid is still missing. — Sunday ACT-2 shows that a quality-selected 12.5% subset can outperform a uniformly sampled 50% subset. Nominal scale is not enough: data distribution, repetition, and coverage can matter more than another hour of collection. My current view: A general law spanning tasks, environments, and embodiments will probably arrive later, because robotics still lacks a shared training and validation distribution. But deployment may not need to wait for a predictive scaling law. If a system is reliable enough on a concrete task at an acceptable cost, products may arrive before a general law does. Perhaps robotics is not waiting for a GPT moment, but an iPhone moment: hardware, software, data, and intelligence finally coming together in a product that works reliably in the real world. Full visual audit, original figures, evidence boundaries, and sources: minghuanliu.com/robot_scalin…
5
6
1
37
1,864
I've long believed hierarchical representations are one of the most promising paths forward for vision problem. QuadTok is our first step: a tokenizer for adaptive image representation and generation. Paper: arxiv.org/abs/2610.10497 Code: github.com/myc634/QuadTok Details below.
5
26
4
197
8,128
Thus spoke.
looking up at the stars with extra awe tonight. "thy sea is so great and my boat is so small."
2
319
Now math is doing math. What should humans do is highly puzzling
Math humbles many of us: there is always something deeper. Watching our internal model last month solved so many problems I’ve long stood in awe of has humbled me even further. It feels like the start of a Copernican shift in our view of intelligence… openai.com/index/sharing-ai-…
4
6
59
5,442