Jongwon πŸ‡°πŸ‡· @ CoRL (@jdubpark) header
Jongwon πŸ‡°πŸ‡· @ CoRL (@jdubpark) profile picture

Head of Eng @PsdnAI / UIUC CS / wanna-be AI frog

Seoul
Joined May 2020
Waiting for some viral posts of Astra and Fable playing GTA 6 as "emergent" behavior
59
Quest: Ego video collection in GTA 6 πŸ˜‚
85
Jongwon πŸ‡°πŸ‡· @ CoRL retweeted
Basically we are at a point now where what makes the programmer is the ability to think at a coherent design and to have an idea of the software she/he wants to build. It's a large shift. Recently I read an article by an Italian thinker (Andrea of Tlon edizioni)
7
9
2
279
14,507
CoRL takeaway: Robots are now fully accessible... deploying is next. With off-the-shelf hardware and agentic robotics (Astra), you can put together a working robot. It will be slow and make mistakes. But this turns the question from β€œcan anyone build an e2e robot pipeline?” to β€œhow do you improve one?” The race is now on to get them working in the field. So the visible moat is deployment knowledge. WAMs/VLAs, sim, hardware, policies, etc. are all being commoditized. And every architecture and method chases the same goal: robots in prod at scale. Ofc, to get there you need a closed feedback loop that includes deployment, like Optimus in the Gigafactory or Amazon FAR in warehouses. Deployment is the back-propagation for robotics. Despite the accessibility, the gap between research, hardware, and real-world usage remains wide. Walden Robotics founder @RussTedrake's keynote impressed many with β€œdeployment data is the key” and a demo on a Toyota factory line. Tbh, that’s common knowledge. But in our minds, the path from software to robots working in production lines is still very murky. And this conference is a great wake-up call for roboticists and companies to be more practical -- get your hands dirty in factories and homes, break things and learn to recover in prod. Perfection matters more than generalization in most deployments - failure is unacceptable. Agentic robotics is emerging fast! Astra and Fable can control robots with no extra training decently (pre-trained on ego videos :p). It’s slow for now, so most people treat it as long-horizon planning. The stack is settling into System 2 (what to do), System 1 (how to do it - agentic robotics), and System 0 (real-time actions). More notes: Progress to a functioning robot in the wild depends on connecting three capabilities: - Learning from human demonstrations - Reacting to contact and the environment with hands and whole body - Retargeting data & skills across different hands and bodies Human video is becoming a source of executable demonstrations, not just for training for generalization. EgoScale, Do as I Do, and Video2Sim2Real all convert video into actions a robot can execute. Lots of imitation learning alongside RL (though it's doubtful whether it's better than RL for prod). Deployment data is the gold standard, since everything in robotics exists to make deployment work. Walden’s 4.7B β€œSage” model trained on 100k hours from a Toyota line and performs very well on that task (unsure for other tasks). Teleop interventions catch failures and record them to improve the model. Synthetic data is growing too, ultimately for real2sim2real: bring reality into sim, train and test at scale, transfer back. GRAIL, Play2Perfect, etc. Dexterity got a lot of attention. T-Rex used 100 hours of tactile data to sharply raise task success. Some papers drop vision and rely on touch alone... but again, why? Feels very detached from practicality. Unlike Walden, tactile research is still far from industry deployment.
1
2
1
9
1,346
Most data collection starts with what’s easy to collect. At @PsdnAI we start with what models get wrong. We care about how data improves the models. (1) Create a holistic benchmark (2) Find where models & APIs fail (3) Run collection campaigns across high-value channels for that data (4) QC and annotate (5) Fine-tune, improve, & benchmark OSS models on that data -> loop We’re now taking the same loop to Physical AI. We care about how ego video, audio, and tactile data improve models and robots in the wild. DM me for more info on how you can get involved and benefit from this loop!
Better data is the breakthrough. Today, we're publishing a case study on how our 10,000-hour Bengali speech dataset reduced Character Error Rate (CER) by ~32% on Whisper Large V3. ↓
2
7
347
Jongwon πŸ‡°πŸ‡· @ CoRL retweeted
Swarms are quickly becoming one of the most important topics in Silicon Valley, sharing a few notes from recent conversations: - The naive view of swarms is that they’re a sum-of-parts method of parallelization / brute force, but we’re empirically seeing the β€œshape” of swarm intelligence be very different. - The way swarms achieve this is through a combination of parallelization, entropy, and specialization, leading to emergent solutions that aren’t covered by a small number of brilliant frontier models acting as central planners. - It’s also not obvious that the economics of swarms follow higher cost = better output. Swarms that mix and match frontier and non-frontier models with different reasoning levels may be more economical than a small number of frontier agents, and broadly, there should be a new Pareto curve here. - Swarms will be orders of magnitude more token-consumptive than agents, just as agents were compared to humans + models, further driving constraints in inference supply. I think it’s important to build intuition here so have been running some experiments that I’ll share soon. A great way to get started with swarms is the /swarms command in the new google anti-gravity. A few of the smartest thinkers on this topic are @alehandromz @jsnnsa @gokhan_egri @pirroh @amasad Love to hear other POVs on swarms if folks have them!!
Swarm riders and Swarm-native companies AI-use even among highly AI-native people is power-law distributed. There seems to be a new class of people β€” call them β€œswarm riders” β€” who are capable of consistently commanding large swarms of agents to productive ends. This is a new kind of managerial skill, analogous to the skill it takes to manage large organizations of people, but wholly different in nature and of course more technical. Based on the swarm riders we know (including many at @Tempo and @Paradigm), these people are capable of productive token spend that is 1000x-10000x higher than median software engineers, and 100x higher than top 5% Claude Code users. The art of swarm riding is different from software engineering and more akin to machine learning, where practitioners grapple with black box systems: defining reward functions and evals, transmuting non-verifiable problems into verifiable problems, running experiments, etc. Swarm-native companies are a new class of company founded by and populated by swarm riders. These companies are organized around swarms in a way that conventional companies will struggle to replicate, as the cost of internal human coordination is sand in the gears of an effective swarm. Effective swarm-native companies will define the right reward functions and evals, remove functional and bureaucratic barriers, and then let the swarms cook. We’ve gotten to see a few swarm-native companies up close β€” reshaping financial rails @Tempo, building a new gaming platform @Spawn, cyber swarms at Cathedral Systems β€” and it’s clear these teams wield a significant advantage over pre-AI teams. Some questions I’ve been wondering about: 1) How fast will swarm riding diffuse through the economy? The implications for token spend could be significant. 2) Can swarms be effectively adopted by existing companies or will it generally track the formation of new swarm-native companies? 3) What becomes scarce as swarm capabilities themselves commoditize (through frontier models becoming better at swarming, swarm capabilities built into harnesses, etc)? Cultures and teams that can define reward functions well and otherwise get out of the way?
69
41
20
634
60,506
Data center construction is heavily bottlenecked by labor, doing so will make compute hilariously autonomous. That's in addition to designing chips using AI. A lot of closed loop coming.
Sam Altman says OpenAI will aim its robotics program at building its own data centers. Speaking on Stripe co-founder John Collison’s Cheeky Pint podcast, he said self-replicating data centers are now β€œwithin grasp”: AI-controlled robots running chip factories and building turbines, solar panels and more data centers, to produce β€œa lot of really cheap compute.” He argued robots could also make the parts that permits and supply chains hold up today, and said OpenAI already has its own chip and is learning to build its own data centers. On the cost of AI, Altman said it takes about $70 billion to build and run one gigawatt of capacity for five years, and that $20 billion would make him β€œmuch happier.”
2
121
State of the art in robotics LLMs are so good on existing tasks bc there's no ood anymore If you collect all factory work as data then your robot would work so much better, just fix hands and locomotion
1
119
If AMD "hires" Jensen, it's game over (ultimate family business)
1
210
We will open source more data and research as we expand our capacity. For researchers at CoRL, reach out for various data! Happy to collaborate on improving models with globally diverse datasets
Today, we're releasing EgoCrowd-500H, a new dataset accompanying our @corl_conf 2026 research on quality failures in crowdsourced egocentric video. β€’ 300 stereo clips β€’ 2.5 hours of footage β€’ 50 human-reviewed clips Out now on @huggingface.
1
4
14
553
Continuous feedback during motion is a must. @JitendraMalikCV says we must RL as if a parent teaches a child atomic skills with video and tactile. Agentic robotic is a high-level policy to use fast "tools" - low-level atomic skills with semantic orientation.
1
1
17
1,558
The theme is "dexterity" at CoRL. Most booths are humanoid hands and robot arms! Cool hands from @origami_robo. @PsdnAI is researching visuo-tactile WAMs with own video and tactile data. Reach out!
4
4
2
37
6,516
Jongwon πŸ‡°πŸ‡· @ CoRL retweeted
We're honored to present research at @corl_conf. Our team studied 500 hours of crowdsourced egocentric video to better understand where data collection goes wrong, and how to improve quality control for Physical AI. One key finding: 90% of footage in flagged clips falls outside labeled failure intervals. Rejecting entire clips can mean discarding a lot of potentially useful data. Tremendous work by @iamrashminagpal, @sebliu, @jdubpark, and Sandeep Chinchali. First look at our research poster below. Catch us tomorrow at the 'Oops, I Erred' #CoRL2026 workshop ↴
15
6
1
36
3,594
Jongwon πŸ‡°πŸ‡· @ CoRL retweeted
Fable 5.2 shows substantially improved skills in controlling physical robots, even on tasks it has never explicitly been trained to do. On RoboDojo, a benchmark with a simulated two-arm robot, it outperformed other models and most AI systems built to control robots.
228
1,071
860
10,691
3,454,099
I see laptops and coding on every seat on Starlink-equipped flight SF to Austin right now πŸ˜‚
3
194
Jongwon πŸ‡°πŸ‡· @ CoRL retweeted
MuJoCo 4.0 from Google DeepMind is the first major version of the physics engine since 3.0, three years ago. Its headline feature, speculative contacts, keeps thin and fast objects from passing through each other, which allows much larger timesteps. The problem β€’ Collision detection checks geom poses only at discrete timesteps. A thin or fast object can move further than its own thickness in one step and tunnel through, so the usual fixes are a smaller timestep or an inflated margin. How it works β€’ Speculative contacts: each fast-moving geom is swept forward through the step to find when it would hit something. The solver gets a contact like any other that stops the motion at the surface, with no sub-stepping. β€’ Deformables are now solved implicitly together with rigid bodies, joints and tendons, which keeps cables, cloth and shells stable at 5-10x larger timesteps. New are penetration-free cloth and elastic rods with bending and twist. β€’ Plasticity: wires, clay, folded paper and metal foil now keep their dents and creases. Does it pay off? β€’ In the team's tests, an arm catches a 0.7 mm playing card falling at over 2 m/s with a 16 ms timestep every time, where discrete collision detection misses it above 8 ms. β€’ Humanoid stair climbing and thin-object grasping run 6 to 9 times faster, because they can use larger timesteps. Also in 4.0 β€’ MuJoCo Studio, a new viewer SDK with physically based rendering. The same tools run in a browser tab with nothing to install. β€’ MuJoCo Playground 1.0 ships alongside. Caveats β€’ The speedups and timestep factors are the team's own measurements, without benchmark details in the announcement. β€’ Speculative contacts predict motion from pre-step velocities, so a slow geom pushed across an obstacle within one step can still tunnel. They cover convex primitives and meshes on CPU. Heightfields, SDFs, flexes, MJX and MuJoCo Warp are listed as coming next. β€’ Breaking changes: composite bodies, the cable and shell plugins and flexvert equalities are removed, Python 3.10 is dropped, and 3.x installs need a pip uninstall first. By @yuvaltassa and the Google DeepMind Robotics Simulation Team.
6
28
4
413
20,851
Jongwon πŸ‡°πŸ‡· @ CoRL retweeted
GPT-6 Astra's robotic capabilities are not emergent. It was trained on robotics specific data. So were GPT-6.1 Sol and Gemini Argon Here's how we know 🧡
34
78
26
1,065
132,961
Jongwon πŸ‡°πŸ‡· @ CoRL retweeted
hearing some chatter that anthropic is now lowering down to $1tn to get this IPO done. let me explain why this doesnt matter and its actually all over- the entire existence of this company is based on a funding model that inherently has extreme asset liability mismatch. what that means is that just like the hyperscalers are diluting their equity fund capex, there is an expectation that anthropic will also have to do the same. ie it needs strong equity to perpetuate the scheme but if you IPO at the wrong price (and dont get a pop which increasingly appears they wont be; spacex already below) then youre now in the death spiral that crypto investors should be very familiar with: its called the reflexivity trap of the confidence game. its the "ill bid FTT at $22 all day" or the "steady lads UST is $1" game, with all its lucid horrors. and here is the key insight- the confidence game exists at the service of the creditors. if there is no credit, there is simply no equity. and there is no credit, if there is no collateral if the credit rating agencies are not going to give IG at $2tn, they are certaintly not going a better HY rating at $1tn. and there is no world in which anthropic can exist without credit because ~$1tn of commit, much of which is noncancellable, is a straight up negative equity pledge yet there were two scenarios in which i was willing to concede anthropic could IPO: first 1) the improbable but possible was if the hyperscalers/chips oligopoly came to rescue and plug the hole, given their incestuous nature of revenue bookings and paper gains/losses at scale. but it is clear now that the market has spoken this will lead to a mutual death, and should not be attempted. or perversely even better, they are actually better off letting anthropic go to zero so they can acquire the pieces on the cheap which is the best nash equilibrium outcome, not just for the dominant player but for society at large. second 2) the us government can nationalize the company and collude with other sovereigns (like UAE, Saudi, Japan who are basically already the checks) to force its funding as a collective bargain. this gives you trillions, which is actually the most effective backstop to restore market confidence. the first edge case does require an IPO, but the second edge case here for which i believe there is still a good chance that this is the ultimate end game- will require a failed IPO to recapitalize at a normalized price for the consortium (and dont forget that openai front ran this investor group in anticipation of this dynamic to play out as a private co last month) i believe the market is crapping out today because the market is recognizing it will not IPO. and actually, as ive said before, this is a very good thing. i might even imagine that the market could quickly rebound on the news of its IPO being delayed because it allows #2) to play out, as not only a national, but an international bailout. the US is more likely to tolerate, even relish, the taxes of foreginers to fund our domestic companies. that is the only end game that really matters, and the end game i am going to be excited to see we'll see if this newly revamped washington can figure it out
176
220
87
2,619
595,211