Views are my own. researching @Microsoft, @UW, contributed to https://xitter.cf/t.co/VzcJa9Skx3

Seattle, WA
Joined August 2012
Linjie (Lindsey) Li retweeted
Introducing Microsoft-Decision-1, our new model for fast decision-making. It delivers top performance on structured decision tasks, outperforming both LLMs and other decision models in latency and quality. We’re already testing it across Microsoft for everything from incident response and quality control to scientific discovery.
708
1,073
604
11,409
1,694,572
Linjie (Lindsey) Li retweeted
Honored that our RSI-Exam made it into this year's @stateofai 🎉 As AI R&D benchmarks saturate, we need evals with real headroom 📈: 88 RSI tasks 🧪 and the top model (Opus 5.5) still scores ~0.53 🤯 Thanks @nathanbenaich!
welcome to the 9th annual @stateofai report! ai building better ai, world models, physical ai, geopolitics, cyber defense and 12-month predictions. read it at stateof(dot)ai or watch my editor’s cut here. a few of this year’s biggest findings follow:
1
10
44
2,518
Linjie (Lindsey) Li retweeted
GPT-6 Astra can solve some seriously tricky spatial constraint tasks: from unlocking interlocked parts to threading a rope through three rings. Try it yourself now 👇 qinengwang-aiden.github.io/d…
64
216
39
2,517
648,475
Linjie (Lindsey) Li retweeted
GPT-6 Astra from @OpenAI is here. We asked: how far can it push the frontier of RSI? On RSI‑Exam, GPT-6 Astra scored 0.5126, taking the #1 spot. That’s 18.4% above GPT-5.6 Sol and 54.8% above GPT-5.5. RSI just took a generational leap. 🚀
13
11
1
73
5,729
Linjie (Lindsey) Li retweeted
We ran GPT-6 Astra on HumanCLAW-Bench, using the same open-source harness and motion generator. It is a big step over the previous best: - FindSR: 64.9% → 75.5% (+10.6) - NavSR: 42.4% → 57.1% (+14.7) - InteractSR: 16.8% → 46.6% (+29.8) The gains compound: Find-to-Nav conversion rises from 65% to 76%, and Nav-to-Sit from 39% to 72%. Astra also solves 147/507 (29%) episodes missed by all nine previous models . Three things stood out. 1/ Better embodied spatial awareness GPT-6 Astra's dDtb—how far the objects it disturbs end up moving—is 0.65 m, the lowest on the leaderboard, while its #Dtb, the number of objects it touches, remains in line with the other models. It still bumps into things; it just does not keep shoving them. 2/ Stronger spatial memory Sitting is the hard case: the agent has to turn away from the target, so the seat leaves its ego view and it must rely on its own history. Earlier models often sat on the wrong object or on thin air. GPT-6 Astra converts 72% of successful approaches into a correct sit, up from 39%. Once it reaches the target, it never forgets to sit (0 of 108 failures). 3/ Better agentic decision continuity Earlier VLMs often jump between decisions: sit down, stand up again, then wander off. GPT-6 Astra commits. Among episodes where a sit lands on the target, 97% end still seated with a clean stop, compared with 78% for Gemini 3.1. It knows what it is doing and finishes it. ⚠️ Caveats: this is a single run in the low-thinking setting, and 53% of sit episodes still fail. Embodied self-awareness remains the bottleneck we described in the paper; the ceiling has simply moved up. Leaderboard: human-claw.github.io All 1,218 episode videos: huggingface.co/datasets/Choi…
🦾HumanCLAW: Can Vision-Language Models Act Through a Body? A VLM can spot the sofa in a second. But can it get a body there and sit down? We call it Action Intelligence, and our work decouples it from motor control so it can finally be measured. Today’s best models turn out to be surprisingly bad at it. 🔻human-claw.github.io/
4
8
6
87
29,427
Linjie (Lindsey) Li retweeted
HumanCLAW: Can VLMs act through a body? Give VLMs a body, ask it to walk over and sit on the couch. VLMs always fail, in 1,218 long-horizon tasks across 41 indoor scenes: walls block... feet catch on furniture... objects get knocked aside... Real robot testing fuse decision and joint control into one system. When they fail, we may not tell what failed. So HumanClaw strips low-level control error out of the result: the body never falls. In practice: a frozen VLM picks one parametric whole-body skill every 0.5s from egocentric view. A pretrained motion generator turns it into continuous motion. So here, failure = decision failure. Where it breaks: - 1. inefficient exploration: once a target is in view, recognizing it is mostly fine. The more common failure is that the model never brings the target into view at all. - 2. it sees the couch and still can't get there. The model does not know how far away it is or when it has arrived. It stops early, or walks straight past. - 3. it is standing in contact with the couch and still does not sit. Sit success ranges from 90% down to 3.5% across models. - 4. the clearest signal is where the collisions land: mostly on the body parts the model can't see, like legs and feet. VLMs describe a wall correctly, then walk a leg into it. A clear gap in embodied self-awareness! Everything is open: - paper human-claw.github.io - benchmark huggingface.co/HumanCLAW - code github.com/Human-CLAW/HumanC… - leaderboard human-claw.github.io/#leader… Done by the amazing @siyao94 @Kuvvius 👇
🦾HumanCLAW: Can Vision-Language Models Act Through a Body? A VLM can spot the sofa in a second. But can it get a body there and sit down? We call it Action Intelligence, and our work decouples it from motor control so it can finally be measured. Today’s best models turn out to be surprisingly bad at it. 🔻human-claw.github.io/
5
21
1
115
18,829
Linjie (Lindsey) Li retweeted
I couldn't make it to ECCV in person this year, so I teleported my avatar to Malmö instead! 🇸🇪✨ While I'm only there in spirit, you can check out our real-life poster on "imaginative perception tokens" here: 📍 ExHall, Poster #13 ⏰ 10:30 AM CEST 📌 Poster Session 3 #ECCV
2
23
1,270
Linjie (Lindsey) Li retweeted
Can frontier AI agents achieve Recursive Self-Improvement by turning a weak method into one that performs better on hidden data? Introducing RSI-Exam 🧵: 88 executable research tasks across 6 domains, now open for task contributions from every field🧑‍🔬. Its task bank spans virtual cells, TPU kernels, chip design, quantitative finance, agent harnesses, model distillation, and much more. Across these tasks, every rollout follows the same core protocol: 🔁 An agent inherits a working method and iterates using visible data. 🔒 Only its final artifact enters a fresh-container evaluation on the hidden set. 📊 The result so far: Opus 5 leads the 88-task leaderboard with a mean hidden-set score of 0.464. 📢 Want to help build the next release? Author an executable task, review one, or audit a rollout. Substantial contributions qualify for paper authorship. Special thanks to @XinyeYee for the tremendous effort in leading this project. Explore: rsi-exam.ai/ Blog: rsi-exam.ai/blog.html Contribute: rsi-exam.ai/contribute.html GitHub: github.com/aiming-lab/RSI-Ex…
19
83
12
384
124,163
Linjie (Lindsey) Li retweeted
Robotics felt different this month. Google DeepMind and Anthropic made the same question hard to ignore: can a reasoning model decide what a whole body/robot should do next? Gemini Robotics 2 put an embodied reasoning model above a VLA for whole-body control. Anthropic tested general-purpose reasoning models at several levels of control. - deepmind.google/blog/gemini-… - anthropic.com/research/claud… The bad news is clear. General-purpose reasoning models still cannot control a humanoid on their own. They need a pretrained policy underneath. Still, I think this might actually work. A strong model (w/ excellent multimodal and reasoning capabilities) could really do it one day. HumanCLAW is our attempt to get an early glimpse of that future.
🦾HumanCLAW: Can Vision-Language Models Act Through a Body? A VLM can spot the sofa in a second. But can it get a body there and sit down? We call it Action Intelligence, and our work decouples it from motor control so it can finally be measured. Today’s best models turn out to be surprisingly bad at it. 🔻human-claw.github.io/
10
31
1
230
44,580
Linjie (Lindsey) Li retweeted
🦾HumanCLAW: Can Vision-Language Models Act Through a Body? A VLM can spot the sofa in a second. But can it get a body there and sit down? We call it Action Intelligence, and our work decouples it from motor control so it can finally be measured. Today’s best models turn out to be surprisingly bad at it. 🔻human-claw.github.io/
3
20
3
90
63,696
Linjie (Lindsey) Li retweeted
Really cool to see @AnthropicAI benchmarking with MolmoAct 🤖 Wild finding: LLM supervisors actually hurt performance on familiar tasks while MolmoAct solo beats every LLM+VLA combo. But on novel tasks the VLA can’t solve alone, the best models provide net uplift. Judgment > deference. anthropic.com/research/claud…
1
8
2
86
13,149
Linjie (Lindsey) Li retweeted
we’re excited to learn about your work on multimodal agents and will select one submission for a best paper award 🏆 please submit by the extended deadline Aug 5 (AOE)‼️
📣 Call for Papers — 1st Workshop on Multimodal Digital Agents (MDA) @ ECCV 2026 How should AI agents see and act across web, desktop, and mobile interfaces? We're bringing together work on vision-centric digital agents — perception, reasoning, benchmarks, safety, and more. 🗓️ Sept 8, 2026 (ECCV) · ✍️ Submit by July 29 (AoE) 📄 Full paper (≤14 pages, ECCV-accepted welcome) or extended abstract (≤4 pages) — non-archival, 🏆 best paper award 🎤 Speakers include Graham Neubig, Ranjay Krishna, Qianhui Wu & Alexandre Drouin Submit / details → mda-workshop.allen.ai/ #ECCV2026 #ComputerVision #AIagents #Multimodal #GUIagents
6
19
3,283
Linjie (Lindsey) Li retweeted
Excited to share that our paper, Imaginative Perception Tokens, has been accepted to ECCV 2026! 🎉🇸🇪 🌼
Picture your living room. If you sat on the sofa, would the TV be on your right or left? You didn't reason in words,you placed yourself in the scene.Imagining in visual space, not text.Exactly what VLMs can't do.Our new paper tackles this with Imaginative Perception Tokens(IPT)🧵
5
31
3,786
Linjie (Lindsey) Li retweeted
IPT is accepted to @eccvconf 2026! Many projects use text CoT for spatial reasoning, but humans imagine the scene. IPT teaches VLMs to reason through imaginative thoughts. #ECCV2026 #ComputerVision #VLM #SpatialReasoning #Robotics #WorldModels
What if VLMs could imagine before answering? IPT supervises visual intermediate states for spatial reasoning: 1. Path tracing → side view 2. Perspective taking → new viewpoint 3. Multiview counting → top-down map Paper: arxiv.org/abs/2606.03988
2
6
31
3,799
Linjie (Lindsey) Li retweeted
We believe in the representation of 3d point trajectory as an alternative form of world modeling. it is universal to all object, more compact than pixel, and can directly be consumed by many downstream task. p.s. the key for us to generate high quality real world data is we respect objectness and do hard work in object grounding -- which allows us to properly filter and smooth noisy trajectories coming out of off the shell 3d tracking methods.
We're releasing MolmoMotion, a 3D motion forecasting model. Given one or a few video frames, 3D points on an object, & an instruction like "Put the white bowl on the table," MolmoMotion predicts where those points will go over the next few seconds in a shared 3D world frame. 🧵
3
11
3
63
137,772
Linjie (Lindsey) Li retweeted
Planning with the views: Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We introduce ViewSuite with 6 DoF camera control and ~165K task instances, testing: Path-to-View View-to-Path Interactive View Planning A sharp Planning Gap emerges: + can roughly "track" how camera action changes views - cannot "compose" a plan towards a target view at all We then try to teach VLMs with Reinforcement Learning. - RL cannot teach VLMs such planning ability, only 2.5% success rate with Qwen2.5-VL-7B. + With View Graph Distillation (our RL-Graph-SFT framework), 2.5% → 47.8% Below, we answer these questions: Q1. What are the failure modes? Q2. How can we make RL work? Q3. What has the model learned? Can we open up the model to see before/after? Can such spatial priors transfer to other view related tasks? Led by @James_KKW, great to work with @LINJIEFUN @zhengyuan_yang @shiqi_chen17 @wzenus @drfeifei @jiajunwu_cs Leonidas Guibas, Lijuan Wang. A joint efforts with @StanfordAILab @StanfordSVL @MSFTResearch.
21
34
4
199
65,019
Linjie (Lindsey) Li retweeted
can AI write engaging news that people can trust? introducing ✨Data2Story: a data journalist agent. give it raw data, it generate a verifiable, multimodal article. 🔍verifiable: every claim is evidence-grounded, traces back to data, code, or a cited source. 🔮multimodal: the article is a generative UI — images, videos, audio, interactive charts. not just readable, but trustworthy and playable. 🧵1/N
12
49
2
107
33,673
Linjie (Lindsey) Li retweeted
What if VLMs could imagine before answering? IPT supervises visual intermediate states for spatial reasoning: 1. Path tracing → side view 2. Perspective taking → new viewpoint 3. Multiview counting → top-down map Paper: arxiv.org/abs/2606.03988
4
15
1
69
9,243
Linjie (Lindsey) Li retweeted
Picture your living room. If you sat on the sofa, would the TV be on your right or left? You didn't reason in words,you placed yourself in the scene.Imagining in visual space, not text.Exactly what VLMs can't do.Our new paper tackles this with Imaginative Perception Tokens(IPT)🧵
1
12
2
31
7,196
Linjie (Lindsey) Li retweeted
What if VLMs could imagine visually before answering spatial questions? New paper: Imaginative Perception Tokens (IPT) teach multimodal LMs to reason about hidden 3D structure — without generating images at inference time. Paper: arxiv.org/abs/2606.03988
5
10
59
69,193