Jise Shen retweeted
A pretrained latent world model's predictions can become unreliable when test-time dynamics differ from training.
Introducing JEPA-TTT, which enables persistent test-time training of latent world models for planning under dynamics shifts.
Project Page: jepa-ttt.github.io/
Paper: arxiv.org/abs/2610.00722
Jise Shen retweeted
Can your agent smoothly adapt to your preferences across sessions?
Meet HyperTrace, a training-free LLM personalization framework. It tracks hypotheses about what you want now and what tends to last, rather than reducing you to a fixed profile or a list of memories. 🧵
On PRISM, HyperTrace demonstrates steady online adaptation. It also leads with 80% of feedback withheld and under frequent topic shifts, remaining robust in realistic settings where feedback is sparse and session switches rapidly.
Paper: arxiv.org/abs/2609.09835
Code: github.com/jiseshen/HyperTra…
Huge thanks to @Mei_keyuu, Minghao Shao, @chuanyang_jin, Yusong Wang and Ailiang Lin for your amazing contribution and support!
Jise Shen retweeted
Embodied Recursive Self-Improvement (RSI) needs more than a model—it needs an environment that learns where the agent fails.
🛵 We introduce DeliveryGym—an adaptive RL environment built in Unreal Engine 5, where embodied agents learn to navigate Paris, deliver food, and earn money.
The key idea is simple: earnings are the reward, and failures shape the curriculum. DeliveryGym automatically finds where the agent struggles and generates harder tasks targeting those weaknesses.
📈 With this adaptive RL loop, Qwen3-VL-4B improves net income by 54.3% in a shift.
✨ A step toward embodied agents that fail, adapt, and improve through an environment that evolves with them.
Jise Shen retweeted
6/ 📄 Paper: arxiv.org/abs/2609.19801
Huge thanks to @YimingZhang_33, @gyyyyy710, @Sinclair4o4err, @jiseshen, @XTRose88, @koe_ye40329, and @Lianhuiq for making this work possible! 🙌
And special thanks to @zhaoyk729 for helping with the demo! 🎬
We’re excited about what adaptive environments can unlock for embodied RSI!
Jise Shen retweeted
A coding agent can build an entire harbor and still forget that objects need something underneath them.
We gave GPT-6 and Fable 5.1 the same prompt and asset pack in Code4Scene. 👇
Curious how other frontier models would handle the same scene. Who should we test next?
Jise Shen retweeted
GPT-6 coded an entire world: a busy harbor you can step inside.
From a single prompt and an asset pack, it assembled a container port with towering cranes, docked ships, rows of containers, and forklifts across the yard, all within an explorable 3D scene.
A first look at our upcoming Code-for-3D benchmark, featuring the text-to-scene setting.
Jise Shen retweeted
How to teach AI assistants to infer human minds and proactively provide online assistance?
Sharing our ICML 2026 paper 🧠 𝙈𝙞𝙣𝙙𝙕𝙚𝙧𝙤: Learning Online Mental Reasoning with Zero Annotations!
🤗 Code and weights are fully open source
🌐 scai.cs.jhu.edu/MindZero
Jise Shen retweeted
What are users thinking during their interactions with LLMs?
We introduce ThoughtTrace — the first large-scale dataset that captures what users think during real-world human–AI conversations, not just what they type.
→ 10,174 thought annotations
→ 2,155 multi-turn conversations, 17,058 turns
→ 1,058 users
→ 20 LLMs
These thoughts improve user behavior prediction (+41.7%) and model alignment (+25.6%).
This opens a new paradigm of user-centric LLM research. Full information in the thread 🧶
Read our paper: arxiv.org/abs/2605.20087
Check our project website: thoughttrace-project.github.…
Jise Shen retweeted
Environment generation is the missing scaling axis for embodied AI.
Introducing SimWorld Studio: a self-evolving factory for endless interactive 3D env where agents act, fail & learn.
Env-agent co-evolvution improves navigation success 50% → 90%.
From a prompt, our SimCoder writes code to automatically build an interactive world. Agents train inside it. And their performance shapes the next world.
Jise Shen retweeted
You can now deploy AI agents in any fancy UnrealEngine environments via SimWorld🔥
Agents can now live like humans in a virtual city built with Unreal Engine: exercising outdoors, strolling through parks, or even playing chess with friends!
🌍 Check out SimWorld: an open-ended simulator for LLM agents to act, perceive, and plan in richly embodied, endlessly diverse virtual worlds.
SimWorld now supports plug-and-play integration with arbitrary UE environments. Deploy agents in your own scenes and build fully customizable simulations for autonomous driving, collaborative games, or social experiments!
SimWorld is a strong testbed for evaluating embodied agents, as it provides:
- ⚙️Physical + social simulation: realistic physics (collision/friction/inertia) and social dynamics like traffic rules/flows.
- 🤖 LLM/VLM-friendly interfaces: a Gym-style API with multimodal observations and grounded language actions.
- 🤔 Long-horizon tasks + reasoning: diverse physical/social tasks for systematic training and evaluation.
1/
#LLM #Agent #UnrealEngine #simulation
Jise Shen retweeted
Agents can now live like humans in a virtual city built with Unreal Engine: exercising outdoors, strolling through parks, or even playing chess with friends!
🌍 Check out SimWorld: an open-ended simulator for LLM agents to act, perceive, and plan in richly embodied, endlessly diverse virtual worlds.
SimWorld now supports plug-and-play integration with arbitrary UE environments. Deploy agents in your own scenes and build fully customizable simulations for autonomous driving, collaborative games, or social experiments!
SimWorld is a strong testbed for evaluating embodied agents, as it provides:
- ⚙️Physical + social simulation: realistic physics (collision/friction/inertia) and social dynamics like traffic rules/flows.
- 🤖 LLM/VLM-friendly interfaces: a Gym-style API with multimodal observations and grounded language actions.
- 🤔 Long-horizon tasks + reasoning: diverse physical/social tasks for systematic training and evaluation.
1/
#LLM #Agent #UnrealEngine #simulation
Jise Shen retweeted
Check out our SimWorld Suite at NeurIPS!
🚨Spotlight Presentation🚨
SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds
📍 Exhibit Hall C/D/E, Booth #401
🗓️ Fri, Dec 5, 11:00 a.m.–2:00 p.m. PST
Poster Presentation
Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration
📍 Exhibit Hall C/D/E, Booth #2210
🗓️ Wed, Dec 3, 4:30 p.m.–7:30 p.m. PST
Hope to see you there! 😀
🚨🚨Can agents earn money, run a business, or even self-organize a society in the physical social world? 🤖🤖
Can agents learn continually to survive and thrive in embodied environments, like how human babies grow? 👶
Super excited to introduce SimWorld, an open-ended simulator of LLM agents in infinite, realistic embodied worlds.
SimWorld features 3 key designs:
1⃣Open-ended realistic world simulation
- built on Unreal Engine 5, with accurate physical social dynamics
- 100+ built-in environments (city, island, wilderness ...)
- language-controllable procedural generation
- text-to-3D asset generation
2⃣Native interface for LLM/VLM agents
- Gym-like agent-environment interaction APIs
- plug in any LLMs/VLMs (GPTs, Gemini, Qwen ...)
- rich multi-modal perception
- open-vocabulary natural-language action outputs
3⃣Diverse physical and social reasoning scenarios
- long-horizon embodied reasoning
- multi-agent collaboration / competition
- easily customizable for any reasoning tasks
SimWorld is fully open-sourced, with a hope to become a foundational infrastructure for real-world agent research across disciplines: robotics, economy, public health, education, etc.
Project website + more details in the thread👇 ...1/